The requested operation could not be executed on VM – error in vCloud Director 9.7 – Step-By-Step

The requested operation could not be executed on VM “VM Name”. Stop the VM and try again.” was the error message that i received when I tried to remove a specific vApp or a single VM in that particular vApp. I even waited a few minutes when we received this message, but it never got cleared. The troubleshooting steps were done in a vCloud Director 9.7 environment.

Investigating on the ESXI/vCenter layer

The very first thing to do is to check the status of every VM in that particular vApp. In my case there was only 1 VM in that vApp and it was showing an pending action on the VM. When selecting the VM or vApp we didn’t had much options available beside of deleting the VM or vApp. The next step was verifying the status of the VM on the ESXI layer. To my surprise, the VM was in an powered off state so the error message in vCD was not really helpfull.

I couldn’t find any other errors in the log that could be linked to a vCenter issue. That leads us to the vCD database that most likely has an transaction that couldn’t complete in a certain amount of time. In my case we have an Microsoft SQL server that runs the vCD database.

Investigating on the vCD layer

We will now begin with checking the current status of the vApp.

Run the following query in SQL:

select * from vm_container where name like '%VAPPNAME%'					

What we are looking for is the creation_status from the vApp. In the picture below we can see that the status is: “DELETING_CONTENTS“. Normally, when a vApp is powered off, it should be reporting “RESOLVED” as status. The vCD database was never updated to resolved of a missed transaction in database.

the requested operation

Now we are going to query all the virtual machines in the vApp to see the creation_status. This will determine the VM that is causing the vApp getting into a non-responsive state. In my case, I only had one VM.

Run the following query in SQL:

select * from vapp_vm where vapp_id in (select sg_id from vm_container where name like '%VAPPNAME%');										

In the picture below we can see that the only VM in the vApp is causing the issue. The VM has the same creation_status as the vApp. We expected to see the status RESOLVED for that VM.

sql

Next step is to verify if the VM has an entry in the deployed_vm table.

Run the following query in SQL:

select * from deployed_vm where vm_id in (select svm_id from vapp_vm where vapp_id in (select sg_id from vm_container where name like '%VAPPNAME%'));								
query

Resolution

In my case, the vApp consist out of 1 VM. When you have multiple VMs in a vApp, you should edit the query to select only the specific VM that has the DELETING_CONTENTS status.

The very first thing we need to do is to create a backup of the vCD database! Also, if you are not confident enough to perform SQL tasks, please contact VMware support to assist you with that. That being said, let’s start with deleting the VM record in the deployed_vm table.

Run the following query in SQL:

delete from deployed_vm where vm_id in (select svm_id from vapp_vm where vapp_id in (select sg_id from vm_container where name like '%VAPPNAME%'));								

We can now change the creation_status of the VM to RESOLVED.

Run the following query in SQL:

update vapp_vm set creation_status = 'RESOLVED' FROM vapp_vm where vapp_id in (select sg_id from vm_container where name like '%VAPPNAME%');								

The last update will change the creation_status of the vAPP to RESOLVED.

Run the following query in SQL:

update vm_container set creation_status = 'RESOLVED' where name like '%VAPPNAME%'								

You can use the previous queries from the “Investigating on the vCD layer” to validate the creation_status of the vApp and VM.

You are now able to delete the vApp from vCD without any issues. To avoid these kinds of missed transactions, VMware recommends to migrate to an PostgreSQL database.

Leave a Reply

Your email address will not be published. Required fields are marked *