This week we experienced an issue with two virtual machines not showing the correct values in the vCloud Director view. Both virtual machines had 2 vCPU when we checked the virtual machine properties in vCD, but the values were totally off when we compared the values with the VM properties from the vCenter GUI. The two virtual machines were also having the status “partially powered off” in the vApp view. The customer confirmed to us that the production environment was still up and running even though vCD reports the two virtual machines as “partially powered off”.

Troubleshooting the partially powered off error

Investigating in the vCloud and vSphere layer

Both virtual machines were residing in a specific cluster according to vCenter, but i couldn’t find those two servers in the resource pool view from vCloud Director. So that brought me back to my previous article where i had the same symptoms. Unfortunately, refreshing the vCenter inventory and VM config didn’t fix the issue this time.

Investigating in the SQL DB layer

Please be advised that on this part with the queries mentioned below, we will not change anything in the database. We are just going to query the database, to see if we can find out what might be off in the database. Open the Microsoft SQL management server and use query below to find one of the two virtual machines in the vapp_vm table.

select * from vapp_vm where name like 'ServerName'
Partially powered off
We now have the vApp_VM information

Use the svm_id from the previous output and create a new query as shown below to retrieve the VM information from the VM table.

select * from vm where id = 0x4BBC4359DADE425BA70E249E8B80813A
vcloud
We now have the VM information from the faulty VM

Make a note of the moref, location_path and cloud_uuid values. In my case i had the following values:
moref = vm-345704
cloud_uuid = 4bbc4359-dade-425b-a70e-249e8b80813a
location_path = [FAS-SVM-hidden_Data001] VS4980.hidden.com_clone/VS4980.hidden.com.vmx

Use the cloud_uuid from the previous output and create a new query as shown below to retrieve all the virtual machines with the same cloud_uuid.

select * from vm_inv where cloud_uuid = '4bbc4359-dade-425b-a70e-249e8b80813a'
duplicate vm
As you can see we have a duplicate VM.

In the previous screenshot we see 2 virtual machines and one of them is a duplicate. Restoring a VM as clone in vCD is an unsupported action. The moref ID from the duplicate vm (the one with _clone in the name) matches the moref ID as the one we noted earlier.

We now need to make a note of the moref, location_path and cloud_uuid values from the correct VM.
moref = vm-35231
cloud_uuid = 4bbc4359-dade-425b-a70e-249e8b80813a
location_path = [FAS-SVM-hidden_Data001] vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1)/vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1).vmx

Resolution

Create a valid SQL database backup of the vCloud database before making any changes to the database.

Note: If you are not comfortable with making any changes in the SQL database, please contact the VMware support team so that they can do this for you. You can easily create a support ticket at VMware by visiting the following url.

Removing duplicate VM from inventory.

The first thing we need to do is to remove the duplicate VM from inventory. This can be done on the vSphere layer by opening the vCenter GUI, selecting (right mouse click) the duplicate virtual machine and click on remove virtual machine from inventory.

duplicate vm

Execute the previous query and verify the is_deleted column.

select * from vm_inv where cloud_uuid = '4bbc4359-dade-425b-a70e-249e8b80813a'
sql
The database will be updated after removing the duplicate VM from inventory.

After the removing the virtual machine from inventory, the duplicate object will be update with “1” in the is_deleted column. This object will be purged automatically, so leave the object as it is.

Change the moref ID from the faulty VM object in the VM table

Changing the moref ID from the faulty VM in the VM table can be done with the following query. Make sure to use the moref from the correct VM as we noted earlier.

update vm set moref = 'vm-35231' where id = 0x4BBC4359DADE425BA70E249E8B80813A and name like 'vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1)'

Change the location_path from the faulty VM object in the VM table

Changing the location_path ID from the faulty VM in the VM table can be done with the following query. Make sure to use the location_path from the correct VM as we noted earlier.

update vm set location_path = '[FAS-SVM-hidden_Data001] vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1)/vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1).vmx' where id = 0x4BBC4359DADE425BA70E249E8B80813A and name like 'vs4980.hidden.com (3a58af01-644f-47cc-bdb8-95e8136d00c1)'

Change the vmoref from the faulty VM object in the computevm table

Lets query the computevm table with the query shown below. Use the moref ID from the duplicate VM.

select * from computevm where vmmoref = 'vm-345704'
vcd

Make a note of the ID value from the object in the computevm table. We will use this ID value to update the object with the right moref ID.

We did the same query again but this time with the correct VM. We didn’t even got any results back. So we can confirm that the computevm object was still referring to the old moref ID. With the next query we will update the vmoref id with the moref ID from the correct VM.

Use the following query to update the computevm object with the correct VM moref ID.

update computevm set vmmoref = 'vm-35231' where id = 0x3A58AF01644F47CCBDB895E8136D00C1

The VM was back in it original state with the correct VM info in the vCloud Director GUI after the change.