VM /ETC

Posts Tagged ‘esx’

Being that it is the end of the week and I have fallen a day or so behind in keeping up with the VMware updates for the August 12 time bomb bug, I decided that I would post the remaining emails I received from VMware. I’m also including an update from John Troyer from the VMTN Blog providing some clarity on version number information differences between the install media and the express patched ESX/ESXi 3.5 versions. Read the rest of this entry »

I just received the third VMware email customer update about the August 12 time bomb bug. Unfortunately, this email informs customers that the effort to find a resolution where VM reboots could be avoided has not produced a reliable process. The email does indicate that the express patches are working properly and are compatible with VMware Update Manager. Also, The previously set deadline of 6 PM PST today (AUG 13) for the download availability of the corrected ESX/ESXi update 2 install media is still on schedule.

Here’s the body of the entire email as I received it: Read the rest of this entry »

As I mentioned in my post My thoughts on the reactions to the ESX 3.5 Update 2 BUG VMware has been straightforward and honest about their responsibility for the time bomb mistakenly left in their code. VMware’s new CEO, Paul Maritz, has provided another example of this with his blog post VMware: The Console: Letter from VMware CEO Paul Maritz. Read the whole post, but here’s some of the letter I find informative and worth repeating:

“The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.

In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.

I am sure you’re wondering how this could happen. We failed in two areas: Read the rest of this entry »

The product expiration time bomb that was mistakenly left in the first versions of the ESX 3.5 and ESXi 3.5 Update 2 download media is no doubt an embarrassing and horrible mistake by VMware. The timing of this disaster couldn’t be worse with Microsoft Hyper-V, Citrix XenServer, and others starting to be considered as an alternative virtual infrastructure platform for companies just beginning to explore the benefits of virtualization. How could this have happened and what are some lessons to be learned, not just for VMware, but for VI administrators around the world? Read the rest of this entry »

Based on the email I received from VMware at 1:14 am EST last night, the express patches for the ESX 3.5 and ESXi 3.5 Update 2 product expiration bug have been released and are available for download. This availability would have been 10:14 pm PST and slightly later than the 9:00 pm promised deadline. This is still impressive customer response in my opinion as the ESX team is no doubt working around the clock to provide a fix around the world. VMware’s next deadline is 6 pm PST tonight (August 13) to provide the reissued upgrade media.

There are now 3 KB articles related to this bug:

• Fix of virtual machine power on failure issue, refer to KB 1006716
• For VI 3.5, refer to KB 1006721 for deployment consideration and instruction
• For VI3.5i, refer to KB 1006670 for deployment consideration and instruction

The whole email is copied at the end of this post, but here is the important information from the email about the express patches:

We have released the express patches for the product expiration issue. Please
go to http://www.vmware.com/go/esxexpresspatches for more information.

1. What do the express patches do?

There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5
Update 2. They are specifically targeted for customers who have installed or
fully upgraded to ESX/ESXi 3.5 Update 2 or who have applied the
ESX350-200806201-UG patch to ESX/ESXi 3.5 or ESX/ESX 3.5 Update 1 hosts. For
customers who haven’t done either, these express patches should not be applied.

To be noted is that these patches have been validated to work with esxupdate.
However, testing with the VMware Update Manager is still under way. In
subsequent communications, we will provide confirmation whether the patches work
with VMware Update Manger or if a re-spin is required.

We are currently testing an option to apply the patch without requiring
VMotion or VM power-off and re-power-on at the point of patch application. To
immediately refresh vmx on the VM, one can VMotion off running VMs, apply the
patches and VMotion the VMs back. If VMotion capability is not available, VMs
can be powered off before the patches are applied and powered back on
afterwards.

The following is the email I received from VMware in it’s entirety. Read the rest of this entry »

I just received word that the following email is being sent (or will be sent) to all ESX 3.5 and ESXi 3.5 customers likely to be impacted by the Aug 12 time bomb BUG. I assume the 6:00 pm deadline is PST as VMware’s headquarters is in Palo Alto, California?

Dear VMware Customers,

Please find the latest update about the product expiration issue. From this point on, we’ll provide an update every two hours. Thanks.

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:

- VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2

- Reports of problems with ESX 3.5 U1 with the following 3.5 Update 2 patch applied.

1. ESX350-200806201-UG

- No other VMware products are affected.

What has been done?

Read the rest of this entry »

I just ran across this rapidly growing thread in the VMware Community forums. Apparently, the date August 12, 2008 is literally a bug in ESX 3.5 Update 2. If you have already upgraded be prepared to experience the following issues as described at VMware Communities: BIG bug in ESX 3.5 Update 2 – If you’re … (check this thread for the latest as it is updating frequently as of this posting):

The bug:

Starting this morning, we could not power on or VMotion any of our Virtual Machines. The VI Client threw the error “A general system error occurred: Internal Error”.

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired.
Aug 12 10:40:10.792: vmx| Be sure that your host machine’s date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: “http://www.vmware.com/info?id=4″.

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around: Read the rest of this entry »