VMware: Don't shut down that virtual machine

Update at 8:35 a.m. PT on Wednesday: Since ZDNet UK published this article, a patch for the flaw has been posted to VMware's Web site.

VMware virtual machines on all hosts with the company's latest hypervisor, ESX 3.5 Update 2, in enterprise configurations have found that it will not power on after being turned off.

The hypervisor refuses to start when the date is August 12, with customers around the world discovering the problem as midnight was passed in their time zones. A flaw in the VMware licensing code is responsible, according to Martin Niemar, group manager of virtual-infrastructure product marketing at VMware.

"We had an issue with 3.5 Update 2. It's actually a licensing problem," Niemar said. "Currently, what we know is that licensing prevents new virtual machines from powering up after shutdowns, and it prevents virtual motioning--moving a virtual machine from one host to another."

Niemar said VMware does not have a patch but that working on one is a "top priority."

"Customers should not stop virtual machines. Keep virtual machines going until we release a patch," Niemar had said. "You can also move the clock backwards on the server."

Some organizations cannot turn server clocks back for legal or technical reasons. Niemar said that, if customers have to turn machines off, and cannot turn clocks back, there is currently no fix. (Editors' note: A patch is now available on VMware's site.)

Niemar could not commit to a time line for a patch, nor could he comment on forum claims that the fix will first be available to customers as a complete reinstallation from ISO or TAR images, with a patch for installed code coming later.

"We understand the bug," he said.

VMware first learned of the issue when Asia-Pacific customers started to come online on August 12. Technical issues have been discussed on the VMware communities blog.

"VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media, including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade TAR and ZIP files by noon, PST, on 13 August," one poster wrote. "These will be available from the page: http://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2. The Update patch bundles will be released separately, later in the week."

At the time of writing, ZDNet UK was unable to confirm this blog comment.

Tom Espiner of ZDNet UK reported from London.

[pjhenry1216]

Leave it to licensing to break something this badly. When will companies learn that things will get pirated anyway so all you do is cause problems for paying customers. This is a prime example. This is a critical issue and this very well may have lost them a LOT of customers.

[ironbyghte]

QFT!!

[Lerianis]

pjhenry hit the nail on the head. If these things are ONLY creating headaches for LEGITIMATE USERS (and that is pretty much the thing right now, hackers/crackers can STILL pirate anything).... it's time to get rid of the licensing schemes and just rely on the fact that whoever wants to get the thing legally and can afford the thing is going to buy it legally.

[kkohnen]

I'd be tempted to sue the bejeezus out of them. If a licensing issue causes a user not to be able to use what he's purchased, that's theft.

They didn't detect the problem before releasing it? That's incompetence.

They don't have a fix for the problem yet? That's gross incompetence.

[Vegaman_Dan]

Did you test your car to see if it could make a grilled cheese sandwich this morning? If not, that's incompetence.

Did you verify your cat was not actually a mutant alien bent on destroying the world throug the power of jello manipulation when you let them out this morning? If not, that's incompetence.

It's easy to toss around such comments without fear of retribution or consequences. I would give them a break- it's clear this took them by surprise and they did have a fix for it within a day. At least they acknowledged the issue and worked on a solution. I give them high marks for that alone.

[The_Decider]

Geez Dan, can you be a bigger idiot?

A VM is supposed to be able to be started and stopped.

I give you high marks for idiocy.

[Orion Blastar]

Patch Tuesday strikes again!

I had friends and relatives that had their computers crash yesterday and their Internet wouldn't connect. After I rolled back some updates, they worked again.

[brickman5721]

Seriously guys, get your facts straight! I am very disappointed in CNet both for publishing a story that broke one and a half days ago, and then when they do write the piece of FUD, they neglect to mention that there IS in fact a patch!!!!! Please correct this immediately!

[Vegaman_Dan]

The story was updated with the reference to a patch and a link to it.

You were saying?

[supergoodnachos]

Since this article August 13, 2008 6:32 AM PDT, and a patch was released 8/12, ok it was late, but still released a day earlier there is no reason this invalid info should be on cnet.

[benjaminstraight]

So a user can't use what they bought because of a license.

[brickman5721]

Just to clarify, CNet, the patch was out 12 hours BEFORE you published this article.

[Penguinisto]

ROTFL... my partner-in-crime @ work does most of the VMs - he's still crapping bricks and testing things 98 ways from Sunday to make sure everything runs - and we hadn't bumped ESX up to 3.5 yet (we were about to start doing so in the next couple of weeks... I'm thinking he wants to wait awhile now).

On the plus side, he's stopped demanding that we put every service we own onto a VM these days. :)

By the by, the patch that a couple of folks are screaming about hasn't been tested yet. Any sysadmin with even the slightest sense of intelligence will test a patch thoroughly before applying it (sometimes the cure can be worse than the disease, you know?)

VMWare uses the licensing as a sort of money machine. You get x number of "CPUs" you can spread things around on (and host system RAM limits too, but I digress). Want to allocate more 'oomph? You pay for more licenses. Otherwise, it's pretty useless to have a licensing server hanging around.

(speakin' of which, wasn't VI 3's grace period --in case of license server outage-- supposed to kick in at some point?)

Either way, man, that's got to be embarrassing... Xen may be a PITA to use and incomplete, but at least it doesn't fall down and go 'splat' on you due to some stupid licensing issue.

[supergoodnachos]

Some of you clearly do not understand the issue at all. The licensing problem was due to the expiration date of the Update 2 BETA that was mistakenly left in the GA code. All VMWare beta products have a hard coded expiration date to ensure that old beta software doesn't make its way into production environments. This has nothing to do with VMWare trying to get rich by, gasp, making people pay for the software. Oh ya, Penguinisto you may want to inform your partner-in-crime that this issue only affected ESX 3.5 Update 2, if you are not even on 3.5 yet this issue is completely irrelevant, so what exactly is he testing?

[Penguinisto]

He's updating to v3.5 on a test machine right now.

[brickman5721]

@Penguinisto

Members of the VMWare community have applied the patch and are no longer experiencing problems. The core of the issue here is that this problem surfaced yesterday morning and a fix was released yesterday afternoon. Yet CNET decides to publish an irrelevant article that contains an excess of fear, uncertainty, and doubt.

[Penguinisto]

Actually, they have an update mentioning the patch. To be honest, it is a big problem. Managers everywhere demanding that we buy more blade server kit and pack it to the gills with VMs, while our local VMWare reseller comes in every other month now with a huge smile on her face.

Now personally, I like using VI and ESX - it's easy to manage, works quite well, and overall it does the job. I also understand that sometimes bad code slips through. OTOH, man... I can't go in and tell folks that VMWare is up to the task of mission-critical when explosions like this surface and spook the bosses. Now to be fair, I can't say the same about Xen either (only because it's still a bit too work-intensive and just a bit ******, for lack of a more precise term). To be doubly fair, I can't say the same for Viridian (a colleague in a sister department of ours finally gave up on that after any and all attempts at clustering with it failed miserably).

But therein lies the rub.

/P

[Vegaman_Dan]

The problem I have with VM's is that the more and more enterprise management only see numbers and believe they can keep stuffing more and more VM's onto the same physical equipment, they don't seem to realize that when that same piece of equipment takes it into its head to go whacko that you just lost all those VM's as well. It's not a good idea for redundancy or high production/reliability systems. It *IS* cheaper than running separate machines, and easier to manage, but when things go boom, you have that single point of failure vulnerability. Rack servers can have redundant power supplies, VRM's, CPU's, memory banks, fans and hard drives- but the system board links them all and it only takes one chip to fail and you've got a dead system on your hands.

I'm still of the mindset to keep them separated on different machines. Redundancy by having parallel systems for load balancing. Sure it costs more for the equipment, but you also aren't likely to take down multiple systems and online services because one machine dies on you.

[Penguinisto]

For once Dan, you're making sense (sorta).

Most VM suites have things like VMotion that keeps some semblance of HA present on a server. OTOH, the shift isn't always easy to catch, and it can still cause disruptions (and possibly data corruption).

My only real kick against VM's has more to do with allocation of resources. Even if you have 16 CPU's and 128GB of RAM on a box (which means licensing will cost you almost as much as if you built the things out separately), you still have bottlenecks (system buses, network, etc), and you still have overhead. It doesn't help that the powers-that-be want to cram in as many active VM's as possible on each physical machine.

[willrsnds]

Id like to start off my commenting to those that say VMware has lost lots of customers and that some are willing to sue. YOU HAVE GOT TO BE KIDDING. First off, they had a fix for there product faster then any other company I have dealt with. That alone makes me feel better as a VMware customer. Also, for that line of thinking sue Microsoft for patches that have screwed up your system or applications, or any other company for that matter. Second, who said you had to update to "Update 2" as soon as it came out. Best practice dictates that you should apply updates to your test envirnoment before your production. So for those who did not do this, that is your own fault and any ramifications for that fall on your shoulders. And I can hear it already, we dont have equipment to do testing. Well then you dont have the resources to implement virtual technology properly. So, learn the product, learn how to do testing, and you wont find yourself in this situation.

[roanry]

I think c | net is owned by Microsoft some how.