My Rant about “The Benchmark”
March 19th, 2009 | 
Author: Hypervisor Test Explained is a Virtualization Review post by Rick Vanover written in response to the fallout over “The Benchmark”. Real quick for those few that do not know, Rick, along with Editor in Chief Keith Ward, recently published (in Rick’s words) “a comparative performance test” for VMware ESX, Microsoft Hyper-V, and Citrix XenServer. The results of that test have been hotly debated since. Mostly between VMware and Citrix, but other skirmishes in the form of comments, tweets, and blog posts have also popped up scattered around the virtualization blogisphere. Until recently I’ve sat on the fence about the test results and the reactions. Partly because I found the test’s outcome startling, but also because I found myself disagreeing with the position and opinions of both sides as I watched the battle.
Now, before I continue let me establish that I immediately questioned why this was even being debated as a hypervisor “benchmark” at all. In my mind it was always what Rick describes now: a comparitive test. The goal wasn’t to say hypervisor A, B, and C can do X amount of Y and Z. Rick’s comment on Jason Boche’s early post on this topic makes it crystal clear what his objective really was:
“… everyone is assuming I’m offering this as information for the enterprise. Not so. I really am targeting this to the customer who is going to select the free hypervisors for small, unmanaged installations.”
To satisfy me for “small unmanaged installations”
all you have to do is use the same hardware, the same VMs, the same applications, and the same test processes. That’s a fair test for a defined scenario. It’s real world. The hypervisors are installed out of the box by SMB administrators. The configurations are left at the defaults. Rick describes it best:
“We have CPU hogs and memory beasts that we want to get virtual, and the canned response of an application not being a virtualization candidate is a warning we may not heed by choice. We work with default configurations because we don’t always have the time or other resources to go about it another way. We don’t work with “camera-ready” databases.
On the other hand, don’t claim absolute victory either. You can proclaim yourself today’s winner in the “small unmanaged installations” scenario. I’ll help you do it, but you cross the line with me when you make broad and general statements based on this test. Wait until 6 months from now when Rick and Keith do the same test (hopefully) with shared storage or all 64 bit hypervisors for example?
In closing, Keith Ward brought up a point I feel needs to be re emphasized in one of his responses to the fallout:
“And I don’t know how I can state it any more clearly: This test only determined one aspect of these hypervisors. To make a buying decision based on this article would be foolish in the extreme.”
Now, if we are going to really benchmark hypervisors then Chris Wolf’s cry for SPECvirt independent testing is the way to go. Let’s hope we get there. I also applaud Chris for organizing a public debate on this topic as well give a standing ovation to both VMware and Citrix for agreeing to participate in The Thrilla in California. I only wish I could be there on July 29 in San Diego to cover this in person!
Related Posts
Posted in citrix, microsoft, vmware | 
Tags: benchmark, citrix, microsoft, virtualization, virtualizationreview.com, vmware
-
Chris Wolf
-
rbrambley
-
Duncan
-
rbrambley
-
Duncan
