There is a lot to consider in the first paragraph, and as any consultant would say, the final decision depends on what other objectives the solution will need to satisfy besides just serving as server infrastructure. Again, for the sake of consideration, I’m going to zoom in on the server infrastructure and leave the “other” out of the implementation.

Again, for the sake of consideration, can the difference between choosing to implement production virtual infrastructure with VMware or Microsoft be simplified to a aligning with either companies strategic vision? I’ll attempt to make that case in this post.

For the sake of being open and honest before I continue, I’ll state up front that I personally have yet to implement a production Hyper-V environment, but as a consulting engineer working for a large Microsoft partner, I’ve sat in certification training, experimented a little in the home lab, and have been looking very closely at the implementation services needed to deploy Hyper-V for customers recently.

This post holds my thoughts on some major implementation differences as I understand them today. Please point out where I have missed the mark or help me consider other factors that I may have missed.

Microsoft virtualization is a part of Windows

Implementing Hyper-V Makes the most sense after you already have Active Directory (AD) with System Center Operations Manager (SCOM) already in place. Sure, Hyper-V Server 2008 can be installed bare metal as a stand alone hypervisor or it’s possible to build VI with Server 2008 core before AD, but are those scenarios really feasible for production?

Hyper-V with HA (Quick Migration), Live Migration, and all of the other advanced features essentially depends on Fail Over Clustering. Fail Over Clustering depends on Active Directory. Therefore, it makes the most sense to establish physical AD infrastructure and join Hyper-V hosts to the Forest or Domain. Sure, most companies will already have a healthy AD already in place.

Should System Center Operations Manager and Virtual Machine Manager be physical as well? They don’t have to, but that decision, in my opinion, ultimately comes down to a comfort level about running the monitoring and management servers inside the environment they are monitoring and managing. Logistically speaking, the SCOM pieces may be the first VMs created on the first several Hyper-V hosts, however. Since the added value of  SCOM is administration of physical hosts as well, does it make more sense to keep SCOM physical too?

Think about my scenario as a consultant. Given a week or so to perform the implementation and server consolidation as a turn key project, I can build stand alone Hyper-V servers and manage them with the Hyper-V Manager easily enough, but I will quickly need Fail Over Clustering. SCVMM is arguably needed shortly after.

These are some implementation reasons that align with the idea that Microsoft has built Hyper-V as an extension of a Windows infrastructure that already exists. Hyper-V seems to favor scenarios where parts of the environment will be consolidated. If you really think about it, of course Microsoft would do it this way.

VMware wants to virtualize 100% of your data center

Active Directory makes life easier in VMware vSphere environments too, but implementing most of the desirable vSphere features is not dependent on it. DNS is needed, but not necessarily Microsoft DNS.

An ESX Cluster is not a traditional, quorum based application cluster but instead a pool of hardware resources joined together by the vCenter Management Server and it’s agents deployed on each ESX host. Microsoft Clustering is not needed and it’s dependencies can be avoided.

Most of the VMware implementations I have been involved with over the last 4 years could have been viewed as complete data center migrations. I’ve suggested that migrating to VMware VI should be treated by the IT Team the same as a physical data center move based on that fact. Therefore, customers usually P2V the Microsoft AD infrastructure entirely to the VI. More times than not there is a tough decision about whether to leave at least one physical DC in the environment.

Final Thoughts / Questions

To summarize the last 2 sections I pose some implementation centered thoughts / questions to VMETC.com readers:

• Is it an important point to understand that a VMware vSphere environment can completely host Windows AD infrastructure as guests eliminating the need for any physical servers other than the vSphere infrastructure?
• Does it matter that Microsoft Hyper-V environments need an established Windows environment? Remember I am talking about production implementation using best practices for your company / my customers. I know Hyper-V can be configured in a workgroup without a DC, but is that really an production implementation option considering feature needs like Live Migration and HA?
• Do you view a layer of separation between Active Directory and the hypervisor as an added benefit? What happens if you corrupt the AD database in VMware versus Hyper-V environments?

Let me know your thoughts!

“Following the lead of analysts and customers, VMware has adopted “cost per application” as a more accurate metric to compare costs between virtualization solutions. Going beyond a simplistic license price comparison, evaluating cost per application takes into consideration the number of virtual machines that can be run on a single server, or the ”virtual machine density” enabled by a specific virtualization solution. The higher the virtual machine density enabled, the higher the server consolidation ratio. The higher the consolidation ratio, the more an organization can reduce its infrastructure costs (including costs for servers, networking. storage, power and datacenter space), and software costs, (including guest operating system and virtualization software licenses). Following this logic, the solution that can provide the higher consolidation of servers – without an impact to performance – also provides the most value to customers.”

I decided to test the calculator with a 50 VM / application example.

Note: I have not verified prices used in the VMware calculator, nor have I deeply analyzed the methodology.I would recommend companies considering virtualization use the results of the calculator as a starting point and adjust accordingly with real figures provided by the representatives and partners of both Microsoft and VMware.

VMware’s calculator requires answering six questions, and the result is a detailed report based primarily on the following formula:

Hardware Cost
(virtualization hosts, management servers, networking, storage)
+
Power and Cooling Cost
+
Datacenter Real Estate Cost
+
Guess Operating System Software Cost
+
Virtualization Software Cost
+
Virtualization Management Software Cost
=
Total Cost of Deployment
/(Divided by)
Number of Application Virtualized
=
Cost per Application

Screen shot of the 6 inputs:

After submitting, a 10 page report (after .pdf conversion) was generated. The Executive Summary section sums up the results:

Based on your inputs, the cost-per-application to virtualize 50 applications using VMware Infrastructure 3 Enterprise Edition is $3,105 — 20% lower than with Microsoft Windows Server 2008 (Hyper-V) and System Center.

The report provides much more detail, but here is a summary table of the total costs:

For more details about the methodology and assumptions of the VMware Cost Per Application Calculator read this white paper.

I would not be surprised if there is a lot of debate between Microsoft and VMware over the methodology and the results.

I hope this this calculator is either expanded in the future or another one is introduced to compare XenServer and VMware VI Enterprise (current and future versions) as well.

Here’s the test setup info from the report:

“Here, we test the top three different virtualization platforms: VMware Infrastructure 3.5, MS Hyper-V and Citrix Xen Server 5 to help you make an informed decision.

Test setup
For testing we used the latest Intel Dunnington Server having 4 processors with 6 cores each and 16 GB RAM. For benchmarking we used CINEBENCH 10 64-bit, POV Ray 64-bit and Linpack. We ran these benchmarks on Windows 2003 Enterprise 64-bit OS which was installed virtually on different platforms. We tried to form a common ground for comparing these platforms based on the number of processor cores in each platform.

We found out that MS Hyper-V and VMware Server allow only upto 4 cores per processor whereas Xen Server allows allocation of 8 cores per virtual machine. So, we took 4 cores as the standard across all platforms. But we also tested for 8 cores with the Xen Server, so that we get to know the difference in performance.”

My understanding of the benchmark results are that VMware actually performed the tests the fastest, but the graphics rendering and fps results were actually better in VMs on Hyper-V and XenServer. In my opinion, the biggest take away was that 8 vCPU VMs on XenServer outperformed all others which, although not too surprising, unfortunately only serves as a big distraction from the standard methodology needed to be maintained for a fair comparison.

It is also disappointing to me that PCQuest refers to “VMware Server” throughout most of the report instead of ESX/ESXi 3.5. Although the report starts off by saying the comparison was made with VMware Infrastructure 3.5, the repeated references to the free, hosted virtualization product from VMware makes me question what hypervisor was really used during these tests. VMware Server 2.0 also supports 64 bit guests and uses a (web) client to configure VMs, so reviewing the build process really does not provide a clear answer to which platform was tested.

Since PCQuest does not allow for a discussion on their site feel free to comment here about your take on this report.

Hyper9 was announced publically back in September, and then caught the attention of many VMworld 2008 attendees. Scott Lowe, Edward Haletky, Eric Siebert, and Keith Ward are some of the many that have already written about their impressions of this tool’s capabilities. In general, Hyper9 is a search based management tool that promises to “rock your world” of virtual infrastructure management. Hyper9 can be used to query for information at the hardware, hypervisor, and VM operating system levels.

So, what exactly is Hyper9 and what’s so special about a tool that can search against all objects in your virtual infrastructure?

Here’s a screen shot showing the Hyper9 results of a query for “MySQL”.

And a screen shot showing the expanded results of a search for “Windows”

As you can see, Hyper9 provides some powerful reporting, monitoring, and analysis capabilities. You also have the option to save your searches for reuse again and again. Not only can you use keyword based queries, but there are pre-built criteria for all VI object fields. For example, you can find all VMs based on a version of the VM Tools, whether the CDrom is connected, or the existence of a snapshot.

As if that wasn’t enough, Hyper9 also has the ability to compare objects – both for current state analysis and for understanding changes over time. The comparison can take place using the same VM or 2 different VMs. Hyper9 calls this feature VMDNA, and it’s a great feature for examining configuration history.

The following screen shot shows how VMDNA can be used against a single machine to find differences over a time period. (click on any of the screen shots for larger versions)

By clicking on the VMDNA results you are able to drill down into a specific area to further investigate as illustrated in the next screen shot.

Using VMDNA against unique VMs is also useful. The next 2 screen shots show how a pre production version of a VM can be compared to production version.

If you are curious about what server requirements are needed to run Hyper9, I was told:

• Windows 2003 R2 (not sure if this is a hard requirement or just what the demo was running)
• The application installs and embedded Derby database, and there is an option to use PostSQL.
• 1-2 GB of available disk space for install with considerable room for growth of database.
• Installation can be on either a physical or virtual server
• Internet Explorer or Firefox web browser for browsing to the Hyper9 interface

Hyper9 does not require the installation of the VI Client on the system you are browsing from.

A lot will be happening over the next 2 weeks at Hyper9.com. There are plans to open a community site for users as well as opening new slots for beta version testers. As I mentioned at the beginning of this post, I will have some of these beta slots reserved for VM /ETC readers. Leave a comment with your email address (will not be published) if you are interested in getting your hands on one of the betas.

For more information about the Hyper9 product visit the product overview web page.

update 12.15.08 – the ESXi 3.5 Update 3 RCLI API was unintentionally opened. The remote administration abilities will be locked down again in the next release. Please see my updated post on this new information.

In case you’ve been “holed up” in the data center for the last month (I don’t doubt that a few admins have) and maybe haven’t heard yet, VMware now offers a free version of ESXi. Yes, VMware has made a version of it’s enterprise class, bare metal installable hypervisor available for download at no charge. There is no trial period or demo license required. There are no guest limitations on the host. So, there has to be a “gotcha”, right? Not really, but you do need to understand the differences between the free ESXi and VI ESXi (licensed) versions.

Although ESXi runs the same code and is on the same distribution schedule as ESX, VMware’s intended usage is a little different than what VMware administrators are used to, and therefore the ESXi product has some functional differences that need to be understood – especially for the free ESXi version. This post explores some of the major differences, and can hopefully help VI administrators make the correct decisions about what version of ESX to implement. I concentrate most on the differences between the free and licensed versions of ESXi as they are the most likely versions to confuse administrators used to VMware’s full ESX 3.5 enterprise solution.

ESXi is a stand alone appliance

Both in the way ESXi is installed and in the way it is updated or patched, ESXi is more like firmware than it is like a operating system. The installation routine does not require normal configuration decisions. For example, the install does not prompt for partition sizes. Once installed, ESXi does not provide root access to a shell like the Service Console available in ESX. In fact, ESXi is VMware’s hypervisor code only and no longer has any remnants of the Red Hat distribution. VMware designed the ESXi host to be managed individually with the VI Client. Finally, patches and upgrades will be full image replacements.

So, what would be some good uses of ESXi? In the VMware Communities Roundtable Podcast #11(linked at the end of this post) the following scenarios were suggested as appropriate:

• • Companies starting out with virtualization
  • IT departments with little Linux experience
  • Small office or department deployments
  • Remote site deployments
  • Creating simple test and development environments
  • Small consolidation scenarios
  • Quick DR site implementations to get most of the most important VMs up and running
  • enterprise companies that want stateless servers with small footprints that can be quickly deployed

Also, if ESXi is a stand alone host only, then why use ESXi instead of the also free VMware Server?

• • ESXi is a bare metal install and VMware Server requires a host operating system.
  • There is more resource overhead used by the host operating system when using VMware Server.
  • ESXi is an Enterprise class hypervisor that can run more guests
  • ESXi can be directly upgraded to VI3.5 Enterprise when ready

Remote Administration and Monitoring

Remember, the key difference between ESX 3.5 and ESXi is the presence of the Service Console OS. Without the Service Console the remote administration and monitoring possibilities have changed significantly.

free ESXi

• Remote administration only possible with the VI Client
• Although the Remote Command Line (RCLI) appliance is available, the free version’s RCLI access is read only
  • Can not change things – Can’t create new vSwitches or configure existing vSwitches for example, just read current config
  • getting a read and write RCLI is an upgrade to VI Foundation, VI Standard, or VI Enterprise.
• Can NOT use SSH / WinSCP / mRemote – there is not a Service Console. ( it is possible to open a busybox console as explained here and here, but this function was intended for VMware Technical Support only. If you listen to the VMTN Podcast mentioned earlier it is very clear that VMware does not support this access by it’s customers and is also seriously considering removing this function completely.)
• can NOT use SNMP monitoring

updated 12.08.08- As of ESXi Update 3 the RCLI is no longer read only. You can read and write changes thus making remote administration possible. It is also possible to use the normal ESXi API for administration, and several new gui based tools are beginning to surface with ESXi management ability.

Correction: Even with the free ESXi version CIM agents are provided and SNMP monitoring is possible

Updated 12.15.08 – The RCLI for the free version of ESXi 3.5 will be locked back down again in the next release

VI ESXi (in VI Foundation, VI Standard, or VI Enterprise)

• Licensed versions of ESXi also lack Service Console and remote SSH ability. (see updated note above for same alternatives)
• Can be managed by VirtualCenter Server as well as VI Client
• RCLI is read and write
• SNMP monitoring available wih built in CIM agents

Scripting of Installation and Configuration

The current scripting available in the full ESX 3.5 is possible because of the Service Console so we will concentrate on the other versions again.

• Web Access does not exist for ESXi
• Kickstart based automated installation is NOT available
• Read only RCLI prevents scripting in free version but licensed VI ESXi can use new RCLI commands to provide scripting
  • VMware recommends current scripts be ported to new RCLI syntax
• RCLI does have a built in command to export host configuration and import to other hosts

updated 12.08.08 – Mike Dipetrillo from VMware pointed out that you can PXE boot and even Kickstart ESXi . installs. See his post on this here: http://www.mikedipetrillo.com/mikedvirtualization/2008/11/howto-pxe-boot.html

As of Update 3 the RCLI is no longer read only so scripting not prevented anymore. see notes above

Correction: The RCLI command to export host configurations apparently is still a future feature. Look for it in upcoming versions.

Updated 12.15.08 – The RCLI for the free version of ESXi 3.5 will be locked back down again in the next release

Backup and Recovery

Since both versions of ESXi lack the Service Console the backup possibilities are limited. The following are some suggestions:

• Use the VI Client Datastore browser to move copies of files
• Use hardware mirror on local VMFS partition
• Use a IP storage network such as a NAS filer to run VMs and leverage filer features for backup
• Use backup agents within guest operating systems

updated 12.08.08 – Actually, the licensed version comes with VCB (in Foundation, Standard, and Enterprise). That’s the best way for doing backups. 3rd party products from Vizioncore and others also work with the free and VI versions.

Licensing

As previously mentioned, managing ESXI with VirtualCenter and obtaining the VI Enterprise features requires purchasing licenses. The major point to understand here is that there is not a a la carte pricing model for features like VMotion, DRS, HA, or VCB. The best upgrade path for these enhancements with ESXi requires a purchase of the same versions VMware has been selling for ESX 3.5. Specifically, VI Foundation, VI Standard, and VI Enterprise. Unless I misunderstand, there does not seem to be a discount for using ESXi instead of ESX 3.5. There is an advantage of easier administration, tightened security, and an overall smaller footprint. The following VMware slide is a great visual of the differences between the versions and gives general list pricing as well. Click it for a larger version.

I used the following 2 primary resources to put this post together. Check out both for more information about the different versions and options of ESXi.

• Charu Chaubal from VMware’s Technical Marketing Group answered questions in the VMware Community Roundtable podcast #11 – ESXi is where it’s at: Community Roundtable podcast #11
• VMware KB article showing comparison between ESX and ESXi – VMware ESX and ESXi Comparison (1006543)

“VMware is obviously the most experienced company when it comes to delivering a virtualisation product. The company has 10 years of virtualisation experience and a huge customer base, including 100% of the Fortune 500 companies and 92% of the Fortune 1000, totaling over 100,000 customers worldwide. VMware also holds 11 virtualisation patents, and in 2007 their revenue hit the $US1.33 billion mark.

This is in comparison to Microsoft which has a new virtualisation product, little enterprise virtualisation experience, and, to date, no Fortune 500 customers who have adopted their enterprise virtualisation product in a production environment. Ask yourself, which of these companies sounds more qualified to deliver your enterprise virtualisation solution?”

David’s article goes on to make excellent comparison points about

support, reliability, 64 bit hardware requirements, Quick Migration vs VMotion, DRS vs NLB, and various other features not matched by Microsoft such as Storage VMotion.

However, reports on the Internet have touted Hyper-V’s price as the ultimate undoing of VMware. David’s article has some interesting costs analysis too.

“To get Microsoft’s Hyper-V, all you need to do is to buy an edition of Windows Server 2008. The cost for that is between $US999 and $US3999. If you use Windows Server 2008 Enterprise edition, at a cost of close to $US4000, you can run up to 4 guests inside of it without
having to buy additional Windows licenses.

However, if you go with Windows Server 2008 standard you will have to buy a license for the host system, as well as each guest OS. And does this include support? No. Are the features the same between ESXi and Hyper-V? No. Suddenly, Hyper-V isn’t as free as it sounded nor as much
of a value.”

David does a much better job explaining the real pricing of Hyper-V then I did in my “Hyper-V costs $28. Yeah Right!” post.

Don’t get me wrong. I have established my career on Microsoft products and certifications. I want Microsoft Hyper-V to become a solid option for consolidating systems as virtual guests. I believe in the multi virtualization vendor data center. It’s just frustrating to hear the market analysts and industry journalists talk about Hyper-V as the answer to VMware VI3. Once again, David sums up my thoughts:

“In the end, it is my opinion that VMware “wins the war” for several reasons. Perhaps most obvious, is that Microsoft is already incredibly behind VMware in terms of virtualisation know-how and may never catch up.”

John announces the new series and the objective of the Roundtable podcasts with the following summary:

“Each week, we’ll bring together experts and leaders from the VMware Communities and virtualization blogs to discuss the interesting topics in virtualization. Think of this as if it were a group meeting up at VMworld over a pint to chat about the latest news.”

The episode lasts somewhere between 50 minutes to an hour and is a recorded call between John and an attendee list consisting of some of the virtualization community’s top minds from all over the world. VMware Community profiles of the individuals contributing to episode 1 are:

• Steve Beaver – sbeaver
• Tom Howarth
• Alex Mittell – mittell
• Eric Siebert – esiebert7625
• Edward Haletky -Texiwill
• Dave Mishchenko

Go to John’s VMTN post to listen or download the podcast, but the following is my quick summary and take-aways from the call.

I just posted about this new free iSCSI SAN for the local VMFS storage. Alex and Tom discuss the product in greater detail and provide some road map for future versions and expanding the features.

ESX snapshots

A great exchange between all attendees about the lack of true understanding of what snapshots are, how they should be managed, and a few example scenarios where VI admins have made some terrible mistakes deleting snapshots.

Eric disagrees with the idea of using VMware Converter as an alternative to committing snapshots due to the overhead of creating an entire new VM and the increased likelihood of corruption due to the use of the network while using Converter. He suggests that using the vmkfstools utility from the console is the best option. I’ll have to research this some more and maybe create another post on this method.

On a related note, the concept of creating a snapshot and then using vmkfstools to copy the frozen vmdk as a method for cloning running VMs is brought up. Another simple but brilliant idea!

ESX VMs in the DMZ

The concept of hosting VMs in the DMZ is explored, and Edward explains how many security professionals are wrongly requiring that the separate ESX and SAN infrastructure be dedicated to the DMZ. The concept of an ESX host straddling the firewall is actually a misunderstanding by security and networking engineers. Edward dives deep into the increased potential for attacks and Service Console compromise by placing ESX hosts in the DMZ.

Virtualization features comparisons

Dave and the group discuss the various web links and posts that discuss emerging virtualization alternatives such as Hyper-V, Citrix XenServer, KVM, Parallels and the attempts to compare the products to VMware’s features and architecture. A point is made about the competition reducing comparisons down to feature checklists is not really talking “apples to apples”.

Personally, I used to run VMware Server 1.x on my XP notebook until I was tempted to try VMware’s first beta of version 2.0. Although not an officially supported OS for any version of VMware Server, my use of Server on XP was for basic research and test purposes, and I chose the free version over VMware Workstation. Like most, after the switch to 2.0 beta 1 I experienced frustration with the web interface. Now, since I discovered Sun’s xVM VirtualBox 1.6 seamless feature and the ability to run virtual machines created in VMware’s .vmdk format, I have switched. I am extremely happy with VirtualBox, and I even consider it to be a closer open-source replacement for VMware Workstation with features that rival the still in beta version of Workstation 6.5. I also use VirtualBox 1.6 on both 64 bit and 32 bit Ubuntu at home.

Jason’s comparison focuses on using the products in a true virtualization host capacity, and he provides some interesting performance analysis.

On xVM VirtualBox 1.6:

“Overall we found VirtualBox xVM performance to be excellent, especially when loaded on a 64-bit machine with AMD or Intel virtualization acceleration enabled. However, even software-based virtualization on 32-bit systems running 32-bit virtualized OSes ran acceptably. While the software could run on as little on a system as little as 1GB of RAM running a single 512MB VM, we would recommend a minimum of 2GB on the host machine for predictable performance using 512MB or 1024MB VMs. Like VMWare, VirtualBox comes with its own set of integration tools which can be installed on the guest OSes to improve performance. VirtualBox also supports Virtual Machine formats of competitive products, such as Microsoft’s Virtual Server and VMWare. Be advised that if you do port one of your older VMs over, you’ll want to remove the “tools” (paravirtualized device drivers) from the guest OS install before running them in VirtualBox, or you will run into a number of compatibility problems.”

On VMware Server 2.0 Beta 2:

“Server 2.0 beta 2 weighs in at a mammoth 450MB download, and requires a fairly serious box to run with good performance. For our tests, we used a dual-core Athlon 64 X2 5000 ASUS mainboard-based PC clone with 2GB RAM, and an IBM xSeries 1U server with twin dual Xeon 3.6gGhz processors with 4GB of RAM, using the 64-bit version of CentOS 5 (a free identical clone of RHEL 5) as the host operating system. VMWare Server 2.0 also comes in a Windows version, but this version was not tested due to time constraints, and the web UI is identical regardless of what client is used. Performance on the 4GB system was excellent, and quite fast when we doubled the host memory to 8GB, but we started to see some scalability and response issues on the 2GB machine with 1024MB VMs. Additionally, we found that the only reliable browsers to use as the administrative client were the beta and release candidate versions of Firefox 3 — Firefox 2 and Internet Explorer 7 seemed a bit glitchy with all the Ajax components of the new Web UI.”

For some great screen shots of both products in action and to read all of Jason’s comparison use the following links:

Screenshots of VirtualBox 1.6 and VMWare 2.0 Server

Virtualization smackdown: Sun xVM VirtualBox 1.6 vs. VMWare Server 2.0 Beta 2 | Between the Lines | ZDNet.com