The following details were taken from the VMware VirtualCenter 2.5 Update 5 Release Notes:

What’s New

Support for High Consolidation in VMware HA Clusters – VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines (VMs) per host in an HA cluster.

For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Upgrading or Migrating to VirtualCenter 2.5 Update 5

This release supports upgrading from VirtualCenter 1.4.1, VirtualCenter 2.0.2 (including Update 1, Update 2, Update 3, Update 4, and Update 5), VirtualCenter 2.5, VirtualCenter 2.5 Update 1, VirtualCenter 2.5 Update 2, VirtualCenter 2.5 Update 3, or VirtualCenter 2.5 Update 4, to VirtualCenter 2.5 Update 5. Review the detailed upgrade and migration instructions and guidelines that are provided in the Upgrade Guide.

Following the above link to KB 1012002 explains that upgrading vCenter 2.5 to U5 is just the start. VI 3 admins also need to make some additional configurations on ESX/ESXi hosts to achieve the 80 VMs per host improvements.

“Starting with the VirtualCenter 2.5 Update 5 release, an ESX Server host in an HA cluster can support up to 80 virtual machines. For all virtual machines to power on on other hosts in the cluster, if hosts within the failover capacity limit fail, you need to ensure that the following parameters in the ESX Server hosts are set with the following values:

Parameter	Value	Applicability
Misc.RunningVCpuLimit	192vCPU	ESX Server 3.5 and ESX Server 3i Embedded
Service console memory	512MB	ESX Server 3.5
vim resource pool memory reservation	1024MB	ESX Server 3i Embedded
Host agent memory configuration	<hostdStopMemInMB>380</hostdStopMemInMB> <hostdWarnMemInMB>300</hostdWarnMemInMB>	ESX Server 3.5 and ESX Server 3i Embedded

For specific instructions on setting each of these configurations on ESX/ESXi 3.5 hosts read KB 1012002.

It also appears that this KB offers some interesting tweaks for improving heavy VM consolidation on ESX/ESXi 3.5 hosts even if not managed by VC2.5 U5. Why would it not benefit admins to make these changes on hosts regardless of VI 3.5 Cluster membership?

Although not stated, I assume ESX Clusters with greater than 80 VMs per host should be upgraded to vSphere 4 in order to insure the best performance and scalability features.

For example, I recently helped migrate a company from ESX 2.X to new a installation of VI 3.5. Since the client did not have any additional space available on their fiber channel (FC) SAN for a new VMFS3 volume, we temporarily used a Windows Server 2003 R2 NFS share to host 2 dozen VMs until the existing FC volumes could be rebuilt and reconnected. The customer actually ran their production environment for 2 weeks in this configuration and was experiencing better performance. Newer hardware for the ESX hosts also contributed to this increase, but the point is that the NFS storage was not a bottleneck.

For those that are considering NFS, I was recently forwarded a list of links that provide sound arguments and testimonials on the unique advantages of using NFS with VMware. Although the published date of some of the posts that are referenced might be a bit dated, the content is still valid. Here is the list with quotes from some of the posts, but be sure to read the all in full for more information.

Dave at NetApp says:

“Since a VMDK is a virtual disk, I had assumed that block-based protocols like iSCSI and Fibre Channel would make more sense than NAS, so I asked several customers why they prefer NFS.

The answer is simple: Managing .vmdk files is much easier than managing LUNs. If you have 20 or 30 virtual machines, then VMFS is great for consolidating the VMDKs into a single LUN. But NAS is much easier and more scalable if you have hundreds or thousands of virtual machines.

The big advantage is that you can use all your file management tools. Group the VMDKs for Exchange servers in one folder, SQL servers in a second, virtual desktops in a third, and so on. Instead of backing up LUNs or virtual machines individually, simply backup a directory tree of VMDKs all at once. (This is much less expensive than buying a backup license for each virtual machine, and also easier to manage.) For disaster recovery, you can replicate the data for a whole group of virtual machines as a single unit.”

http://blogs.netapp.com/dave/2007/09/why-run-vmware-.html
—————————————————————————–
Chuck at EMC says:

“I think that, in the long term, we’ll find high-end NAS much more friendly for high-end VMotion / DRS farms than today’s SANs.  And I think that NAS has the potential to offer a few benefits that we might not find in the SAN world.

Not only does NAS deliver a flat name space that will probably make implementing VMotion far easier, you get other potential benefits that might not be obvious:
• You get to manage a file system, rather than a collection of LUNs
• You get some modicum of access control through the file system mechanisms
• You get access to advanced NAS features, like thin provisioning, snaps, replication, etc.”

http://chucksblog.typepad.com/chucks_blog/2006/12/vmware_virtual_.html
—————————————————————————–
Why VMware over Netapp NFS
http://viroptics.blogspot.com/2007/11/why-vmware-over-netapp-nfs.html

“NetApp NFS advantages:

• You get thin provisioning by default with NFS. FC and iSCSI VMDKs are thick. This can save 50% of your disk space.
• Adding NFS datastores are simple. Mount the NFS volume using the GUI and start creating VMs
• Adding additional Netapp filers for datastores requires no down and no cabling changes.
• You can have large datastores that span many disks. 16TB for Netapp.
• You can use A-SIS to de-duplicate your datastores for a 50-80% reduction in disk space
• You can expand AND decrease NFS volumes on the fly
• You can use snapshots of the volumes to restore individual VMs
• You can use snapmirror to backup VMware volumes to a DR site over a WAN
• You don’t have to deal with VMFS or RDMs
• You don’t have to deal with FC switches, zones, lun sizing, HBAs, and identical LUN IDs
• You can restore multiple VMs, individual VMs, or files within VMs.
• You can instantaneously clone (Netapp Flexclone), a single VM, or multiple VMs
• You can also backup whole VMs, or files within VMs using NDMP or any other backup software
• ESX server I/O is small block and extremely random which means that bandwidth matters little
• No single disk I/O queue, so your performance is strictly dependent upon the size of the pipe and the disk array.
• Failover to your SnapMirrored copies can be done in minutes. iSCSi/FC requires LUN resignaturing.
• You can clone a single VM or create 100’s of VMs from a template in seconds”

—————————————————————————–
Why NFS?
http://communities.netapp.com/message/8904

• Deduplication – possible to use deduplicated space savings with LUNs but MUCH more complicated (have to mess with fractional reserve, LUN thin provisioning, etc. — possible to get caught overprovisioning and have real issues)
• VMware Datastore sizing — easy datastore growth (possible with VMFS) and shrinking (not possible with VMFS)
• Larger datastores – no need to keep datastores smaller like with VMFS – up to 16 TB
• Snapshots – can retrieve individual vmdk’s from snapshots and/or mount vmdk’s from snapshots for single file restore
• SMVI – main benefit is ability to do faster VM restores (uses SnapRestore rather than LUN clone so can instantly restore a single VM to any previous snapshot)
• VMDK Thin Provisioning
• Ease of addition – somewhat easier than LUNs/VMFS
• VMFS/RDMs – no need to deal with them
• Single-file FlexClone (future feature) – can clone a vmdk instantly for fast provisioning
• No single disk I/O queue as with iSCSI/FC so performance limitations are purely governed by pipe size and disk array size.
• Faster failover to SnapMirror remote copies (less steps plus faster steps) – no need to do LUN resignaturing
• ESX server I/O is small block and extremely random meaning that bandwidth is less important (i.e. GigE works well).
• Can dump individual VM’s via NDMP
• No FC zoning, switch cost, HBA’s, compatibility matrices, or LUN IDs

—————————————————————————–

You should also check out the following NFS related posts:

Pros and cons of using NAS NFS with VMware
http://searchstorage.techtarget.com/tip/0,289483,sid5_gci1351138,00.html?track=NL-57&ad=695051&asrc=EM_NLT_6173872&uid=8231339

VM File-Level Recovery with NetApp Snapshots
http://blog.scottlowe.org/2007/10/08/vm-file-level-recovery-with-netapp-snapshots/

Full VM Recovery with NetApp Snapshots
http://blog.scottlowe.org/2007/10/08/full-vm-recovery-with-netapp-snapshots/

VMware over NetApp NFS: A Customer’s Testimonial
http://blogs.netapp.com/storage_nuts_n_bolts/2008/01/vmware-over-net.html

Thanks to my Softchoice colleague Charles Tyler for providing this list. Charle’s expertise on NFS and all things NetApp is well known here in the Atlanta area. Follow Tyler on Twitter or watch for his occasional helpful comments on NetApp related blog posts!

Current VMware VI customers with active SnS contracts will automatically receive vSphere 4 licenses per the chart below. (click for a larger view)

VMware has mostly done away with the a la cart features licensing model and has standardized on the 6 editions above – 2 for SMB customers and 4 for Enterprise customers. More about licensing upgrade paths and the costs of the new versions can be found in this whitepaper.

Jason Boche points out some deeper details about the new licensing in his post vSphere licensing notables.

64 bit hardware

If your current ESX hosts are running on 32 bit hardware you will need to purchase 64 bit hardware for vSphere. There is not much else to say about that.

Fault Tolerance

While you are pricing new 64 bit hardware, if you want to take advantage of the Fault Tolerance (FT) feature (zero down time and lockstep fail over to a shadow VM) you will need CPUs with virtualization assist. Eric Sloof provides a table of compatible processors in his post CPU compatibility with VMware Fault Tolerance that is worth checking out.

Upgrades

Be sure to understand the new licensing model before you upgrade. The FlexLM centralized licensing server will be removed and could present some challenges if you plan to co exist between the current ESX version(s) and vCenter/vSphere 4.

For those that are not sure if they need to upgrade or are considering other alternatives, check out Mike D’s post. For a more conservative viewpoint check out Eric Seibert’s post Upgrading to VMware vSphere: Test first, deploy later.

Resources

To be prepared for when vSphere is finally generally available, customers should start exploring publicly available vSphere resources.

If you are a new customer, most of the considerations mentioned above still are appropriate.

So, if running VI 3.5 on VMware Workstation is so popular why do it on VMware Server 2.0?

• VMware Server is free
• VMware Server can run on Linux. You can dual boot your PC/notebook when you need to run VMware Server.
• Even if you use Ubuntu, for example, with the graphical Gnome desktop you should have less overhead on a PC/notebook then a Windows Vista or XP OS and slightly better performance.

Let me know if you try it. I will be making the changes to my set up this weekend.

updated 04.17.09 – I was not clear here originally. I’m not sure if VMs are able to run on this config. I will test and let everyone know.

I found out about the new whitepaper from the virtualization.info post:

Tech: How to run VMware ESX 3.5/i on VMware Server 2.0

With all the recent talk about the software from Rove and VMware that provides mobile device access to managing VMware Infrastructure, administrators now need to consider how they will actually connect mobile phones to vCenter and ESX hosts behind the firewall. Fortunately, a thread from the VMware vCenter Mobile Access Technology Preview Community sheds some insight on how VMware actually tested the vCenter Mobile Access virtual appliance (vCMA) with several different mobile devices.

From The official “how will I connect to this thing” thread:

1. BlackBerry, with a BlackBerry Enterprise Server (BES). This is by far the easiest way, because BlackBerries that are connected to a BES are already on their company’s intranets. Loading the app should be as simple as pointing at the app URL (the format of which will be documented at release).

2. Iphone. We tested two methods successfully. The first was to create an ipsec vpn connection to our intranet. Once we were in things worked flawlessly. The second method worked on every device we tested but is slightly specific to our setup. We used our generic ssl/vpn gateway to authenticate, then we used the web proxying feature of the vpn landing page to get to the url we needed. The resulting app ends up with a vpn “bar” at the top since the page is hosted in this case but functionality is not impaired.

3. Symbian/Nokia. The ssl/vpn solution worked well. There might be vpn solutions out there, check with your network administrator.

4. WinMo. The ssl/vpn solution worked well. We couldn’t get the built-in vpn client to negotiate with our network but it certainly seems possible.

5. Android. The ssl/vpn solution worked well. There might be vpn solutions out there, check with your network administrator.

6. BlackBerry without BES. Can you guess? The ssl/vpn solution worked well. There might be vpn solutions out there, check with your network administrator.

I would guess that the majority of admins will have a Blackberry on a BES (Enterprise Server) like me (scenario 1 above). A quick test to make sure that your Blackberry device can see internal servers in this case would be to open your phone browser and go to the FQDN or ip address of a vCenter or ESX server. You should get the VMware web interface. I was able to do this on my Blackberry 8900 Curve. Now I am wondering why I never thought to do this sooner?!!

VMware announced yesterday that the vCMA is now available for download as a technology preview beta. Rove’s Mobile Admin is available in a 14 day trial.

“Following the lead of analysts and customers, VMware has adopted “cost per application” as a more accurate metric to compare costs between virtualization solutions. Going beyond a simplistic license price comparison, evaluating cost per application takes into consideration the number of virtual machines that can be run on a single server, or the ”virtual machine density” enabled by a specific virtualization solution. The higher the virtual machine density enabled, the higher the server consolidation ratio. The higher the consolidation ratio, the more an organization can reduce its infrastructure costs (including costs for servers, networking. storage, power and datacenter space), and software costs, (including guest operating system and virtualization software licenses). Following this logic, the solution that can provide the higher consolidation of servers – without an impact to performance – also provides the most value to customers.”

I decided to test the calculator with a 50 VM / application example.

Note: I have not verified prices used in the VMware calculator, nor have I deeply analyzed the methodology.I would recommend companies considering virtualization use the results of the calculator as a starting point and adjust accordingly with real figures provided by the representatives and partners of both Microsoft and VMware.

VMware’s calculator requires answering six questions, and the result is a detailed report based primarily on the following formula:

Hardware Cost
(virtualization hosts, management servers, networking, storage)
+
Power and Cooling Cost
+
Datacenter Real Estate Cost
+
Guess Operating System Software Cost
+
Virtualization Software Cost
+
Virtualization Management Software Cost
=
Total Cost of Deployment
/(Divided by)
Number of Application Virtualized
=
Cost per Application

Screen shot of the 6 inputs:

After submitting, a 10 page report (after .pdf conversion) was generated. The Executive Summary section sums up the results:

Based on your inputs, the cost-per-application to virtualize 50 applications using VMware Infrastructure 3 Enterprise Edition is $3,105 — 20% lower than with Microsoft Windows Server 2008 (Hyper-V) and System Center.

The report provides much more detail, but here is a summary table of the total costs:

For more details about the methodology and assumptions of the VMware Cost Per Application Calculator read this white paper.

I would not be surprised if there is a lot of debate between Microsoft and VMware over the methodology and the results.

I hope this this calculator is either expanded in the future or another one is introduced to compare XenServer and VMware VI Enterprise (current and future versions) as well.

Here’s some details from Srinivas’ post about vCMA:

“Did you ever get paged when you are in a meeting about some virtual machine that needed to be restarted?  Or, did you ever get an alert when you are at your kid’s soccer game or at the movies that a particular server is overloaded?  Actually, imagine any scenario where you need to actively manage your datacenter but you are nowhere close to a PC.  Wouldn’t it be nice to act on the notification from your mobile phone?  After all, we are now a generation that doesn’t leave home without a mobile phone…

Introducing VMware vCenter Mobile Access (vCMA).  vCMA allows you to monitor and manage VMware Infrastructure from your mobile phone with an interface that is optimized for such devices. Specifically, it allows you to:

• Search for virtual machines in your data center
• Migrate virtual machines from one host to another using vMotion
• Execute recovery plans using VMware Site Recovery Manager
• Access Scheduled Tasks, Alarms and Events
• And much more…”

This YouTube video shows off the vCMA on a Blackberry.

vCMA is the second mobile phone VMware VI administration product I have covered this week. I also posted about Rove’s Mobile Admin 4.2.

Fortunately, ESX/ESXi 3.5 TCP Segmentation Offload, or TSO, can remove some of the networking burden from the host’s CPUs and improve overall performance. When the ESX server’s physical NICs support it, enabling TSO is as simple as choosing the right virtual network adapter, the Enhanced VMxnet adapter, for the VM. Surprisingly, making the Enhanced VMxnet adapter available to the VM is not a straightforward process because the Enhanced VMxnet adapter might not be an option in the virtual NIC properties or the Add New Hardware wizard.

First, you may be wondering how TSO reduces CPU overhead.

Wikipedia has a good explanation:

“When large chunks of data are to be sent over a computer network, they need to be first broken down to smaller segments that can pass through all the network elements like routers and switches between the source and destination computers. This process is referred to as segmentation. Segmentation is often done by the TCP protocol in the host computer. Offloading this work to the network card is called TCP segmentation offload (TSO).”

Not sure if your ESX host’s NICs support TSO?

Since TSO in the VMs requires TSO support in the VMkernel, use the following Service Console command:

#esxcfg-vmknic –l

updated 3.20.09 - The original wording was misleading and confusing in the rest of this section. I have updated the post to be more clear.

Each VMkernel interface should appear on the list with TSO MSS set to 40960

Doing this only confirms the pNICs tied to the vkernel portgroups, but if you have the same pNICs on other vSwitches you’ll know they are supported.  You could also check with your pNIC vendor docs. Offloading cycles off from the CPU to the pNICs is handled by the VMkernel so it makes sense thats where you would have to look.

Note: Although not shown in the example screen shot above, if you are using IP based storage you will also see a second vmk interface.

As of version 3.5 ESX/ESXi has TSO enabled by default, but you still need to turn it on for the VMs on an individual basis.

This process is described in a VMware KB article titled Enabling enhanced vmxnet adapters for Microsoft Windows Server 2003.

To enable the enhanced vmxnet option when configuring a new virtual machine

1. In the New Virtual Machine wizard, on the Choose Guest Operating System page, select the version Microsoft Windows Server 2003, Enterprise Edition (64-bit), even if you will install a different version of the operating system on the virtual machine.
   
2. On the Choose Networks page of the wizard, select the adapter Enhanced vmxnet.
   
3. Proceed through the wizard.
   
4. Install the desired guest operating system on the virtual machine.
   
5. Install VMware Tools on the virtual machine.
   

To enable the enhanced vmxnet option when configuring an existing virtual machine

1. In the VI Client, power off the virtual machine.
   
2. Right-click the virtual machine and select Edit Settings.
   
3. In the Virtual Machine Properties dialog, select the Options tab.
   
4. Click General Options, and change the guest operating system to Microsoft Windows Server 2003, Enterprise Edition (64-bit).
   
5. Click the Hardware tab, and then select the Network Adapter.
   
6. Click Remove.
   
7. Click Add. The Add Hardware wizard appears.
   
8. Select the device type and click Next.
   
9. Select Enhanced vmxnet as the adapter type, and click Next.
   
10. Click Finish.
    
11. Click OK to dismiss the Virtual Machine Properties dialog.
    
12. Power on the virtual machine.
    

The bottom line, and the not so straightforward process I mentioned earlier, is that even if your VM is running Windows 2003 Standard (32 bit), you need to select 2003 Enterprise (64 bit) in the General Options to make the Enhanced VMxnet adapter available in the Add Hardware wizard.

To confirm your VM is now using TSO enter the following at the command prompt

>netsh int ip show offload

The following screen shot shows the TCP Large Send response that indicates TSO is enabled.

Scott Lowe has previously posted about how to enable the enhanced vmxnet adapter without changing the OS type in the General Properties of the VM. Scott’s post explains how to modify the .vmx file.

A VMware KB article titled Choosing a Network Adapter for Your Virtual Machine also explains when, and to some extent why, different virtual network adapter types are available depending on the scenario:

The Choose Networks window makes available only those network adapters that make sense for the virtual machine you are creating. [omitted] Here’s an overview of what Choose Networks might offer you:

• For virtual machines native to VMware Workstation 4x, VMware GSX Server 3, or VMware ESX Server 2.x, you can explicitly choose between Vlance and vmxnet.
• For most 32-bit virtual machines native to VMware Workstation 5 or 6, VMware Server 2, or VMware ESX Server 3, only the Flexible adapter is available.
• For most 64-bit virtual machines and for 32-bit Microsoft Windows Vista virtual machines, only the e1000 adapter is available.
• For certain guest operating systems on VMware ESX Server 3.5 and later, you can choose the Enhanced vmxnet adapter in addition to the Flexible or e1000 adapter mentioned for that guest type in the previous bullets.

The “certain guest operating systems” mentioned above where the enhanced vmxnet adapter are available:

• 32/64-bit versions of Microsoft Windows 2003 (Enterprise and Datacenter Editions). You can use enhanced vmxnet adapters with other versions of the Microsoft Windows 2003 operating system, but a workaround is required to enable the option in the VI Client. See http://kb.vmware.com/kb/1007195.
• 32/64-bit versions Red Hat Enterprise Linux 5.0
• 32/64-bit versions SUSE Linux Enterprise Server 10
• 64-bit versions Red Hat Enterprise Linux 4.0