VM /ETC

Part of the magic hosting multiple virtual machines (VMs) on VMware ESX server is accomplished by leaning on the host’s CPUs to simultaneously handle networking loads. The more network I/O generated the more the CPUs have to work. When this happens the performance of the ESX host and the VMs can suffer because the result is limited access to available physical processing. Some common network I/O examples are software iSCSI adapter or NFS access to data stores, live migration of VMs between ESX servers via VMotion, and even administrator access with the VI Client.

Fortunately, ESX/ESXi 3.5 TCP Segmentation Offload, or TSO, can remove some of the networking burden from the host’s CPUs and improve overall performance. When the ESX server’s physical NICs support it, enabling TSO is as simple as choosing the right virtual network adapter, the Enhanced VMxnet adapter, for the VM. Surprisingly, making the Enhanced VMxnet adapter available to the VM is not a straightforward process because the Enhanced VMxnet adapter might not be an option in the virtual NIC properties or the Add New Hardware wizard.

First, you may be wondering how TSO reduces CPU overhead.

Wikipedia has a good explanation:

“When large chunks of data are to be sent over a computer network, they need to be first broken down to smaller segments that can pass through all the network elements like routers and switches between the source and destination computers. This process is referred to as segmentation. Segmentation is often done by the TCP protocol in the host computer. Offloading this work to the network card is called TCP segmentation offload (TSO).”

Not sure if your ESX host’s NICs support TSO?

Since TSO in the VMs requires TSO support in the VMkernel, use the following Service Console command:

#esxcfg-vmknic –l

updated 3.20.09 - The original wording was misleading and confusing in the rest of this section. I have updated the post to be more clear.

Each VMkernel interface should appear on the list with TSO MSS set to 40960

Doing this only confirms the pNICs tied to the vkernel portgroups, but if you have the same pNICs on other vSwitches you’ll know they are supported.  You could also check with your pNIC vendor docs. Offloading cycles off from the CPU to the pNICs is handled by the VMkernel so it makes sense thats where you would have to look.

Note: Although not shown in the example screen shot above, if you are using IP based storage you will also see a second vmk interface.

As of version 3.5 ESX/ESXi has TSO enabled by default, but you still need to turn it on for the VMs on an individual basis.

This process is described in a VMware KB article titled Enabling enhanced vmxnet adapters for Microsoft Windows Server 2003.

To enable the enhanced vmxnet option when configuring a new virtual machine

1. In the New Virtual Machine wizard, on the Choose Guest Operating System page, select the version Microsoft Windows Server 2003, Enterprise Edition (64-bit), even if you will install a different version of the operating system on the virtual machine.
   
2. On the Choose Networks page of the wizard, select the adapter Enhanced vmxnet.
   
3. Proceed through the wizard.
   
4. Install the desired guest operating system on the virtual machine.
   
5. Install VMware Tools on the virtual machine.
   

To enable the enhanced vmxnet option when configuring an existing virtual machine

1. In the VI Client, power off the virtual machine.
   
2. Right-click the virtual machine and select Edit Settings.
   
3. In the Virtual Machine Properties dialog, select the Options tab.
   
4. Click General Options, and change the guest operating system to Microsoft Windows Server 2003, Enterprise Edition (64-bit).
   
5. Click the Hardware tab, and then select the Network Adapter.
   
6. Click Remove.
   
7. Click Add. The Add Hardware wizard appears.
   
8. Select the device type and click Next.
   
9. Select Enhanced vmxnet as the adapter type, and click Next.
   
10. Click Finish.
    
11. Click OK to dismiss the Virtual Machine Properties dialog.
    
12. Power on the virtual machine.
    

The bottom line, and the not so straightforward process I mentioned earlier, is that even if your VM is running Windows 2003 Standard (32 bit), you need to select 2003 Enterprise (64 bit) in the General Options to make the Enhanced VMxnet adapter available in the Add Hardware wizard.

To confirm your VM is now using TSO enter the following at the command prompt

>netsh int ip show offload

The following screen shot shows the TCP Large Send response that indicates TSO is enabled.

Scott Lowe has previously posted about how to enable the enhanced vmxnet adapter without changing the OS type in the General Properties of the VM. Scott’s post explains how to modify the .vmx file.

A VMware KB article titled Choosing a Network Adapter for Your Virtual Machine also explains when, and to some extent why, different virtual network adapter types are available depending on the scenario:

The Choose Networks window makes available only those network adapters that make sense for the virtual machine you are creating. [omitted] Here’s an overview of what Choose Networks might offer you:

• For virtual machines native to VMware Workstation 4x, VMware GSX Server 3, or VMware ESX Server 2.x, you can explicitly choose between Vlance and vmxnet.
• For most 32-bit virtual machines native to VMware Workstation 5 or 6, VMware Server 2, or VMware ESX Server 3, only the Flexible adapter is available.
• For most 64-bit virtual machines and for 32-bit Microsoft Windows Vista virtual machines, only the e1000 adapter is available.
• For certain guest operating systems on VMware ESX Server 3.5 and later, you can choose the Enhanced vmxnet adapter in addition to the Flexible or e1000 adapter mentioned for that guest type in the previous bullets.

The “certain guest operating systems” mentioned above where the enhanced vmxnet adapter are available:

• 32/64-bit versions of Microsoft Windows 2003 (Enterprise and Datacenter Editions). You can use enhanced vmxnet adapters with other versions of the Microsoft Windows 2003 operating system, but a workaround is required to enable the option in the VI Client. See http://kb.vmware.com/kb/1007195.
• 32/64-bit versions Red Hat Enterprise Linux 5.0
• 32/64-bit versions SUSE Linux Enterprise Server 10
• 64-bit versions Red Hat Enterprise Linux 4.0

Related Posts

• Design a clustered VM application that can fully leverage VMotion, DRS, and HA? (7)
• Low Cost Iomega iSCSI and NAS Storage On VMware HCL (3)
• Stuck in a VM – to release the mouse press the host key (8)
• Joint VMware and NetApp Best Practices for Running VI3 on IP Based Storage #TA2784 (7)
• thehypervisor.com reports VMware Workstation faster than Hyper-V (6)
• Mounting a Windows share from ESX (1)
• Create a NFS share for VM ISO files with Windows 2003 Server R2 (11)
• Design Challenges Of Virtualized vCenter With A vNetwork Distributed Switch (6)
• 24 Hour Timeout Prevents Removing ESX vNetwork Distributed Switch (0)
• Simply Automating Virtual Machine IP Addressing For Disaster Recovery Sites (without scripting) (15)