• Increase Simultaneous VMotions as well as Increase Performance
• Guest blog entry: VMotion performance » broche.net – VMware …
• VMware Communities: ESX Tips: Increase number of VMotions per host
• Increase number of VMotions per host

One possible scenario for changing this setting would be to temporarily increase VMotions allowed in order to evacuate ESX hosts within a short maintenance window. I prefer to leave the setting at the default, so for this scenario be sure to change it back after the maintenance is complete. if you read the links provided above, others suggest they have changed the settings permanently.

This rest of this post contains a cut and paste of the steps necessary to make the configuration change with a brief explanation about setting the appropriate value. I am pasting from a VMware Partner PDF communication assembled by Michael White, VMware engineer.

“How can I increase the number of simultaneous VMotion activities?

It is possible to increase the number of simultaneous VMotions but it is something that you should be careful of. Do not due it lightly or on a regular basis but in fact use it on a project basis when required.

While it is not documented by VMware officially, it is in fact supported. Really. You only need to change a variable in the vpxd.cfg file and restart VC. Use the steps below:

1. Connect to your vCenter server.
2. Locate the vapid.cfg file – it is normally in C:\Documents and Settings\All Users\Application Data\VMware\VMware Virtual Center.
3. Make a back up copy of vapid.cfg.
4. Edit the file and insert the information below between the and sections. X
5. The value of X is what you need to calculate carefully.
6. A cold migration has a cost of 1, and a Hot Migration (VMotion) has a cost of 4. So a value of 16 for X would mean you could do 4 simultaneous VMotions (4 * 4 = 16).
7. Now save your changes.
8. Restart the VMware VirtualCenter Server service to make the changes live.”

It may be obvious, but I’ll point out the above setting also means you could do 16 simultaneous cold migrations (VMs powered off): 16 * 1 = 16

The following was taken from Eric’s post Don’t know much about resource pools:

“Now, let’s say that one of your fellow administrators wants to use SCVMM to manage this VMware environment.  He uses the SCVMM console to migrate a QA virtual machine from one ESX host to another in order to perform maintenance.

The VMotion completes without incident and everything seems OK…

Before too long, a QA engineer is on the phone asking what happened to his VM.  From his perspective, the VM has disappeared.

It’s not really gone, it has just been moved out of the resource pool.  Fortunately, you can log in with the VI Client and fix this problem by moving the VM back to the resource pool.”

I must first admit I have not done this personally, but I assume 2 things are happening here.

In my opinion, Eric points out an inconvenience that needs to be understood for companies that are considering SCVMM as a “single pane of glass” tool to administrate a multi vendor virtual data center, but not a “show stopper”. Afterall, this is a scenario where a manual VMotion was used. I would prefer to have DRS manage migrations automatically for the majority of the time.

Don’t get me wrong, it needs to be fixed. I’ve listened to Microsoft claim that one of SCVMM’s aspirations is to have every feature available via VirtualCenter. Microsoft, hopefully you are now aware that SCVMM needs to be updated to include the Resource Pool membership attributes of VirtualCenter managed VMotions.

When the green circles do not appear, the Maps view helps troubleshoot what is wrong because you can see exactly which pieces are not connected correctly in the diagram.

What I did not realize, and Eric’s post points out, is that 

there are different shades of green that appear on the Maps tab.

“If one of your ESX servers is under a heavy load it’ll show a light green colour. On the other hand, when one of your ESX server is not CPU intensive at all, it will show up dark green.”

Now that I know this, recognizing what hosts have a darker green circle from the Maps tab is a quick and convenient way to pick a host that has the best available resources for a manual VM migration.

“has VMotion evolved at all to the point where differences between Intel chips are less of an issue than they were in June when you blogged this?? “.

It always amazes me when the virtualization blogisphere seems to be talking about the same topics, and no sooner did I finish making my comment to Tom I found the “What is Enhanced vMotion Compatibility anyway?” post from the VMGuy.

So, this post expands on VMGuy’s example using the Matrix from my June post to show how VMware’s EVC can help with previously incompatible CPUs. You decide if this is an evolution or not. Personally, I’d prefer a cluster of hosts that use like, modern hardware which all take full advantage of the latest CPU advancements. Maybe EVC is acceptable in a lab scenario, but I want all “the bells and whistles” of my processors in production.

First, VMware KB Article ID:1003212 titled Enhanced VMotion Compatibility (EVC) processor support explains how EVC works:

Enhanced VMotion Compatibility (EVC) simplifies VMotion compatibility issues across CPU generations. EVC automatically configures server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. After EVC is enabled for a cluster in the VirtualCenter inventory, all hosts in that cluster are configured to present identical CPU features and ensure CPU compatibility for VMotion. The features presented by each host are determined by selecting a predefined EVC baseline.

The article also explains EVC’s requirements:

To use the EVC feature, you must be running ESX Server 3.5 Update 2 or higher with Virtual Center 2.5 Update 2 or higher and have only compatible processors that are listed in one of the following tables. EVC does not allow for migration with VMotion between Intel and AMD processors.

Go to the KB article to see the tables of compatible processors.

To enable EVC you have to first shut down all of your VMs in the cluster. EVC is then enabled in the properties of the ESX cluster from VirtualCenter just like HA and DRS.

In VMGuy’s post he uses the following example:

“Let’s say that the 3 hosts have Intel E5310 processors in them.  You start to load up your VM’s on that cluster over the next 6 months.  When the cluster is full you go back to your OEM vendor of choice, only to find that they no longer make that server model with a E5310 processor.  The same model from the OEM now contains a Intel E5420 CPU.”

Looking at the matrix from my post in June (the following image) you can see that the Intel 5300 series and the Intel 5400 series are not compatible. Click the image for a larger view.

With EVC enabled you can use VMware’s “magic” to find the baseline compatibility between these CPU families and be sure that your VMs use the common denominator of instruction sets across all ESX hosts.

In summary, if the CPU families are found in the compatibility tables in KB 1003212, EVC will keep the applications on the VMs from causing blue screens because a uniform set of instructions will be available on each ESX host.

Be sure to read the KB article and VMGuy’s post in full.

Download the full paper at the link above, but the following image shows the CPU compatibility matrix for quick reference. Click on the image for a larger version.

Some other links with useful information about VMotion compatibility:

• VMware Communities

• • A thread from April 08 lists 3 links from VMware followed by a lot of deep dive info. It also has a ink to a December 2007 version of Dell’s compatibility matrix as well as a link to HP’s VMotion compatibility document.

• Popular virtualization blogs have posted about this topic. Here’s a few from the last year:

• • Mike D posted about migrating from Intel to AMD,
  • VMwarewolf’s post Are your servers VMotion Capable?
  • Override VMotion compatibility restrictions for CPU mismatch at NTPRO.NL

• Here at VM /ETC

• • I posted the IBM VMotion Compatibility Guide back in November 07. I’ll have to see if there is an updated version now available!

Technorati Tags: virtualization, vmotion, vmware, dell

For example, the post explores several options:

”

• Option A: is to literally just stretch the storage fabric and have the storage at one side – but man, you better have insane connectivity between sites, and this is at severe risk to “smoking hole” site failure – say bye-bye to your array.
• Option B: is to use something like EMC Invista, Yotta Yotta or the like, and have a distributed synchronous LUN with two storage arrays.
• Option C: some vendors (Lefthand Networks comes to mind) have a neat trick where they are in essence doing software RAID across x86 platforms on which they run software – so the RAID mirror node can be remote, and “take over” if you lose a site.
• TIP: do the math. Regardless of which method is used, and even if you use compression – it’s a huge amount of bandwidth. My favorite analogy is to ask a customer to stick in a USB flash drive into their laptop, copy a big file and looking at the throughput. “

Chad goes on to point out that he feels stretching ESX Clusters is a bad idea in general and lists 4 solid reasons to support why. Check out all the whole post at the link above.

BTW, Chad revealed that he is the originator of the ESX home lab hardware shopping list I posted about back in March, and you can also see his original post titled Building a Home VMware Infrastructure Lab at his Virtual Geek blog as well.

First, what is a fail over scenario? The first thing that comes to mind is a problem. One or more of your ESX hosts unexpectedly crashed. This is considered unplanned downtime. Another fail over scenario to consider is planned downtime such as rebooting after applying ESX patches. For both of these types of scenarios you want to make sure your VMs stay online.

VMware’s solution for planned downtime is VMotion. The solution for unplanned downtime is the HA feature of ESX Clusters. When determining your ESX capacity be sure to allow room to leverage these features.

VMotion migrates a VM to a different ESX host without users losing connectivity. Evacuating an ESX server by VMotion enables you

I’ve heard a lot of companies adopting an N + 1 mentality recently. This is typically based on a consolidation estimate’s results. If the capacity analysis study determined that 4 ESX hosts are needed then include an extra ESX host just for added capacity and fail over resources. That’s a good start, but it might not be enough.

Remember that a capacity analysis project is centered around the “WOW! factor” of demonstrating how many servers can be consolidated on ESX hosts. The ESX host utilization is often targeted to be 80% or higher – meaning host as many VMs as possible until the host’s cpu and ram is 80% in use. Using the N + 1 strategy that means that losing one host will work fine. 80% load spread over 4 hosts is maintained by having the 5th ESX host. What if you need to allow for 2 ESX host failures? In that case there is a problem. Here’s some math to demonstrate why:

• original design was 4 ESX servers at 80% = 320%, but if you lose 2 ESX hosts, even with the N + 1 server, 3 hosts will be running at over 106% utilization.

That may be acceptable for a short period of time, but an Administrator should be ready to add another ESX server ASAP to maintain normal performance levels. It gets worse when you lose more ESX hosts simultaneously. A consolidation scenario where ESX hosts are only 65% utilized helps the numbers, but it also means more hosts.

The reality of the IT budget and the reliability of modern hardware keep most companies from planning for more than 1 ESX host failure. Maybe that’s why the N + 1 strategy is so popular? If this is your case you maybe able to still plan for more than 1 host failure by categorizing your VMs as critical and non critical. Create 2 separate ESX Clusters. Enable DRS and HA on the Cluster with the critical VMs, and only enable DRS on the non critical cluster. The N + 1 design already gives you enough capacity for planned downtime (by rotating maintenance on one ESX at a time), and by limiting the number of VMs that HA will restart, you might be able to maintain critical VM availability through multiple unplanned host failures. You can achieve the same design with a single cluster by setting the non critical VMs fail over behavior to not restart in the properties of the Cluster.

Monitoring your ESX Clusters is critical to be sure to maintain the design’s expected availability. Obviously, As your number of VMs increases there will quickly come a point where the number of ESX hosts will need to increase.

For more information about HA check out VMware’s HA page. For more information on VMotion then check out VMware’s VMotion page.

Well, it’s not really about how many NICs ESX needs. I’m not recommending it, but the fact of the matter is that VI3 really only needs 1 NIC per ESX host. It’s just smarter for a company to build some redundancy and load balancing into their VI design. So, let’s say then that ESX just needs 2 NICs minimum.

The real question is “How many NICs does your network infrastructure and VI performance require?” Do you have or will you have:

• a Management network?
• a Backup network?
• a DMZ?
• a dedicated VMotion network?
• an iSCSI network?
• LAN/WAN network – obviously! But maybe multiple LANs?

There’s probably more I can’t think of right now. Aside from the LAN/WAN network, none of the above are required. Some are best practices like the dedicated Management and iSCSI networks. All are possible, and if you will have VMs connecting to all and all have a separate core network switch then you will probably need 2 NICs for each network in the list. (assuming redundancy!)

Of course you can use trunking and VLANs to reduce the number of NICs! You could also group your ESX servers by the the type of VMs hosted. For example create separate ESX clusters for the DMZ VMs or for the different LAN subnets (if using different switches).

Most of the time ESX servers have 4 to 6 NICs. This is a great start and is enough NICs to be flexible in most designs. Understand though that unless you really analyze your current networking and determine your future design you are just guessing!

So, I’m asking you. “How many NICs does YOUR ESX server need?”