VM /ETC

Mike,

Thanks for the clarifications, and I changed your URL in the body of the post. Did it recently change?

Most importantly, thanks for the great VMworld presentation!

Mike, I agree with you in the 2nd issue, “scaling out with multiple single CPU VMs allows for more flexible VM scheduling and also removes the overhead in a multi processor Guest OS and App stack”.
For examnple , if we have a host with two dual core processors, one thing I observe (both in Windows and Linux) is that if I put a single vCPU in a VM the VM’s performance increases. That’s OK. In the performance tab in VIC, we can see the VM reaching nearly 100% of CPU utilization when we stress/test it. And it takes less time to do things cpu-intensive like compilations, etc..

But at the same time if we look at the performance tab of the HOST we can see that tfor a 100% of VM cpu utilization the host’s graph only reaches 25%!!!
That is because the VM is stressing only one core (2 processors * 2 cores = 4. So 1/4 = 25%) ?????

Can we achieve more utilization?

PS1: I’ve checked the affinity of the VM and It can be ran in all the cores.

PS2: It is true also that with 1vCPU, the %RDY parameter (esxtop) decreases wich is good. What is the good value for %RDY? Less than 5%?

Thanks for your time!
J

Good questions, J. vCPUs (virtual CPUs) get directly mapped and scheduled onto lCPUS (logical CPUs – aka cores or hyperthreads). Since you only have 1 vCPU in the guest you’re only going to use 1 lCPU or core on the host. We don’t split threads – no x86 virtualization solution does. If you had more guests then host utilization would increase. Of course if that same VM had 2 vCPUs then you would see 50% utilization on the host since you’d be using 2 lCPUs at the time. Actually, you’d see a little more than just 25% or 50% both times because you have the Service Console of ESX which gets scheduled on one of the cores (unless you’re using ESXi), the vmkernel (the traffic cop) gets scheduled on one of the CPUs, and the invisible helper world (the I/O path) for that VM gets scheduled on a CPU. So in theory if you only had 1 VM running on the host and it had 1 vCPU then it would use 1 core, the vmkernel would use 1 core, the Service Console would use 1 core, and the helper world would use 1 core. All 4 cores would be getting exercised with just 1 VM running with just 1 vCPU. A lot of people don’t realize this. This brings us to %RDY.

%RDY (READY spelled out) means the percentage of time that a VM is ready to run but can’t run because it can’t get onto a CPU. Basically the VM was next up in line to get scheduled but all of the lCPUs were busy. The higher the %RDY the more contention you have on the host. There are 2 things you could do to reduce %RDY:

1) Reduce the number of vCPUs in your guests.I see the case all the time where people create 2 vCPU guests by default whether the app needs it or not. This is because they had 2 CPU physical hosts so they’re just following what they did in the physical world. The problem is 2 vCPU guests take up twice as many lCPUs when scheduled. The more lCPUs in use with more VMs means someone has to wait and %RDY goes up. This is just fine most of the time but I generally say if you see %RDY higher than 50% then you should look at your VMs and determine if they really need that other CPU. If you decide that you can knock down the vCPU count in a VM make sure to change the HAL in Windows or kernel in Linux to match the single CPU VM. Going from multi-proc HAL/kernel to single-proc HAL/kernel is easier (and supported) in the Linux world than the Windows world.

One clue that the VM doesn’t really need the second vCPU is to look at %WAIT. If %WAIT is higher then 30% there’s a good chance the second CPU is just sitting idle when the VM gets scheduled.

2) The other thing you can do is buy another physical host to increase the core count in your cluster.

NOTE: The recommended percentages I’m giving you for %RDY and %WAIT are my personal recommendations just based on seeing this stuff for 6 1/2 years at various customers. You may feel comfortable with lower limits. I know of one customer that won’t go over 30% for %RDY or 10% for %WAIT. I’m just telling you what to look for. With some practice tuning in your environment with your workloads you’ll be able to find out the exact threshold you need.

I hope this helps.

Interesting recommendation for %ready times. I’ve always heard / read that double digit ready time was bad (so >10%). As always, it’s relative to the application on each VM and the user experience impact. More on this topic at http://communities.vmware.com/docs/DOC-7390.

Rich,

It does all come down to comfort level and number of VMs. Obviously if you have 40 VMs on a host and 50% of the time they’re getting blocked from running you have a problem. Of course if those 40 VMs really weren’t doing anything all the time (maybe you have 40 idle print servers or DHCP servers or something) then 50% ready really isn’t a big deal. I too used to say anything over 5% ready is bad. I’ve slowly increased it over time to the 50% threshold. Again, these are my recommendations – not VMware’s. You’ll have to find something that you’re comfortable and as always LOOK and UNDERSTAND your applications. I’ve seen too many customers that have no clue what they’re app is doing or is supposed to be doing and then they want to start troubleshooting at the virtualization layer. Bad idea.

Thanks Mike and Rich,vary very interesting.
It was exactly what I was supposing. I’m seeing 20% RDY (average) in my hosts (sometimes 45%…). Most of the VMs are configured with 2vCPU and are VMs with little load.
The extrange thing is that I’m seeing 400 or 500 or 600 as values for the %WAIT parameter!!
Is it a bug? I’ve to read this like 40 or 50 or 60% values? May be I ‘ve to read the %TWAIT or %BWAIT calues insted the %WAIT value?
Or may be it’s correct.
This seems to happen both in ESX 3.0.2 and ESX 3.5.

TIA,
J

J, if you want you can email me off-line to figure out what’s going on: http://www.mikedipetrillo.com/about.html. Where are you getting the values from? Virtual Center? esxtop?