VM /ETC

These two statements seem contrary:
- “Do not exceed 768 MB of RAM for Linux guests”
- “Reduce Linux VM swapping however possible”

What happens when my workload uses more than 768 megs of ram, I will swap. If your workload needs more
memory, your host has capacity and you aren’t at
a vmware archtitectural limit, add it.

I can’t wait till hotswap ram comes arrives.

graeme,

Yes, I agree the statements are contrary for an individual VM’s OS. I think the statement is aimed at the sum of all the VMs in the virtual environment. In other words, keep the bulk of your Linux VMs below 768 MB of ram with minimal swap files. Only create VMs with larger memory and swap for applications that truly utilize the extra RAM. Remember that each VM also has an ESX swap file in it’s volume folder too. Assigning excessive amounts of RAM to VMs will increase the overhead ESX needs to do it’s job as well.

Too often administrators are used to the physical server mentality of more ram makes a server better. With VI less is more not just at the ESX host level, but at the VM level too.

OK, so I’m back to being confused. For starters we’re a pretty big unix shop. The only reason I use windows at all is to run the VI client, in fact.

We have several application running on Linux, most notably Oracle, tomcat, and apache.

All three of these are multithreaded and benefit from multiple cpus, so why would making the vms have 2 vcpus be so bad?

Also, in the case of Oracle, more memory is certainly better, why would I reduce it to the bare minimum? If I assign too much memory, it’s not like it’s being locked up just for that machine.

I’m genuinely confused.

Chris,
there is an interesting but a little bit outdated (it’s from 2006) paper from IBM about websphere performance on ESX: http://www-900.ibm.com/cn/crl/download/ESX_WAS_WP_24Jul06_final.pdf
In this paper there is a comparison of running multiple single vcpu vms vs. running smp vms with the same total number of cpus. The result is that single cpu vms perform about 20% better than vms with multiple cpus.

Chris,

Running 2 vCPUS on your VMs is not a bad thing by itself, but isolating your 3 VMs on the same host could be. For example, if your 3 SMP VMs are all hosted on a host with only 2 physical processors you could create a cpu ready time issue as your VMs are ready to execute but can’t get scheduled time to the physical CPUs. That’s a small, simple example.

Check out the VMware Ready Time Observations white paper for more information. http://www.vmware.com/resources/techresources/641

“To achieve best performance in a consolidated environment, you must consider ready time — the time a virtual machine must wait in a ready-to-run state before it can be scheduled on a CPU. This paper provides information to help you understand the factors that influence ready time on an ESX Server 3.0 system.”

I agree that using single vCPU VMs is more efficient, but in a Linux VM, that means that you must load the normal kernel, not the SMP one, doesn’t it?

I mean, the VM configuration (one vCPU) and the configuration of the OS inside the VM MUST match, don’t they?

Thanks for the answer,
J

J,

Great point. Yes they must match. This is most frequently discussed with Windows VMs since the majority of P2V conversions are automated for Windows servers which result as SMP VMs. The HAL must be switched to use a uniprocessor driver. The same is true for Linux VMs, but because there are so few P2V tools that can do a Linux P2V a net new VM is often built and the apps and data are transfered.

Here’s a post on the process for Windows VMs I did earlier this year:
http://vmetc.com/2008/06/11/how-to-p2v-multi-processor-servers-to-uni-processor-vms/

I’m not a Linux expert, but I believe the recommendation concerning RAM usage is about how Linux uses free memory as file system cache. So if you look at the amount of RAM in use in VMtools will include the amount in use as cache. I think they are hinting that you don’t want to waste server RAM real estate on file system caching. But I agree with the statement that if your workload requires it you should configure more to avoid paging.

I found it surprising that Mike didn’t provide the Linux timekeeping KB link that’s actually been useful in my experience:

http://kb.vmware.com/kb/1006427

Just stumbling across the nice set of notes from my presentation at VMworld. It seems there’s some confusion on a couple of points so let me clarify.

1) If you need more than 768 MB of RAM then go ahead and allocate it. Jason Willey’s comment hit the nail on the head. What I meant to say was there are a lot of Linux VMs out there doing simple tasks that were given 1 GB of memory as a default install and they really don’t need it. Once you break the 768 MB threshold in most Linux distros then you end up allocating another chunk of memory for file system cache. Staying under 768 MB if your VM doesn’t need it prevents the extra memory usage for file system cache which means you can get more VMs on a host (generally memory is the limiting factor to VM density).

2) Nearly all apps out there will scale better when you stack multiple single CPU VMs on a host versus fewer multi CPU VMs – even apps like databases that like multiple CPUs. The reason is writing a true, preemptive multi-processing, multi-threaded application that behaves properly with the underlying operating system’s scheduling is very difficult. Whenever you add more CPUs to the mix from a scheduling standpoint whether it be physical or virtual you add overhead since you must keep the multiple CPUs in sync. Scaling out with multiple single CPU VMs allows for more flexible VM scheduling and also removes the overhead in a multi processor Guest OS and App stack. Now, if you want to scale out with multi CPU VMs that’s fine. I was simply making the point that VMs allow you to think differently and scale differently. Of course it’s a though process that’s uncomfortable for most.

3) I’ll be posting the PXE booting of ESX soon.

4) The correct URL for my blog is http://www.mikedipetrillo.com.

5) Andy Leonard posted a great URL that I had forgotten to hand out before for time sync in guests. Make sure to check it out in the comments above.

I hope that clears some things up.