How to get ESX Host and Virtual Machine Disk I/O Stats

Posted on June 25th, 2008 in SAN, esx, esx3.5, how to, storage, vc2, vc2.5, vmware by Rich

Lately, I have had several clients and peers ask me how to get disk usage and performance statistics from their current virtual infrastructure of ESX hosts and virtual machines . Some needed data for planning and sizing a new SAN, while others needed statistics for upgrading, adding more disks, or for optimizing multi path and VMFS performance. In one case the customer was trouble shooting poor VM performance issues. Regardless of the objectives there are some built in tools in both VirtualCenter and ESX server that can get this information for you. This post explains 2 native methods:

  • Using the VI Cleint to access the Performance data in VirtualCenter
  • Using esxtop from the ESX Service Console

I have included plenty of screenshots. As always, click on them for larger views. The rest of this post is in an outline format, but should be easy enough to follow.

730 Days Later - Replace The VirtualCenter Default SSL Certificate

Posted on June 25th, 2008 in vc2, vc2.5, vi client by Rich

Yes, this post uses another movie reference.

In the film 28 Days Later the Rage virus infects the Island of Great Britain turning all but a few survivors into zombie-like monsters called “The Infected”. The virus was unleashed when animal activists released medical research chimpanzees which ended up attacking the activists and scientists. This post is about what could cause a similar rage 730 days after installing VirtualCenter, potentially causing VI administrators to become lifeless, rabid, and insane.

After installing VirtualCenter (VC), you should check the installed SSL certificate used by the VI Client because you will most likely need to manually replace it. After a fresh installation the default certifcate expires in 730 days (or 2 years). If the certificate expires you will be unable to log in to the VirtualCenter Management Server using either the VI Client or the web administration interface.

Unfortunately, it is unclear to me at this writing if upgrading the VC Server within the 730 day period updates the certificate store.

P2V error: File size is larger than maximum size supported by datastore

Posted on May 22nd, 2008 in P2V, SAN, converter, storage, vc2, vc2.5 by Rich

VMFS block size optionsI was helping a customer P2V a large development SQL server this week and ran into a VMFS configuration issue that failed the conversion. We were using the Converter Enterprise for VirtualCenter 2.5 plugin. Almost as soon as we kicked off the job it failed with an error starting with “file size is larger than the maximum size supported by datastore”. The VMFS LUN we were using as the target was an empty 1.5 TB volume, and the new VM consisted of 2 virtual disks that totaled roughly 450 GB. We had plenty of room, but the problem was not the available storage space. Instead, the issue was that we exceeded the maximum possible .vmdk size for the default VMFS 1MB block setting.

When you add new storage to an ESX host and you format the LUN with the VMFS file system you have to choose what block size setting you want to use. See the screenshot for the dropdown box used to make this choice. Notice the Maximum file size description supposedly provided to help you understand this setting. It’s hardly intuitive in my opinion, so let me try to translate - Choosing the block size determines what maximum possible .vmdk size can be created on this LUN.

If you do not change the default setting when you format a VMFS LUN

Use VMware Converter to Solve ESX Snapshot Issues

Posted on May 21st, 2008 in blogs, converter, how to, vc2, vc2.5 by Rich

Carlo Costanzo over at ipmer.com has a great post about how to quickly and easily solve issues resulting from VM snapshots. As Carlo points out, too many administrators misunderstand the ESX snapshot to be a point in time backup and unfortunately do not realize it is instead a live and growing file. More often than not the snapshot is forgotten until the LUN is completely out of space at which time the VM is unstable. Trying to commit the snapshot becomes a time consuming burden.

The post 70GB Snapshot, YIKES! explains how Carlo used some “outside of the box” thinking to use VMware Converter to rescue VMs without going through the commit process. The idea is so simple it’s brilliant! Carlo writes:

VirtualCenter is sluggish while waiting for hung tasks.

Posted on May 19th, 2008 in esx, esx3.5, vc2, vc2.5 by Rich

A Google Group discussion thread asking for help with hung tasks in VirtualCenter points out an important Service Console configuration best practice. Killing tasks on vcenter starts off by asking if there is a way to kill several VirtualCenter tasks that are bogging down the management server. Although the method for killing individual tasks is never realized, understanding potential reasons for the problem is the more important take away. Fixing the VirtualCenter performance problem results after correcting ESX host resource issues.

Troubleshooting ESX logs

Posted on May 13th, 2008 in breakout, esx 3i, esx3.5, how to, vc2, vmware by Rich

Another session I attended at VMware’s Partner Exchange last week was titled ESX Log Analysis - Tech 207. I did not realize it when I signed up, but this was essentially the same session that I previously attended at VMWorld 2007 last September. I did a quick Google search on this topic to find the VMWorld slides and noticed that Scott Lowe live blogged from San Francisco while attending this very session. Then Searching on the VMWorld.com site I found that this was also a session at VMWorld Europe 2008 titled VI3 Advanced Log Analysis. You can get a copy of the .ppt used at the VMWorld Europe 2008 session on my Files Page.

There is nothing really too new about t-shooting ESX logs here, but the following are my notes from last week. On the other hand, there are some general notes directly related to ESXi logs and using Update Manager included.

I cleaned up my notes a little, but the following is still a raw outline. use These notes and the .ppt mentioned above to hopefully help educate yourself on this topic.

FREE Disk space monitoring solutions for VMware virtual infrastructure

Posted on April 14th, 2008 in how to, scripts, storage, vc2, vc2.5, vi3 by Rich

VMware VirtualCenter comes with built in alerting and a handful of alerts preconfigured. Unfortunately, alerting for disk space usage of either the ESX hosts or the virtual machines is not included. Administrators continue to use common physical infrastructure monitoring and reporting applications such as NetIQ and MOM for VMs, or SNMP capable programs like HP Openview or IBM Director for ESX host monitoring. A less complex and less expensive ( cost of installing and configuring agents on each VM OS ) alternative would be to tap into VirtualCenter’s central management ability to monitor, alert, and report on disk space. This post lists a few free solutions that can already use VC2.x or quickly be configured for ESX hosts and therefore save administrators time and money. Hopefully, a future feature of VC2.x will include vital disk space metrics and alerting.

Failed to install the virtualcenter agent service

Posted on April 2nd, 2008 in blogs, how to, vc2, vc2.5 by Rich

After upgrading Virtual Center from 2.0.1 to 2.5, all of the ESX 3.0.1 hosts were disconnected. Right clicking a host and selecting connect failed after about 5 minutes with an error that started with “failed to install the virtualcenter service.” Turns out the resolution is to manually create a missing directory on each of the affected ESX servers. Without this directory the agent install process will fail. The issue is not related to VC 2.5, but occurs when upgrading to earlier versions of VC 2.x as well.

The directory in question is /tmp/vmware-root. If it does not exist then follow these steps from VMwarewolf to reconnect your ESX hosts to VC again:

Designing ESX Resource Pools

Posted on March 4th, 2008 in cluster, drs, esx, esx 3i, esx3.5, fail over, how to, services, vc2, vc2.5, vi3, vmetc.com by Rich

How do you design resource pools in an ESX Cluster? There are two strategies that are the most popular in my experience. The first strategy creates resource pools based on CPU and Memory shares for host resource conflict management, and the second strategy uses reservations and limits to guarantee physical resources and ensure VM containment. This post will use a 3 ESX host example to explain both strategies. Please feel free to comment on the pros and cons of each or why you think one is better than the other.

In the example scenario three ESX hosts each have 16 GB RAM and 2 dual core 3.0 Ghz CPUs. The three hosts will all be members of the same ESX cluster.

Script to clean up Virtual Center database

Posted on February 1st, 2008 in how to, scripts, vc2, vc2.5, vmware by Rich

I was directed to a VMware Knowledge Base article about a script for cleaning up the VC database while upgrading to VC2.5. The company had about 18 months worth of data in their VC SQL database and wanted to preserve some recent historical data, but didn’t need to keep all of it. Purging data helped free disk space and reduce the time the upgrade actually took to complete.

The VMware article links to scripts for both

Should Virtual Center run as a Virtual Machine?

Posted on December 28th, 2007 in esx, vc2, vc2.5, vi3 by Rich

Sure, you can run Virtual Center Management Server as a Virtual Machine. VMware supports it and has published a technical note about doing it - http://www.vmware.com/pdf/vi3_vc_in_vm.pdf. A lot of companies have built their VC2 server this way, but is it really wise to have “the manager” of the environment running in the environment it is managing?

Think about it. Virtual Center provides VMotion which in turn enables DRS and HA. Agents running on each ESX host are communicating back to VirtualCenter for these features to work. A lot could go wrong with some pretty severe consequences. This is even worse if a company has completely virtualized their network services such as DNS. I’ve personally seen where a client “shot themselves in the foot” because Virtual Center and all networking services where provided by VMs.

In my opinion the VMware tech note is a not exactly a “glowing endorsement” for installing VC2 in a VM. It is best practices guide that explains critical design criteria.

First, why would you put VC2 in a VM? The tech note begins by offering some good reasons:

VI3.5 New Feature Summary

Posted on November 17th, 2007 in esx, esx 3i, esx3.5, srm, stor vmotion, vc2, vc2.5, vcb, vdm, vmware by Rich

VI3.5 what's new summary

VMware VI3.5 is scheduled to be generally available by the end of 2007. Based on a public .pdf released by VMware, this post is my summary of the new features. I’m guessing this .pdf is really a .ppt presentation that VMware is delivering, but I have not attended this presentation myself.

The screen shot is from the document and shows the new features in the defined layers.

After some sales and marketing slides the document classifies the VI3 features today across 3 layers. These layers are then used to group the new features for the rest of the presentation.

The features are divided into 3 layers: