Update and Auto Patch ESX
Even though VI 3.5 is now generally available you may not be ready to upgrade from version 3 just yet. You should, however, make sure all patches and updates are applied to your VI3 infrastructure. This post explains the upgrading and patching methods I have standardized on and provides some tips for avoiding common issues like HA agent configuration and auto applying numerous ESX patches. All scripts and files referenced are linked at the bottom of this post.
Prepare
Preparing for the upgrade and patching is critical to minimizing what can go wrong.
Upgrade Virtual Center
Notice the compatibility table in the ESX 3.0.2 Release Notes. VC2 is compatible with previous versions of ESX, but ESX 3.0.2 is not compatible with previous versions of VC2.
To minimize issues with upgrading VC2 be sure to download and install Virtual Center 2.0.2 update 1. You need a username and a password to download it. If you were not the contact for your company when you purchased VI3 then have the appropriate person log in and download it for you.
Update the VI Client
Be sure to download the new VI client from your VC2 web interface and update all workstations where you use the client. Update the client on the VC2 server as well.
Enable NTP sync
I use WinSCP to put the set_ntp.sh script in /tmp on all ESX hosts. I use Putty to run the script. Verify the time is accurate on all your hosts. Check out my post about how to configure NTP here.
Update /etc/hosts
Vmware Support employee and blogger VMwarewolf describes this as My Number 1 Tip for ESX and gives a good description about how to change the /etc/hosts file. He also gives some explanation as to why it is important.
You want to add every ESX host machine that is in your cluster to each other’s hosts file. Not only does this make HA much more robust, it makes DNS lookups redundant, and that’s a good thing. Ask yourself, if my DNS has an outage for just 12 seconds, do I really want all of my HA nodes going into isolation mode?
Why do we need to do this? I’m not sure why it helps with VMotion, but HA needs it. HA you see was not written by the same developers as ESX. HA was developed by Legato, which is owned by EMC, as is VMware. It’s a marriage made in heaven, but the devil’s in the details!
Get the ESX 3.0.2 .update 1 ISO
Download ESX 3.0.2 update 1 ISO from here. You will need to already have a username and password. This version rolls up all ESX 3.0.X patches prior to 11.15.07 and will upgrade all ESX 3.x servers. Burn it to CD so you can boot your ESX hosts from it.
Download current patches
http://www.vmware.com/download/vi/vi3_patches_302.htm
You do not need a username and password to download patches. As of this post there are 3 patches released on 11.15.07 and 6 patches released on 11.20.07. Save them all to the same location. Eventually they will be extracted and transferred to the Http Depot.
Build an Http Depot
Virtrix - Virtual Tricks: VMware: Autopatching your ESX host is where I first heard about using the Http Depot method of automating applying multiple patches to ESX. The Virtrix post walks you through setting up an IIS server to use as the depot, and provides the original patchESX.pl script code. Over time and from various different contributers the script I now use ( linked at the bottom of this post) changed to include the following features:
- Open/close firewall port
- centralized variable section
- log creation
- sleep in between patches (VMware recommendation)
- Patch status
I usually give my clients the option to build their own Http Depot, or if they want me to “just do it” then I use a VMware Server VM I have on my notebook. For my VM I run Apache on Fedora 6.
Extract all of the downloaded patches and put them in the directory created on your web server. On a Windows computer I use ExtractNow to extract the .tgz format.
Create the patchlist.txt file and put it in the web directory with the extracted patches. This file contains the order the patches will be applied and should be in the order they were released by VMware. Be sure not to include the “ESX-” prefix of the patch folder names. This post has the current, as of this writing patchlist.txt linked at the bottom.
Modify patchESX.pl script
You have to make a few modifications to patchESX.pl script before you transfer it to each ESX host. Open the script and only change the variables in the first section:
#!/usr/bin/perl
# PatchESX.pl -- auto update esx perl script
# Version 6/28/07
#
# Set the following variables
# -webserver = IP address or domain name of the server hosting the updates.
# -webdirectory = Subdirectory of the webserver where the updates reside. Use "/" if the updates are at the root.
# -webport = The port that the webserver is listening on (default 80)
# -patchfile = The name of the text file that contains the numeric updates in order.
# -sleep = The seconds to wait between patches (VMware recommends 30 seconds)
#
$webserver = 'http://10.201.1.30';
$webdirectory = '/';
$webport = '80';
$patchfile = 'patchlist.txt';
$sleep = '15';
#
# DO NOT MODIFY ANYTHING PAST THIS POINT
#
Save the modified patchESX.pl script and use WinSCP to copy to /tmp on each ESX host.
Now you are ready to actually do the upgrading and patching.
Upgrade and Patch
Disable DRS and HA
In my experience I have had the best results with preventing the HA agent issues by unchecking both DRS and HA in the Cluster Properties before I begin upgrading and patching. This also serves the purpose of ensuring VMs will not be auto vmotioned during the process.
Maintenance mode / evacuate with vmotion
During the process you will have to “play a shell game” with your ESX hosts if you to keep your VMs online. You can only do this if you have available capacity on your other ESX hosts. If you don’t have capacity to evacuate all the VMs to other hosts then you will have shut down VMs.
Once you vmotion or shutdown guests you can put the host in maintenance mode from Virtual Center. It’s not required if you’ve already disabled DRS and HA, but it only takes a few seconds and gives you some extra peace of mind if you have multiple administrators who may not be aware of your work.
Upgrade with 3.0.2 update 1 .iso
Put the ESX 3.0.2 update 1 CD in the CD/DVD drive of your ESX host and reboot the server. Once the server boots from the CD choose the graphical installation. When you are prompted be sure to choose Upgrade.
Be extra cautious to make sure you choose the correct data store where you have ESX installed. If you did not take the time to disconnect your SAN (who does?) then I have seen where the installer selected a VMFS volume by default.
After about 10 to 15 minutes and rebooting the server you have successfully upgraded your server and applied all patches up to 11.15.07.
Run patchESX.pl
To apply the post ESX 3.0.2 update 1 patches use Putty or a local console and enter the following commands:
#cd /tmp
#perl esxPatch.pl
This changes you to the /tmp folder and kicks off the auto patching process. Sit back and watch for another 30 minutes until it’s done. Manually reboot your server.
Post upgrade
Exit maintenance mode
Go back to Virtual Center and take the host out of maintenance mode.
If this was not your last ESX host to upgrade and patch then vmotion evacuate the VMs on your next ESX host and repeat the process. Do this until all of your ESX hosts are accounted for.
Enable DRS and HA
When all hosts are completed turn back on HA and DRS. Let the new agents start up and make sure all is OK. You will probably noticed auto vmotion taking place or have migration suggestions if not in automatic mode.
All files discussed in this post can be downloaded from the VM /ETC Files Page.
Related Posts
Tags: esx, fedora, how to, issues, patch, SAN, script, scripting, scripts, upgrade, VAC, vmware, windows
