Here is a general outline for automating the DR ip addressing with this method:

At the Primary Site

• For these instructions assume the production vSwitch at the primary site has a Portgroup named VM Network
• Build a new vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named DR Network
• For each VM you need to fail over to a DR site, add an extra vNIC and attach it to the DR Network Portgroup

At the DR Site

• Create your DR site production vSwitch, attach physical NICs and add a Portgroup named DR Network.
• Create another vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named VM Network

All you have to do for this to work is

configure the extra vNIC in each VM at the primary site with the correct VLAN and ip address of the DR site production network. This ip address configuration will be cloned in the replicated DR VM, and when it’s time to power on the VM at the DR site both vNICs will connect to their respective virtual networks. The DR Network Portgroup will now be on the vSwitch with the physical connectivity to the DR production network and the VM Network vNIC will be isolated to a local only vSwitch.

Vizioncore deserves the credit for showing me the idea, but there’s no reason why this won’t work with any VM replication or VM backup product. It’s not automated DR fail over workflow like VMware SRM, but using this extra virtual networking strategy eliminates some manual or even scripted ip re-addressing.

Simple!

VMware customers who currently own SRM with active Sales and Support (SnS) contracts will receive the upgrade at no additional cost. VMware has not changed the per processor licensing model or cost for new customers wishing to purchase.

Some quick links for SRM 4:

• SRM 4. Release notes – http://www.vmware.com/support/srm/srm_releasenotes_4_0.html
• Upgrade KB article – http://kb.vmware.com/kb/1013166
• SRM download – http://downloads.vmware.com/d/ and select Site Recovery Manger
• Michael White’s post SRM 4.0 is here! The wait for vSphere and NFS support is over! on the VMware Uptime Blog covers upgrade strategy options.
• VMware’s documentation for all version of SRM – http://www.vmware.com/support/pubs/srm_pubs.html

vSphere 4.0 Supported

The main requirement to enable the new compatibility and features in SRM is upgrading vCenter to version 4. In fact,

SRM 4 can manage the fail over of multiple VMware Datacenters configured from a single pairing of a primary vCenter instance and a secondary vCenter instance at a recovery site (DR site). The key to this configuration is the dependency on and the location of the 2 vCenter 4 servers. Although I do not recall it being specifically stated in the product pre release call I participated in, technically speaking, it seems as if VI storage at multiple physical locations could be replicated to multiple DR sites if all the sites involved were managed by only the 2 vCenters. This implies vCenter management, and the necessary Site Recovery Agents (SRA), would be remote to some of the shared storage replication taking place. Since the storage replication is configured independent of vCenter and SRM, SRA connectivity and SRM workflow automation via the SRM vCenter plugins would be possible.

NFS Support

New SRAs are being released from the storage vendors for automating fail over between sites with NFS storage replication in place. Although not all vendors currently on the SRM HCL will have NFS SRAs available today, we were told most of the missing vendor agents will be available immediately. Each vendor has to go through a separate qualification process for the new storage protocol.

My notes about NFS included a bullet about SRM 4 now being able to support up to 1000 VMs. This is up from a 500 VM maximum previously.  This is per VMware Datacenter in vCenter. Multi VMware datacenter pairings would be the method to go beyond the 1000 max. I’m not sure if this is only for NFS storage, or if the maximum number of VMs applies to all supported SRM storage protocols.

vCenter Linked Mode Supported

vCenter Linked Mode enables an administrator to manage objects from multiple instances of vCenter in a single VI Client (VIC) session. With SRM now supporting Linked Mode, VI administrators can more easily monitor and manage SRM at both the primary and secondary sites. Note that SRAs must be installed separately on both instances of vCenter, but the SRM plugin can be installed once in a administrator’s VIC connecting to a Linked Mode vCenter.

Other Notes on SRM 4

Probably the largest feature of interest which SRM users have been
waiting for has still not been implemented in this latest release – an
automated fail back method. The process still requires manual reconfiguration effectively reversing the primary and secondary sites after storage device replication is established (in the reverse direction as well).

If you are interested in possible long term futures and the roadmap for SRM, check out the session titled BC3421
SRM Architecture & Features: The Road Ahead from VMworld 2009 recently in San Francisco. PDF of the session available at http://www.vmworld.com/docs/DOC-3498.

I am not suggesting that SRM is a sham. In fact, it provides automation of virtual infrastructure fail over between sites that is truly wizard-like. Understand however, VMware SRM software is just the last piece of the total data center recovery “machine”. Many organizations may be seeking the semblance of automated site fail over, but have they really considered in detail what it takes to start up their business critical systems at a secondary location?

A simple determination of readiness for SRM’s wizardry is answering this question:

For those that are considering SRM, take the time to at least put on paper every possible step you would need to restart your business in another data center. This includes the mundane like providing power and cooling, racks, internet access, and telephones as well as the mission critical like employee access, email, and business processing. Analyze how much time it would take to rebuild and restart systems, and what recovery point objectives (RPO) and recovery time objectives (RTO) are acceptable for each system or service. When it’s time to map your disaster site fail over solution to the runbook you will clearly see the efficiencies and the speed that VI 3.5 and SRM allow.

Therefore, unlike the line from The Wizard of Oz, you better pay close attention to the man and machine behind the curtain in order to achieve the prodigious expectation and pyrotechnic results of a SRM implementation.

I attended this VMworld 2008 session on Wednesday 09.18 at 9:00 AM. The presenter was Sunder Parameswaran who is a Senior Product Manager at Symantec. The session was about using Veritas Cluster Server (VCS) in VMware virtual infrastructure (VI) to overcome application high availability and disaster recovery (DR) challenges.

My main interest in this session was on the topic of Geo Clustering with VCS. Also known as Metro Clustering, this is the ability to have one node of the cluster at your primary data center location and the second node at a separate physical location like a disaster recovery site.

Sunder began by outlining various VI challenges.

• Need for heterogeneous management of mission critical applications in both physical and virtual
  servers.

  • VI makes management more complex because a distinct set of tools is used today for both physical and virtual systems

• VI creates a larger emphasis for handling unplanned downtime and DR

With over companies using VCS for over 10 years in the field, VCS provides the following features to meet
these challenges:

• Can manage mixed physical and virtual environments

• can manage various server operating systems

  • solaris

  • aix

  • linux

  • windows

  • VMware

Next Sunder talked specifically about how VCS provides application aware high availability for VMware above
and beyond VI Enterprise HA features of an ESX Cluster. The session discussion also covered VM application clustering advantages unique from Microsoft’s clustering (MSCS)

• VCS can monitor and fail over several mission critical applications such as Exchange, SQL, Oracle,
  and SAP

• There is no need for passive standby hardware

• Requires agents in the server OS

• provides DR ability to cluster between physical data center locations

Geo Clusters (Metro Clusters) with VCS

• Requires synchronis replication between 2 sites within 80 km of each other

• It is directly effected by the link latency and the quality of the replication technology

• single click cluster configuration that is completely automated.

• can handle split brain network scenarios

Global DR with VCS

• VCS Firedrill provides zero downtime testing of DR plans

• requires array based replication

• leverages storage snapshots of thefiler at the DR site

• can be scheduled (example of DR testing every month)

• provides network isolation of DR test servers

• Is not fault tolerant and client will notice downtime

VMware HA versus VCS

• VCS protets against application crashes where HA watches for ESX or VM crashes

• VCS can control the specific start up order order

• VCS protects against admin errors in the VM configuration.

  • Configuration changes

  • admin error

  • Server NIC failure (not usually a problem in VMs, but possible)

• VCS protects against storage connectivity failures

VCS Compatibility

• VCS is not supported in ESX version 2.5.x or earlier

• VCS is certified with various SAN vendor replication technologies

  • HDS Truecopy / Universal

  • EMC SRDF

  • IBM Metromirror

  • HP EVA is on the roadmap

  • NetApp Snap Mirror is on the roadmap

• VCS Application support

  • IIS

  • SQL 2005 and 2005

  • Apache

  • Linux

  • Oracle 10g

  • SAP

  • Exchage 2003 and 2007

  • Websphere

  • Sharepoint

  • Weblogic on Linux

  • Framework to customize for any application

  • is not supported / can not use or combine with MSCS

VCS Considerations for VMware Vmotion, DRS, and HA

• Considered by Symantec as complimentary solutions

• Postion VCS for applications and scenarios where Vmotion and DRS do not cover availability needs.

• VMware HA is power off crash consistent fail over and VCS is clustered fail over

  • recommendation is to disable VMware HA when using VCS

• VCS installs agents on each ESX host. State of each host is known by other hosts

Protecting VirtualCenter Server with VCS

• Discussion was focused on VC as a VM, but I imagine this applies to physical VCs as well

• Has additional logic to auto reconnect ESX hosts after fail over to second node

• • Same goes for Geo Clustering of VC (Metro Clustering)

VCS Futures

• Future VCS ESX installations will utilize VIMA. It will be optional whether using a dedicatd VIMA or a
  shared VIMA

• Prioritized applications – the ability to give priority to specific application fail overs

• browser based management

• enable active server provisioning based on workload

• multi tier application control and support

There are currently 2 licensing models for VCS. You can purchase licensing by number of VMs or by the ESX
CPU.

virtualization, vmware, veritas, vcs

SRM requirements:

• VirtualCenter licensed and running at both the primary and secondary sites.
• ESX hosts licensed and running at both the primary and secondary sites.
• Backend database for SRM at both the primary and secondary site.
• Shared storage capable of, and licensed for, block level replication (SAN based) at both the primary and secondary sites.
• Site Recovery Agent (SRA) installer provided by the SAN manufacturer.

Note: if your SAN manufacturer does not create a SRA you can not use SRM. There is not an official SRA manufacturer list at this time. The instructors in my lab indicated most all major SAN manufacturers are working on developing SRAs, but the list will not be official until product release. My lab used a LeftHand Networks SRA.

Preparation for SRM:

LUNs

Organize your LUNs / volumes for SRM because you will probably not need all of your VMs to be replicated. For example, your VC server, Update Manager server, and SRM server (if they are VMs) should not be replicated. If other similar infrastructure server VMs should not be replicated then move these VMs out of the LUNs / Volumes containing the VMs that will be replicated. Move all of your non critical VMs to their own LUNs. The idea is to reduce replicated traffic as much as possible by syncing only select LUNs containing your critical VMs. Obviously, this will be a significant project for some organizations by itself. Leverage storage vmotion to minimize VM downtown during this process as much as possible.

Database

I was told the database requirements are exactly the same as VirtualCenter. Since the product has not been fully user tested yet, my instructors indicated there are no solid best practices yet for SRM database design. In other words, can SRM be installed on VC? Can Update Manager and SRM be installed on the same server? Should all of these servers be VMs or physical servers? Personally, considering VirtualCenter, SRM, Guided Consolidation, and Update Manager all require a database, I am of the opinion it will make the most sense to use ODBC connections to a stand alone database server. If I am going to make all of these servers VMs then I would make a separate server for each as well. Be sure to include this design in the LUN preparations!

Lab and Configuration overview:

Primary Site

1. Install SRM code on SRM server (SRM was installed on VC in my lab)
2. Install SRM plugin in VI Cleint connecting to VirtualCenter and enable plugin
3. Install SRA on SRM server

Secondary Site

1. Install SRM code on SRM server
2. Install SRM plugin in VI Client connecting VirtualCenter and enable plugin
3. Install SRA on SRM server

Replication from Primary Site

1. Use SRM plugin to configure and verify replication to secondary site

Test fail over and work flow using the SRM plugin from the primary site

When the lab materials are released to the public I will post a copy here on vmetc.com. I do not have them as of this writing.

Quoting from the Best of VMworld (more on this award later in this post) white paper downloadable from the everRun link above:

” everRun creates a paired virtual machine environment that appears as a single virtual machine environment for both administration and application operations. The environment presents a single IP address and hostname to the application network so users never have to make client-side changes when failures occur. With everRun ComputeThru technology, protection can be configured so that component and even server failures go unnoticed. You simply keep computing through the failure, rather than failing over between servers.”

“The protected virtual machine appears and is managed just like a standard Windows server. Disk data is mirrored synchronously to redundant storage and network and server operations are protected from failure. The administrator loads and configures the application in the protected virtual machine as though it was being loaded onto a physical server and then simply walks away. There is nothing else to do!”

“Failure detection is automatic and recovery policy is embedded. There is no need to create and test complex fail over scripts. everRun works equally well with all classes of storage devices, unlike clusters that require expensive shared storage. Configuration is simple; everRun presents a single protected virtual machine to the administrator and transparently manages redundant resources for availability, removing opportunities for operator-induced errors.”

The everRun VM virtual appliance is called the everRun Availability Manager, and it is accessible and configured via a web interface. The browser based interface provides single glance access to events and status.

Although everRun VM is not clustering, it is like clustering in that it requires a shared private network between XenServer hosts. The technology is loaded as a virtual appliance on each host that monitors the VMs under it’s protection. For designs with more than 2 hosts the private network becomes a “token ring” -like network for XenServer hosts to communicate the status of VMs. The key requirement to the solution is latency of the private network.

Also unlike clustering, it should also be stressed that the everRun VM solution is application independent. All Microsoft applications are supported (yes, Exchange and SQL). Since the mirroring takes place at the VM level the application is irrelevant to the process.

The everRun VM solution can be scaled for one to many VM protection scenarios or even used as remote site disaster recovery fail over. For remote data center fail over with the previously mentioned latency requirement was recommended by Michael Bilancieri at the Atlanta event to be less than 10 ms.

There are three levels of protection for a VM offered by everRun:

1. Level 1 – High Availability (Basic fail over)
2. Level 2 – Continuous Availability (VM-level assured)
3. Level 3 – Fault Tolerance (Uninterrupted computing)

My notes and my memory from the event are unclear about the difference between Level 1 and Level 2, but “big red button” fail over (power off crash consistent recovery) like VMware ESX’s feature falls somewhere between these first 2 levels. Level 3 is live fail over like, for lack of a better example, clustering. everRun offers the ability to set the level of protection for each VM independently.

Marathon won an award in the New Technology Category from SearchServerVirtualization.com while at VMworld 2007. Interestingly, Marathon provided a demo in their booth at VMware’s conference running on XenServer. Marathon and XenSource were one of 2 winners in the category from over 100 exhibitors at VMworld.

Finally, everRun is not a solution only for XenServer. It will work with VMware ESX. everRun’s HA features can compliment or replace current VI3 HA functionality by offering separate protection to VMs, or offer an alternative to SRM as a Ethernet based solution for DR site fail over.

In the example scenario three ESX hosts each have 16 GB RAM and 2 dual core 3.0 Ghz CPUs. The three hosts will all be members of the same ESX cluster.

I tried to create design names that illustrate the strategy. I am not aware of any specific resource pool design naming from VMware.

The Tug of War design

This design is simple and does not limit any VMs from any physical resources. Using the ESX shares mechanism, if two or more VMs are competing for the same physical resources the tug of war that results will be decided by the resource pool memberships of the VMs.

The ESX cluster will have three resource pools defined.

• A “High” resource pool will have no initial reservation and unlimited/expandable RAM and CPU settings. CPU and Memory shares will be set to high. This resource pool will be devoted for mission-critical VMs.
• A second “Normal” resource pool will have no initial reservation and unlimited/expandable RAM and CPU settings. CPU and Memory shares will be set to normal. Normal priority service VMs will be placed into that resource pool and the shares guarantee that VMs in this resource pool do not have priority over mission critical VMs.
• A third “Low” resource pool will have no initial reservation and unlimited/expandable RAM and CPU settings. CPU and Memory shares will be set to low. Low priority service VMs, development, and non critical VMs will be placed into this resource pool and the shares guarantee that VMs in this resource pool do not have priority over normal or high VMs.

The three resource pools configured by shares will ensure that the proper VM has priority to physical resources when a conflict occurs. Resource conflict management can take place with the full capacity of the ESX cluster’s physical resources.

The Pizza Design

updated 03.09.08

This design takes the sum total of all physical resources and slices it up across the resource pools. Although the following design only uses two resource pools, many more “slices” could be created. The most basic Pizza Design would be to reserve all memory and cpu, but the following example helps also illustrate reservations and limits.

The ESX cluster will have two resource pools defined.

• A “Critical Services” resource pool will have an initial reservation of 32GB RAM and 8GHz CPU, and unlimited/expandable RAM and CPU settings. This resource pool will be devoted for mission-critical VMs. Shares for RAM will be set to high, but shares for CPU will be set to normal.
• A second “Non Critical Services” resource pool will have no reservation and a limit of 16GB RAM and 4GHz CPU. Shares for Memory and CPU will be set to normal. Low priority service VMs will be placed into this resource pool and the limits guarantee that VMs in this resource pool do not over consume resources.
• The remaining balance of CPU is fully available to the Critical Services pool because of the expandable reservation.

This specific design example was based on a mix of VMs that were expected to contend for memory more often then cpu.

Final Thoughts

I usually start a design with the tug of war strategy. Resource Pools can be confusing, complicated, and difficult to troubleshoot if the design gets complex. Starting out simple makes sense until a specific VM performance issue is determined. On the other hand, the pizza design is a better starting point if you know you will have heavily used development VMs mixed in with your production VMs. The pizza design ensures that if a developer launches a rogue script on one of the VMs then the production system will not be impacted.

LOL! After writing this I am hungry for some pizza ….

First, what is a fail over scenario? The first thing that comes to mind is a problem. One or more of your ESX hosts unexpectedly crashed. This is considered unplanned downtime. Another fail over scenario to consider is planned downtime such as rebooting after applying ESX patches. For both of these types of scenarios you want to make sure your VMs stay online.

VMware’s solution for planned downtime is VMotion. The solution for unplanned downtime is the HA feature of ESX Clusters. When determining your ESX capacity be sure to allow room to leverage these features.

VMotion migrates a VM to a different ESX host without users losing connectivity. Evacuating an ESX server by VMotion enables you

I’ve heard a lot of companies adopting an N + 1 mentality recently. This is typically based on a consolidation estimate’s results. If the capacity analysis study determined that 4 ESX hosts are needed then include an extra ESX host just for added capacity and fail over resources. That’s a good start, but it might not be enough.

Remember that a capacity analysis project is centered around the “WOW! factor” of demonstrating how many servers can be consolidated on ESX hosts. The ESX host utilization is often targeted to be 80% or higher – meaning host as many VMs as possible until the host’s cpu and ram is 80% in use. Using the N + 1 strategy that means that losing one host will work fine. 80% load spread over 4 hosts is maintained by having the 5th ESX host. What if you need to allow for 2 ESX host failures? In that case there is a problem. Here’s some math to demonstrate why:

• original design was 4 ESX servers at 80% = 320%, but if you lose 2 ESX hosts, even with the N + 1 server, 3 hosts will be running at over 106% utilization.

That may be acceptable for a short period of time, but an Administrator should be ready to add another ESX server ASAP to maintain normal performance levels. It gets worse when you lose more ESX hosts simultaneously. A consolidation scenario where ESX hosts are only 65% utilized helps the numbers, but it also means more hosts.

The reality of the IT budget and the reliability of modern hardware keep most companies from planning for more than 1 ESX host failure. Maybe that’s why the N + 1 strategy is so popular? If this is your case you maybe able to still plan for more than 1 host failure by categorizing your VMs as critical and non critical. Create 2 separate ESX Clusters. Enable DRS and HA on the Cluster with the critical VMs, and only enable DRS on the non critical cluster. The N + 1 design already gives you enough capacity for planned downtime (by rotating maintenance on one ESX at a time), and by limiting the number of VMs that HA will restart, you might be able to maintain critical VM availability through multiple unplanned host failures. You can achieve the same design with a single cluster by setting the non critical VMs fail over behavior to not restart in the properties of the Cluster.

Monitoring your ESX Clusters is critical to be sure to maintain the design’s expected availability. Obviously, As your number of VMs increases there will quickly come a point where the number of ESX hosts will need to increase.

For more information about HA check out VMware’s HA page. For more information on VMotion then check out VMware’s VMotion page.