http://kb.vmware.com/kb/1031106

The KB was updated 7.26.11

As always, I’ll leave the deep technical details of Drobo’s unique and patented Drobo BeyondRaid technology to my fellow storage bloggers and stay focused on how Data Robotics fits in virtual infrastructure. In this post I expand a little on why the Drobo storage device is a VMware HCL certified, simple to implement and expand iSCSI SAN targeted for SMB customers that is an exciting alternative. Finally I offer opinions based primarily on my virtualization server administrator perspective.

Data Robotics told us during their TFD presentation that to date they had about 80,000 customers. I mention the BeyondRaid technology later in this post, but I’m sure a lot of these customers were intrigued by the cost of the device and not just the technology. As mentioned on a previous Infosmack podcast hosted by Greg Knieriemen, a DroboPro with 16 TB of usable disk storage can be purchased for roughly $4000! That cost and capacity is based on the current cost of 2 TB SATA drives – the largest capacity available today. Check out the Infosmack TFD wrap up episode as well for more comments from attendees on Drobo.

It should be mentioned that Data Robotics was surprised that a healthy percentage of those customers began using Drobo for ESX storage before Data Robotics knew about it. That discovery lead them to get the VMware HCL certification.

DroboElite released
On 11/23 Data Robotics introduced 2 new products, the DroboElite and DroboS. DroboElite is aimed squarely for the SMB datacenter and improves the DroboPro’s prevision data center shortcomings. With dual ethernet ports and official support for up to 16 ESX hosts in a cluster, the DroboElite is obviously engineered to maximize it’s role as a ISCSI SAN option for ESX Clusters.

TFD organizer and Microsoft Storage MVP Stephen Foskett states it best in one of his posts:

“… DroboElite delivers nearly everything a small business could want from a computer closet storage device. It is a real honest-to-goodness iSCSI SAN array, supporting multiple servers and sporting dual gigabit Ethernet ports and dual-drive data protection.”

The press release linked above also provides the following virtualization related info:

“Tested VMware-ready Features – With dual iSCSI ports and the ability to create up to 255 Smart Volumes, DroboElite can support growing VMware environments and advanced VMware functionality including VMotion, Storage VMotion, snapshots, and high availability.”

BeyondRaid Overview
Data Robotic’s BeyondRaid eliminates provisioning arrays and fully utilizes total disk capacity even when used with a different size disks in the bays. For example, a Drobo storage device can have 2TB, 1TB, 500GB, 300GB, and 250GB SATA drives (or any combination of sizes for up to 8 SATA drives simultaneously) without limiting the data distribution to the smallest drive size. Furthermore, the drives can be any commodity SATA drives (buy them from where ever you want!) and can be added and removed while Drobo is powered on (up to 2 drives at a time with DroboElite). Best of all, you can remove the smallest drive and replace it with a larger one and increase the total disk capacity – all while the Drobo is serving live data! This means as SATA disk capacity and performance increases in the future, the total capacity and performance of a Drobo will do the same. Simply remove the old disks and replace with new disks on your own schedule.

At Tech Field Day we witnessed first hand the ease of creating Drobo storage volumes in the hands on labs. We were freely able to remove and insert SATA drives. Simplicity is an understatement in this process. Any non-technical employee could monitor and easily handle administration of the Drobo products.  Watch the various videos linked in this post to see what I mean.

My Virtualization Admin Opinions
Data Robotics’ BeyondRaid is an exciting technology. it seems to be the perfect mix of ease to administer, redundancy, self healing, and self optimization. The announcement of the DroboElite product and the VMware HCL status legitimizes Data Robotics’ devices as a VMware storage option at an incredible price.

VMware customers with 16 or less ESX hosts probably can’t find as much storage for so little cost. I’m not usually focused on price so leave me comments for other options in the >10 TB usable disk for less than $5k category. I definitely want to know about those!

When considering alternatives to Drobo based on price, can you even buy a feature comparable server that could be used as an iSCSI target? Even if you can you will have to install an OS. There are cheap and free operating systems to serve iSCSI storage, but what about administering the hardware, drivers, firmware, etc when buying a server. Drobo still seems to be the best option to minimize the total cost of storage implementation and ownership.

Unfortunately there is only one power supply on all models of the Drobo. Hopefully this is on the short term roadmap. I do not recall being told as much, but I don’t recall anyone asking either. One power supply makes sense for the Drobo on the desktop, but in a rack in the data center redundant power is a must have for production deployments.

Today DroboElite seems to be a perfect fit for Dev/Test Labs. Perhaps even remote branch office scenario where a field engineer is not normally present.

It will be interesting to see if Drobo evolves their product line for enterprise customers or if an enterprise storage vendor acquires Data Robotics for integrating the BeyondRaid technology in their product lines.

If your curious about performance, fellow GestaltIT author and co Delegate at Tech Field Day, Devang Panchigar posted some basic testing results.

updated 12.06.09 – Chris Evans, another TFD delegate, posted about his experience implementing the DroboPro. Chris expresses the same concern about a single power supply as well as some other interesting quirks when deploying as an iSCSI SAN. Note that Evans’ post is not about the new, DroboElite, however. Check out the comments for a lengthy reply from Brad Meyer, Senior Product Marketing Manager, Data Robotics, Inc.

Disclosure: I received a vest/jacket and a certificate to purchase products at a discount from Data Robotics while at Tech Field Day. I have not purchased a Drobo as of this writing.

When you decide to maintain a server’s existing iSCSI partitions as a VM, there are several configuration considerations to plan for.

Multipathing Support for iSCSI is no longer needed in the VM

When you were configuring the iSCSI initiator chances are you used two physical network interface cards (NICs) for a redundant connection in the server operating system to the storage. You then used the NIC manufacturer’s drivers/management software to create a team and a virtual ip address. Your SAN was configured to allow an iSCSI initiator to connect via that NIC team virtual ip address.

As a VM that same team ip address will probably still be maintained as the initiator, but the need for two NICs and the former manufacturer’s drvers and software will be removed. The VM only needs a single vNIC to the iSCSI storage. The virtual host should be configured with a vSwitch mapped to two pNICs. Therefore the virtual host provides the redundant connection to the storage.

Be sure to remove the team configuration and the old NIC drivers and software.

Dedicate a vSwitch with it’s own pNICs for the VM iSCSI traffic

Separate the VM’s iSCSI traffic from the virtualization host’s iSCSI traffic. You could add an extra portgroup to your iSCSI vSwitch in VMware ESX for example, but ideally, you want 2 NICs dedicated to the host, and 2 other NICs dedicated to the VM(s). This requires separate vSwitches. This will maximize performance and provide redundancy.  

Consider the cables needed to the SAN switches

Before P2V each server needed 2 cables to the storage switches for redundancy. After P2V, each virtualization host will need 4 cables. Two of the cables will be replaced by the connection to the host’s dedicated LUNs where the VM’s operating systems and and other partitions are encapsulated. The second set of two cables will be for the VM’s initiator to access it’s own iSCSI partitions. 

Disconnect the iSCSI initiator before P2V

This is not a must do, but rather a safety net for the P2V migration process. Disconnecting the server’s iSCSI initiator ensures the LUNs you need to maintain will not be selectable as disks to be converted during the migration.

Be prepared to recreate any file shares and permissions

If you disconnect the iSCSI initiator as previously mentioned then be prepared to recreate any file permissions and shares that were configured. To be honest, I am not sure of the best way to prepare for this or if it’s even necessary, but in my experience I have had to recreate shares. Thank goodness it was never a complex user or department hierarchy as you can imagine the impact and administrator time needed that overlooking this would cause.

Check out this VMTN Communities thread on this topic too:

VMware Communities: P2V when server has a LUN through iSCSI? …

“More and more of my new implementations of VMware Infrastructure are being connected to iSCSI SANs (EMC, LeftHand, and Equallogic) and the question has come up about whether or not to spend extra dollars on TOE (TCPIP Offload Engine) Network cards.”

This made me realize that I have yet to implement an iSCSI SAN connection to an ESX Cluster with a hardware initiator. My customers have all used the native ESX iSCSI software initiator instead. So it made me wonder why am I not even considering the hardware iniator anymore, and should I present the option to my customers at all? Afterall, as Carlo points out, the number of supported TOE cards has increased from 2 to 16 (a table is provided at the end of this post)

I came up with a short list of reasons why I haven’t been using the iSCSI hardware initiators.

• the cost of iSCSI HBAs doesn’t justify the expected performance gains (around $200 for a 1GB nic versus over $1000 for a TOE per ESX host) unless you have a large number of heavy I/O VMs
• ESX 3.5 Clusters are designed with an N+1 number of hosts strategy so there is always extra available CPU cycles for the software initiator anyways.
• I did not know there are more iSCSI HBAs supported now!

I’m curious now about how prevalent the hardware initiator is used today. Maybe I’ll use this as a topic for a future VM /ETC poll.

In case you are wondering what a TOE or iSCSI HBA does, the following is from a slightly dated version of the vmware iSCSI design guide:

“iSCSI HBAs are interface adapters that provide high-performance offloading of both TCP and iSCSI
processing onto the adapter. Although this adds cost to the adapter, it delivers high-speed iSCSI transport with minimal CPU overhead.”

Be sure to check for yourself to get the current supported list, but here is a table of all the current VMware supported iSCSI HBAs taken (at the time of this post) from the online Hardware Compatibility Guide web application :

Notes:

1. Requires patch ESX-1001906.
2. Ethernet is not supported.
3. Supported on Intel E1000 and Broadcom running on a gigabit link only.
4. QLogic iSCSI adapters require minimum BIOS v1.09 and minimum firmware v2.00.00.38.

* Device Drivers:

1. megaraid_sas version 3.0.9
2. mptscsi_2xx version 2.06.48
3. qla4022 version 3.30
4. qla4022 version 3.24
5. qla4010 version 3.10
6. iscsimod version 3.6.3
7. iscsimod version 3.4.2

The section on disk I/O made me think – specifically about whether creating RDMs to raw disks was really a better choice than carefully planning VMFS LUNs. There is just too much convenience with using VMFS in my opinion, and I’ve never really been convinced that any report or testing has proven RDMs offer better performance than carefully planned VMFS. So, instead of a long comment on the vm0 post I decided to blog about it here at VM /ETC.

Be sure to read the entire post for the RAM and CPU recommendations, but here is the section that inspired me:

“I/O

I/O is by and large the biggest concern
when running a database on an ESX cluster. Your main concern here is
going to be ensuring that your database files are not impacted by I/O
operations on your other VMs. We can ensure this through Raw Device
Mappings.

Generally, for a high-performance database, you’ll want your data files, log files, and program/OS/backup files totempdb all live on separate spindles. To accomplish this, we do the following:

Data Files – RAID5 or RAID10 (RAID5 is more economical but you will suffer a hit on write performance).

Log Files – RAID10

TempDB – RAID10

OS/Program Files/Backup/File Share – This can be run from a standard VMDK.”

But do you really need RDMs? What about multiple, but dedicated and separate LUNS on the same, larger RAID5 or RAID10 arrays? Give the SQL server it’s own separate VMFS LUNs to spread out .vmdks across. The multiple .vmdks controlled by different virtual SCSI adapters combined with the fact that each LUN has it’s own I/O queue should provide for more than sufficient I/O performance. Add as many disks as possible to the larger arrays so you increase your spindle count.

Sure, if you have such a big data warehouse that tweaking the .vmdks and VMFS as described above won’t help it then probably the next best thing to do would be to dedicate an entire ESX host just for that “monster SQL VM”. Actually, dedicate 2 ESX hosts so you can have a “Monster SQL VM ESX Cluster”. You’d be looking at spending the cash for a multi-node physical cluster anyways, right? This way you get the mobility and conveninence of VMotion and all the administrative benefits of a VM.

As always, I ask, YOU, the vmetc.com readers to chime in with your opinions, preferences, and experience with heavy I/O virtual infrastructure design.

Although this post discusses NetApp, I imagine any NFS filer with similar features could be used in this same fashion. I’m curious for feedback on not only this solution, but the pros and cons of other vendor’s features in a similar scenario.

Here’s a logical diagram illustrating a NFS volume strategy followed by some high level details of each volume.

Click the image for a larger version.

2K3_Master – this is the home of a VM with a base install of Windows 2003 Standard. This volume and the VM it contains will be Flexcloned as new volumes for other Windows 2003 servers. After building the VM , this volume will not be mounted by the ESX hosts. To start the VMware VDM server will be the only Server created from this volume.

XP_Master – this is the home of a VM with a base install of Windows XP without any applications. This volume and the VM it contains will be cloned 4 times to create the 4 departmental template VMs. After building the VM, this volume will not be mounted by the ESX hosts.

ISO – a volume to store .iso files of installation and application CDs used by all of the VMs.

VDM – this is the home of the VMware VDM Server virtual machine created from a Flexclone of the 2K3_Master volume. The advantage of this Flexcloned volume is that it takes up no additional storage space to create, and the VM is thin provisioned by default (because it’s on NFS) and will only consume the space needed to install and configure VMware’s VDM instance.

XP_VM volumes for Marketing, Finance, Contractors, and IT Dept – 4 different volumes that are each created from a Flexclone of the XP_Master volume. Each volume will start with the default configuration of the XP VM, but then customized with unique application sets. The resulting 4 department VMs will be turned into VirtualCenter gold image templates. Desktop VMs will be deployed on these volumes from the VirtualCenter templates. Although the gold image departmental template will take up no extra space and be a thin provisioned .vmdk, the desktop VMs that are deployed via VirtualCenter cloning will be thick .vmdks. Deduplication will be enabled to reclaim as much space as possible.

SWAP – the templates, the desktop vms, and the server VM will all separate the Windows pagefile on to a dedicated .vmdk which will be stored on this volume. This is necessary to be able to exclude this transient data when scheduling backups and/or SAN replication.

Other General Design Notes:

• The NFS volumes can be extended or reduced as needed by adding more shelves of disks to the aggregate. (another term for the array)
• The 2K3_Master and XP_Master volumes can be Flexcloned to new volumes for future server or desktop VMs not currently provisioned in this solution. 32 volums is the max number of volumes ESX can connect to, however.
• VirtualCenter VM cloning was considered less complex by the IT staff (automated, sysprep-ed and customized deployment of new VMs) than duplicating VMs via NetApp volume Flexcloning
• The NetApp FAS3020 has a volume limitation of 1 TB for deduplication. Upgrading to a larger model increases the deduplication limit as shown in the following table provided by NetApp:

For some additional reading on NetApp NFS for VI, Flexclones, and deduplication check out the following posts by Scott Lowe. Scott has written a lot about NetApp Flexclones, configuring ESX for flexcloned volumes, and the pros and cons of SAN cloning compared to VirtualCenter cloning.

• http://blog.scottlowe.org/2007/05/11/how-to-provision-vms-using-netapp-flexclones/
• http://blog.scottlowe.org/2008/08/05/flexclones-versus-deduplication-with-vmware-infrastructure/
• http://blog.scottlowe.org/2007/05/15/netapp-flexclones-with-vmware-part-1/
• http://blog.scottlowe.org/2007/05/17/netapp-flexclones-with-vmware-part-2/

Also check out the following NetApp technical reports: (the design discussed in the post is basically a hybrid of these 2 reports. 3612 is for VDI recommendations, and 3428 is for server VM recommendations.

• http://www.netapp.com/us/library/technical-reports/tr-3612.html
• http://www.netapp.com/us/library/technical-reports/tr-3428.html

This AutoSupport indicates that there is a configuration issue in the FCP partner path.

Information:
============

[hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured.
[hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured – Host I/O access through a non-primary and non-optimal path was detected.

The AutoSupport email even explained what causes the misconfigured path.

This message occurs when the system detects that host I/O access to logical units (LUNs) is not through a primary path. NetApp clustered storage controllers allow access to LUNs through primary, optimized paths, and secondary, non-optimized paths.  Secondary(non-primary) paths provide access to LUNs through the partner storage controller’s FCP target ports. Under normal operating conditions, a host should not perform I/O to LUNs using a non-primary path. Access through a non-primary path should only occur when a host’s MPIO software detects a failure of all primary paths.

The mystery for me was “what caused a failure of the primary path?“. Believe it or not,

I’ve always been able to rely on the native ESX multipathing and keep the Service Console free from a SAN vendor’s agent. So, of course I did not have the NetApp Host Utilities installed. Now I know. A looming implication of this necessity is ESXi. Manual multipathing reconfiguration MUST ALWAYS OCCUR after every host reboot because ESXi does not have a Console OS. (See the link to kb44784 at the end of this post)

Scott Lowe and Duncan Epping helped me understand this scenario better via a combination of tweets and emails, and Scott has also since written a good explanation of the issue with ESXi and NetApp FC storage on his blog. Here’s a quote from Scott’s post explaining why it happens:

“… setting preferred paths from VirtualCenter is apparently not persistent across reboots because the VML LUN name is not used. Hence, when working with ESXi, neither method of setting the preferred path—vicfg-mpath or VirtualCenter—can provide persistent settings. Hence, the recommendation in the NetApp KB article.

But what about VMware ESX? The issue about using VirtualCenter remains, but esxcfg-mpath allows users to use the VML LUN name during the command to ensure that paths are persistent across reboots. In fact, there’s a note in the help screen of esxcfg-mpath (“esxcfg-mpath -h”) that mentions the fact that VMHBA names are not persistent across reboots, and users should use VML LUN names to be sure of consistency.”

Installing the NetApp Host Utilities avoids the VirtualCenter problem and eliminates the need to use the esxcfg-mpath Console command.

The following links are the relevant NetApp NOW documents I found while troubleshooting this issue. You will need a NOW account to access these articles.

• NetApp ESX Host Utilities 3.1 http://now.netapp.com/NOW/knowledge/docs/san/
• How to Verify VMWare ESX Fibre Channel Configurations with Multipathing I/O (MPIO) https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb30679
• What do FCP Partner Path Misconfigured messages mean? https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb30541
• Recommended Settings for VMware ESXi 3.5 Update 2 when connected to NetApp Storage Systems https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb44784

Also check out the VMTN Communities Forum thread Netapp host utilities for esxi when are they released?

However, Vizioncore and Datadomain offer a feasible backup solution using commonly available IP based storage that should make IT departments with even the tightest budgets look twice. Combining deduplication with LAN based VM backups to CIFS shares or NFS mounts, this solution provides a cost effective transition to full VM backups with minimal storage space consumed. Datadomain and Vizioncore established a certified partnership for VMware infrastructure in September of 2007 and therefore provide established solutions with technologies proven to work well together.

The image to the right was taken from Vizioncore’s solution .pdf titled Cost Efective Backup & Recovery & Storage for Virtualized Environments with Vizioncore Solutions and Data Domain Deduplication Appliances. The .pdf provides an overview of the design as well as recommendations for ensuring performance. This diagram illustrates a solution for not only VMs but also physical servers backed up as VMs by vRanger Pro’s P2V-DR feature.

To provide more specifics about this solution’s possibilities

Carlo Costanzo illustrates 3 design options for live virtual machine backups using vRanger Pro and CIFS or NFS datastores in his vmwareinfo.com post Backing up Virtual Machines using Vizioncore’s vRanger to a Data Domain.
Carlo does a great job providing the pros and cons for each option. I’ve trimmed Carlo’s original content a little for this section, but be sure to follow the link above for the post in it’s entirety.

NFS Export

Using Data Domain’s ability to create an NFS mount, you can configure the ESX hosts to use the Data Domain as a DataStore. This will allow the VizionCore vRanger machine to direct the individual ESX hosts to backup the VMs over a vKernel port group to the Data Domain server.

Pros:

1. The NFS export will be available to ALL ESX hosts so multiple ESX hosts could transmit backup traffic simultaneously increasing throughput and shortening the backup window.
2. Full backups would be taken each night which would make restores a little easier.
3. Vizioncore server could be virtualized since it would only manage and NOT PASS traffic.

Cons:

1. Virtual Machine backup traffic from the vKernel to the Data Domain will be transmitted over TCP connections.
2. NFS does not support incremental backups which mean that every night full backups would have to be transmitted over TCP to the Data Domain possibly increasing backup window times.

CIFS Share

Using Data Domain’s ability to create an CIFS mount, you can configure the Vizioncore vRanger Server to use the Data Domain as a Windows Share. This will allow the Vizioncore vRanger Server to directly backup the Virtual Machines over TCP to the Data Domain Server.

Pros:

1. Windows CIFS mounts are the only destination supported by vRanger for incremental backups. This will allow for much smaller backup windows.
2. Vizioncore vRanger server could be virtualized.

Cons:

1. vRanger server could prove to be a bottleneck in throughput and lengthen backup windows.
2. Backup traffic would be transmitted from the ESX server and to the Data Domain server over TCP connections.

VCB proxy with NFS Mount

A physical VCB proxy server with vRanger installed could be fiber connected into the SAN fabric allowing VCB/vRanger to perform LAN free backups from the SAN to a local NFS mount provided by the Data Domain device.

Pros:

1. The physical VCB server could perform incremental backups over the Fiber directly shortening backup times.
2. The NFS export could be mounted through the VCB Windows server supporting incremental backups.
3. The Backup Proxy would shoulder the processing load of the backups allowing the backup window to possibly extend into work hours.

Cons:

1. vRanger / VCB proxy would need to be a physical server connected via fiber into the fabric.