Upgrading XenServer 6.2 to 6.5 with Dell Equilogic iSCSI LUNs

The Dell Equilogic only expose a single iSCSI target and one IQN, through one IP address.
With default configurations of XenServer, it is not possible to establish multiple iSCSI connections if the storage array exposes only a single iSCSI target, and a single IQN, through a single IP address. By default, XenServer establishes only one connection per iSCSI target.

For more information about the configuration of XenServer 6.2 iSCSI Configuration read my article about the configuration of Dell Equilogic on XenServer 6.2.
Or read this instructions: http://support.citrix.com/article/CTX138429

Some hardware summaries:

Each server is connected with interface eth2 and eth3 to the iSCSI network. I use Dell Blade Servers and the storage unit is an Dell Equilogic PSM4110.

Upgrade XenServer 6.2 to 6.5 with multipathing configuration on Dell Equilogic

First I started with the standard upgrade process. I stopped HA on the cluster and moved all virtual machines from the master to slave hosts.

Than I rebooted the master with the XenServer 6.5 install CD and select „Upgrade“. After the upgraded master has rebooted and reconnected to the new XenCenter the problems began:

First you see all your iSCSI LUNs with a small red icon. In the detailed view you can see that the master has only one iSCSI Connection to the LUNs and not 2 like the slave hosts.

The first reason for that is, that you have lost your complete iSCSI configuration during the upgrade process. The first step is to reconfigure the iSCSI on the upgraded host.

1. Change the /etc/multipath.conf

device {
  vendor "EQLOGIC"
  product "100E-00"
  path_grouping_policy multibus
  getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
  path_checker tur
  failback immediate
  path_selector "round-robin 0"
  rr_min_io 3
  rr_weight priorities
}

2. Reconfigure the two iSCSI Interfaces:

# iscsiadm -m iface --op new -I c_iface2 
# iscsiadm -m iface --op=update -I c_iface2 -n iface.net_ifacename -v xenbr2

# iscsiadm -m iface --op new -I c_iface3
# iscsiadm -m iface --op=update -I c_iface3 -n iface.net_ifacename -v xenbr3

On XenServer 6.2 this was all you had to do to configure your iSCSI. But now after a reboot of your host or only after a restart of multipath ( /etc/init.d/multipatfd restart ) you still will have one iSCSI connection.

For more informations read this article.

There should be two iSCSI connections / sessions over each single interface / IP of your iSCSI interfaces.
But there is only one default connection over one of your interfaces.

First rediscover nodes (LUNs) with new interfaces:

iscsiadm -m discovery -t st -p :3260

This command lists all iSCSI LUNs of your Storage device like this example:

:3260,1 iqn.2014-09.com.equallogic:0-af1ff6-23d42f8d4-29e0000002e54e44-LUN01
:3260,1 iqn.2014-09.com.equallogic:0-af1ff6-c3142f8d4-0c61000003154eec-LUN02
:3260,1 iqn.2014-09.com.equallogic:0-af1ff6-16442f8d4-39a0000003754ef2-LUN03

Now login with new discoverd interfaces:

iscsiadm -m node iqn.2014-09.com.equallogic:0-af1ff6-23d42f8d4-29e0000002e54e44-LUN01 --login

Now you have 3 iSCSI connections one for each interface and the default connection one. On the storage unit you see 3 connections as well. One for each IP-Address and one IP with an extra connection (the old default one).

You can check this with:

multipath -ll

This should show you three active connections:

af1ff623d42f8d429e0000002e54e44 dm-0 EQLOGIC,100E-00
size=300M features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:0:0 sdb 8:16 active ready running
  `- 2:0:0:0 sdc 8:32 active ready running
  `- 3:0:0:0 sdd 8:48 active ready running

Inside of the XenCenter the LUNs now show 3 of 3 connections and they are healhy again.

Now remove all default connections from your XenServer, first logoff:

iscsiadm -m node -I default -u

and delete default interface from nodes:

iscsiadm -m node -I default --op=delete

You will notice your LUNs are again red because now there are 2 of 3 connections. On the storage unit you see one active connection per IP-Address.

But how reset XenServer from 2 of 3 connections to 2 of 2 connections ?

After some research i found a python script on the XenServer in which the connection ( max and total ) for the iSCSI SR LUNs are counted and saved.

/opt/xensource/sm/mpathcount.py

If you notice that ‘multipath –ll’ shows you all the paths correctly and XenCenter shows some paths are not connected, you can refresh the information by executing this script.

(More details: http://support.citrix.com/article/CTX118791)

But there is a small bug in this script: It only increses the maximal number of iSCSI Connections for an LUN. It NEVER decreased this value !!

Here is the code and why it doesn’t work. Max is only set to new total if total is larger than the current max.

        if rdaclun:
            (total, count) = get_path_count(SCSIid)
        else:
            count = get_path_count(SCSIid)
            total = get_path_count(SCSIid, active=False)
        max = 0
        if len(entry) != 0:
            try:
                p = entry.strip('[')
                p = p.strip(']')
                q = p.split(',')
                max = int(q[1])
            except:
                pass
        if total > max:
            max = total
        newentry = [count, max]

But you can set this value by hand via xe sr-param-set. First list all LUNs and make sure you only list the remote LUNs for the current host:

xe pbd-list host-name-label=XenServer_Hostname

here is an example of one LUN:

uuid ( RO) : 		c8764684-7b25-49f0-9661-ce5e0601cb96
host-uuid ( RO): 	b25e7965-83fe-4724-8129-68845b78d048
sr-uuid ( RO): 	 	688b23d5-93c6-b120-e458-052800540cd0
device-config (MRO):	multihomelist: 192.168.4.10:3260; SCSIid: af1ff623d42f8d429e0000002e54e44; targetIQN: iqn.2014-09.com.equallogic:0-af1ff6-23d42f8d4-29e0000002e54e44-LUN01; target: ; port: 3260
currently-attached ( RO): true

Than list the other-config of this LUN:

xe pbd-param-get uuid=c8764684-7b25-49f0-9661-ce5e0601cb96 param-name=other-config

this should look like:

storage_driver_domain: OpaqueRef:70f64663-3e19-3d34-62df-454e6ac49ba0; 
iscsi_sessions: 2; 
mpath-360fff14a8d2f14c3ec4e15030000600c: [2, 3]; 
multipathed: true

You see the [2, 3] ? Now update it to [2, 2]:

xe pbd-param-set uuid=c8764684-7b25-49f0-9661-ce5e0601cb96 other-config:mpath-360fff14a8d2f14c3ec4e15030000600c="[2, 2]" 

Watch carefully on the IDs and UUIDs !

Now your LUN should be healthy again and should show 2 of 2 connections.

Woohooo the configuration of your first host is completed ! Now move all virtual machines from one slave to other hosts (maybe back to the master) and connect the install CD to this host and reboot it.
After the upgrade process you have to repeat the iSCSI Coniguration steps from above for this host.

Congratulations you upgraded your XenServer 6.2 Cluster with an DELL Equilogic iSCSI storage to XenServer 6.5 !

If you want to install Dell OpenManage on your XenServer Hosts, please read this artice.

www.mogilowski.net/2015/03/03/dell-openmanage-omsa-7-4-on-xenserver-6-5/