XenServer iSCSI multipathing

Like many others, I use XenServer with a iSCSI SAN. To leverage a single point of failure, I enabled multipathing. When the IP or path of a host adapter changes, it can be challenging to get XenServer to recognize the new path.

The kb articles that Citrix provides about such issues are good pointers, but some commands are just plain wrong.

So this should serve as my little cheat sheet, and it might help others out there as well.

Check existing iSCSI sessions:

iscsiadm -m session

More verbosive information can be gathered with:

iscsiadm --mode node --loginall=all

If an IP has indeed changed, the old path needs to be forgotten. This can be done with:

iscsiadm –m node –T iqn.2008-01.com.openfiler:store01.lun01 –-portal=ip-address-of-the-storage -u

To reactivate a SR with multipathing one needs to get the SCSI ID from XenCenter, and then issue this:

mpathutil.py add

This step didn't work for me, however. Here's what did work in my case. Discover the targets manually:

iscsiadm -m discovery -t sendtargets --portal=ip-address-of-the-storage

Then, with the iSCSI IQN manually log on to the target:

iscsiadm -m node -T iqn.2008-01.com.openfiler:store01.lun01 --portal=ip-address-of-the-storage -l

This should yield in something like this:

Logging in to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]
Login to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]: successful

The added path should immediately be shown in iscsiadm -m session. If there are still issues, restart the multipath daemon:

 service multipathd restart

It's noteworthy that this doesn't disrupt any services. So it should be safe in a production environment.

If XenCenter still reports wrong number of active/inactive paths, a refresh can be forced by running:

/opt/xensource/sm/mpathcount.py

 

XenServer 5.6 and Dell OpenManage 6.5

Citrix recently released XenServer 5.6 Service Pack 2. While you probably have found out already, that upgrading from XenServer 5.6 FP1 only works through XenCenter and not by booting from the ISOs, those of you running XenServer on Dell boxes might appreciate some pointers in getting OMSA up and running again.

Dell actually recognizes XenServer as an operating system for their servers, so there is an ISO that can be used as supplemental pack when XenServer is installed. Yay! The same ISO can be used to install it on an existing system. Instructions can be found here, at Dell's Enterprise Technology Center. This site also has directions where the latest OMSA download can be found, so check it out!

I had OpenManagae 6.4 installed on all of my XenServers prior to upgrading them to 5.6 SP2. The installer failed because some conflicting packages were already installed. So I basically had to remove OMSA 6.4 before I could install 6.5. To save you the hassle of identifying the individual packages, here's how to get rid of it quickly:

yum remove srvadmin-smweb srvadmin-smcommon srvadmin-omilcore srvadmin-sm srvadmin-iws libsmbios sblim-sfcc

 After running the install.sh script on the downloaded ISO, it is necessary to bounce the srvadmin services by issuing:

/opt/dell/srvadmin/sbin/srvadmin-services stop
/opt/dell/srvadmin/sbin/srvadmin-services start

Thanks to jh for pointing out the mistake. 

iperf woes

Yesterday I blogged about iperf, CentOS and monit. Well, it turns out it was a bit early to report success. Even with monit, iperf is inherently unstable on CentOS. Monit's log shows this:

[CET Dec 21 11:26:14] error    : 'iperf' process with pid 20557 is a zombie
[CET Dec 21 11:26:14] error    : 'iperf' failed, cannot open a connection to INET[127.0.0.1:5001] via TCP
[CET Dec 21 11:26:14] info     : 'iperf' trying to restart
[CET Dec 21 11:26:14] info     : 'iperf' stop: /etc/init.d/iperfd
[CET Dec 21 11:26:44] error    : 'iperf' failed to stop
[CET Dec 21 11:27:44] error    : 'iperf' process with pid 20557 is a zombie
[CET Dec 21 11:27:44] error    : 'iperf' failed, cannot open a connection to INET[127.0.0.1:5001] via TCP
[CET Dec 21 11:27:44] info     : 'iperf' trying to restart
[CET Dec 21 11:27:44] info     : 'iperf' stop: /etc/init.d/iperfd
[CET Dec 21 11:28:14] error    : 'iperf' failed to stop
[CET Dec 21 11:29:14] error    : 'iperf' process with pid 20557 is a zombie

And ps shows iperf is defunct. This is obviously a bad situation. When I run iperf without -D parameter, it responds one time and then shuts down. But I couldn't get this behaviour to work with iperfd and monit properly. For the record, I used "iperf version 2.0.4 (7 Apr 2008) pthreads".

So in the end I resorted to use iperf on a Windows box and run it as a service. So far there have been no problems.

CentOS, iperf and monit

I'm quite fond of FAN and so I have several CentOS-based Nagios systems to check various things. I recently needed to check the available bandwith between two servers, so I decided to use check_bandwidth which is based on iperf. So far so good.
Because I need to rely on iperf to be running on both ends, I ended up using monit.

"Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources."

Greg from the brain dive was kind enough to post the steps to set up iperf and monit on CentOS here. These steps almost led me to success, I had to do a few extra steps to make it work:

chmod +x /etc/init.d/iperfd
add "set daemon 60" to /etc/monit.conf

I found monit a bit weird to debug, because it doesn't write anything to it's logfile. So to find out why it's dying, I ran it in foreground with "monit -vI" which gives nice diagnostic output.

XenServer and Dell OMSA

As I have written before, there is a way to install ans use Dell OpenManage Server Administrator on vanilla Citrix XenServer. I have got it running on three production servers for five months now without issues. One is a R710, the others are PowerEdge 2950.


I've had to update the firmware of one RAID controller in order for it to work with OMSA. This was tricky, as there is no support for doing this in Linux. So I've had to use a Windows live CD, and upgrade the firmware from there. Worked like a charm.