XenServer 6.5 SP1, Dell OpenManage and SNMP.

I've written before about XenServer 6.2, Dell OpenManager and SNMP and XenServer 6.1, Dell OpenManage and SNMP. I'm happy to report that the steps also work on XenServer 6.5 SP1 with very few modifications.

XenServer 6.2, Dell OpenManage and SNMP

I've written before about XenServer 6.1, Dell OpenManage and SNMP. I'm happy to report that the exact same steps also work on XenServer 6.2 SP1.

XenServer 6.1, Dell OpenManage and SNMP

I finally updated our productive environment to XenServer 6.1. The upgrade went without a hitch, but there are still many issues with Windows VMs and the XenTools. In the end completely removing the XenTools and reinstalling them did the trick.

Anyway, this post is about Dell OpenManage and SNMP. I've found a spot-on post in the Citrix forums here - thanks Josh Phillips. Read on for the details.

XenServer 6.1

As I've previously written about here, I'm using the Citrix XenServer. I've been using the free version in a production environment for a few years now. Sure, there were minor issues or hiccups, but no deal breakers.

I'm still running XenServer 5.6, although XenServer 6 and even 6.1 were already released.
Why didn't I upgrade yet?

XenServer and "The VDI is not available"

I've been happily running XenServer for a few years now. Last week I encountered a weird issue with seemingly corrupted VDIs, which turned out to be fine after a server reboot.
In the aftermath I had a corrupted storage repository, that manifested like this:

• Unable to scan the SR:

Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=Error scanning VDI <uuid> ]

• Unable to copy VDIs from or to the SR
  This was nasty, because this meant it wasn't possible to backup virtual disks.

After reading comments on the XenServer forums for many hours I found a working solution. First of all check the file /var/log/SMlog on the pool master server and then trigger a rescan of the bad SR. In my case I had a VDI with an invalid footer.
The log looks like this:

***** VHD scan error: vhd=VHD-<uuid> scan-error=-22 error-message='invalid footer'
*** vhd-scan error: <uuid>
Raising exception [46, The VDI is not available [opterr=Error scanning VDI <uuid>]]

I verified this by running

/usr/sbin/vhd-util scan -f -m "VHD-*" -l VG_XenStorage-<storage-reporitory-uuid> -p

Which, amongst others, gave me this line:

vhd=VHD-<uuid> scan-error=-22 error-message='invalid footer'

 Now many people recommended running vdi-forget to get rid of this VDI. I tried that, but that didn't fix anything. Others recommended running vdi-destroy. After I ran vdi-forget I wasn't able to run vdi-destroy anymore. So I had to work it out using lvremove like this:

lvremove /dev/VG_XenStorage-<storage-repository-uuid>/VHD-<uuid>

This gave me a few warnings like "open failed: Read-only file system" but in the end it did the trick. I re-scanned the storage repository and it finally worked. A few orphaned VDIs showed up which I promptly removed. Subsequent rescans of my SR coalesced VDIs without problems.

I found these links to be helpful: CTX122001 and Thread: Issue with reclaiming disk space in xenserver 6.0.

For the record, I'm running XenServer 5.6 SP2.

XenServer iSCSI multipathing

Like many others, I use XenServer with a iSCSI SAN. To leverage a single point of failure, I enabled multipathing. When the IP or path of a host adapter changes, it can be challenging to get XenServer to recognize the new path.

The kb articles that Citrix provides about such issues are good pointers, but some commands are just plain wrong.

So this should serve as my little cheat sheet, and it might help others out there as well.

Check existing iSCSI sessions:

iscsiadm -m session

More verbosive information can be gathered with:

iscsiadm --mode node --loginall=all

If an IP has indeed changed, the old path needs to be forgotten. This can be done with:

iscsiadm –m node –T iqn.2008-01.com.openfiler:store01.lun01 –-portal=ip-address-of-the-storage -u

To reactivate a SR with multipathing one needs to get the SCSI ID from XenCenter, and then issue this:

mpathutil.py add

This step didn't work for me, however. Here's what did work in my case. Discover the targets manually:

iscsiadm -m discovery -t sendtargets --portal=ip-address-of-the-storage

Then, with the iSCSI IQN manually log on to the target:

iscsiadm -m node -T iqn.2008-01.com.openfiler:store01.lun01 --portal=ip-address-of-the-storage -l

This should yield in something like this:

Logging in to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]
Login to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]: successful

The added path should immediately be shown in iscsiadm -m session. If there are still issues, restart the multipath daemon:

 service multipathd restart

It's noteworthy that this doesn't disrupt any services. So it should be safe in a production environment.

If XenCenter still reports wrong number of active/inactive paths, a refresh can be forced by running:

/opt/xensource/sm/mpathcount.py

 

XenServer 5.6 and Dell OpenManage 6.5

Citrix recently released XenServer 5.6 Service Pack 2. While you probably have found out already, that upgrading from XenServer 5.6 FP1 only works through XenCenter and not by booting from the ISOs, those of you running XenServer on Dell boxes might appreciate some pointers in getting OMSA up and running again.

Dell actually recognizes XenServer as an operating system for their servers, so there is an ISO that can be used as supplemental pack when XenServer is installed. Yay! The same ISO can be used to install it on an existing system. Instructions can be found here, at Dell's Enterprise Technology Center. This site also has directions where the latest OMSA download can be found, so check it out!

I had OpenManagae 6.4 installed on all of my XenServers prior to upgrading them to 5.6 SP2. The installer failed because some conflicting packages were already installed. So I basically had to remove OMSA 6.4 before I could install 6.5. To save you the hassle of identifying the individual packages, here's how to get rid of it quickly:

yum remove srvadmin-smweb srvadmin-smcommon srvadmin-omilcore srvadmin-sm srvadmin-iws libsmbios sblim-sfcc

 After running the install.sh script on the downloaded ISO, it is necessary to bounce the srvadmin services by issuing:

/opt/dell/srvadmin/sbin/srvadmin-services stop
/opt/dell/srvadmin/sbin/srvadmin-services start

Thanks to jh for pointing out the mistake. 

XenServer and Dell OMSA - addendum

As I've written here it is indeed possible to install Dell OMSA on a XenServer 5.6 host without DDK. I've taken a brand new Dell R710 machine and decided to give it a go.
After I followed the instrcutions I encountered some weird issues though. The OMSA Webinterface was buggy, and only a few sensors were available.

For instance

# omreport chassis temps
Error! No temperature probes found on this system.

certainly didn't seem right. After some googling I thought I found the cuplrit in missing modules, namely dcdbas and dcdipm. I found only dcdbas was present, but not loaded:

# modprobe -l -v|grep dcd
/lib/modules/2.6.27.42-0.1.1.xs5.6.0.44.111158xen/kernel/drivers/firmware/dcdbas.ko

Loading it didn't help though. After more googling I found that openipmi is required, but isn't being installed during the OMSA install. So I installed it, restarted all omsa services and everything works fine:

# omreport chassis temps
Temperature Probes Information

------------------------------------
Main System Chassis Temperatures: Ok
------------------------------------

Index : 0
Status : Ok
Probe Name : System Board Ambient Temp
Reading : 28.0 C
Minimum Warning Threshold : 8.0 C
Maximum Warning Threshold : 42.0 C
Minimum Failure Threshold : 3.0 C
Maximum Failure Threshold : 47.0 C