XenServer 6.5 SP1, Dell OpenManage and SNMP.

I've written before about XenServer 6.2, Dell OpenManager and SNMP and XenServer 6.1, Dell OpenManage and SNMP. I'm happy to report that the steps also work on XenServer 6.5 SP1 with very few modifications.

Logitech Media Server and Ubuntu Server 14.04.1 LTS

Logitech Media Server (LMS) was formerly known as SqueezeboxServer, SqueezeCenter and SlimServer. It's been running fine on my Ubuntu Server 13.10 box. I've recently decided to take the plunge and update to 14.04.1 LTS. There were some minor issues, but nothing critical, except for Logitech Media Server which refused to start.

TortoiseSVN with kerberos authentication

I'm a big fan of TortoiseSVN (TSVN, short) on Windows. Working with multiple SVN servers and different authentication methods has its pitfalls, however.
I just found out how to make the current TSVN (1.7.11 as of this writing) work with Kerberos authentication on a Linux server that uses krb5 as I previously described here.

check_email_delivery and SMTP SSL

For all of you who want/need/have to monitor an email server's performance or delay, there is a nice plugin for nagios out there written by Jonathan Buhacoff. It's not only a single plugin either, but a suite of scripts to check various smtp- and imap-related things.

discover Rancid

For those of you who don't know Rancid:

RANCID monitors a router's (or more generally a device's) configuration, including software and hardware (cards, serial numbers, etc) and uses CVS (Concurrent Version System) or Subversion to maintain history of changes.

Essentially, this allows to have full coverage over routers, firewalls, network devices configuration and changes. Every network administrator's dream, right? Well, it certainly seems too good to be true..! Let's find out.

I've set it up on a CentOS 5.4 box to tinker with it. Installation is pretty straightforward. Configuration is a bit tricky, but the sample files are well documented. There are a few pitfallse, however.

• "Show VLAN" output has different line-wrapping behaviour on different IOS versions; as diff is line oriented, this leads to unneccesary changes and notifications
• "call-forward all" directives are shown for cisco callmanagers or unified communication managers; if a user forwards his phone, this causes the configuration to change, and notifications to be sent out.
• The notification emails show diff output in plaintext; no colors, no highlighting
• Notificaions are only sent to a single email address; no distribution lists are possible

As rancid is comprised of a set of perl and bash scripts, the above issues shouldn't be too difficult to address, right? Well, there is  fork called rancid-git that provides support for git as well as various other improvements. Most notably is perhaps support for html-mail. So after tinkering a bit with vanilla rancid, I decided to give rancid-git a go. After all it can be used as a drop-in replacement.

Unfortunately it turned out that most of the advertised patches/enhancments are more trouble than they are worth. First of all rancid-git is based on an outdated version of rancid, which means that certain bugfixes and fixes are missing. I didn't see any ill effects with the devices that I monitored (mostly Cisco and proCurve) but ymmv.

So I fired up rancid-git and immedately got a notification with some profound config changes. It was an HTML mail, but there weren't any colors to be seen. I use subversion as repository, so the diff command doesn't show colors. Lucliky there is an easy workaround.

The problem now is, that rancid always sends out notifications, even when there are no changes. So this patch isn't ready for prime-time just yet.

I did like the detailed commit messages of rancid-git, however. So I decided to blend together my own patchset, based on vanilla rancid. Here's what I did:

• Comment out "show vlan", "show vlan-switch" in bin/rancid to get rid of the ever-changing vlan port memberships
• Added "/^ *(call-forward all)/ && next;" to the dog gone cool matches in bin/rancid to filter out unwanted phone forwardings in the notification mails
• Added "/.*coredumpinfo.*/ && next;" to ShowFlash in bin/rancid to get rid of the coredump bug on Cisco ASAs
• Added "svn propset "svn:mime-type" text/x-ios $router" to bin/control_rancid so cisco config files can be properly syntax highlighted in trac
• Added proper commit messages, blatantly copied from rancid-git
• To get syntax highlighting working in trac, install Pygments and this ios lexer; simply put it in the lexers subdirectory and run _mapping.py. If you get an error for a missing attribute, add a line like this "__all__ = ['IOSLexer']" between the imports and the class.
• For trac to recognize the mime-type and the IOS lexer, simply add "text/x-ios:ios" to the mime_map in trac.ini. I've tried to add the mime-type to the IOS lexer itself, but this didn't work.

This way, you can use the trac interface to view diffs and the configs. Instead of notification emails, simply rely on the RSS-feed of trac. Profit!

Apache on Linux and Single-Sign-On with Active Directory

There are many howtos and blog posts that describe how to get Apache on Linux to work against Active Directory with single-sign-on. I found none of them to be current and complete. After tinkering with this topic for a while and getting it to work on my Centos 5.4 box, I wanted to post what I did here.
In my case, I wanted single-sign-on to work for agents in OTRS. 

1. Use samba to join the AD
   No need to create a special user account and tinker with ktpass and the likes on windows. Everything that needs to be done in AD can be done from Linux through samba, and it works fine.
   I've tried to go down the route with the user-service account in AD and it was troublesome, to say the least. There is no resource kit for Windows 7 anymore, so there is no way to get ktpass.exe. The version from the Windows Server 2003 resource kit doesn't work in Windows 7 x64. In the end I had to run ktpass in the domain controller directly, which I'm sure isn't possible in all environments.
   
2. Compile mod_auth_kerb from source, as described by Ben Chapman here. Unfortunately the link he cites is no longer available. While authentication works with the mod_auth_kerb in the centos repositories, authorization seems to be troublesome.
3. Configure Kerberos and Apache accordingly.
   My config files are attached. krb5.conf is complete, the rest are just the relevant parts. Case matters to krb5! I tried to define aliases so it would be less picky about that, but to no avail. Test kerberos functionality with klist and kinit.
   

   kinit username@DOMAIN.LOCAL

   Should prompt for password, and exit silently if all worked. If not, go through your config again. I've set Apaches loglevel to debug, otherwise there is no useful output in case of errors with mod_auth_kerb. I had many issues of
   

   gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, No principal in keytab matches desired name)

   In my Apache error_log. It took me a while to figure them out. Somehow mod_auth_kerb wasn't using the correct name for the kerberos service principal, despite me using canonical names. I had to override it, and this finally worked.
4. Configure OTRS
   I first decided to set up authorization in OTRS via Active Directory through LDAP. The OTRS Wiki provides an excellent article that outlines the necessary changes in Kernel/Config.pm. This will not only authenticate existing users through AD, but also create new OTRS users and sync them from AD. For this to work no special Apache configuration is required. If this works, move on to enable Single-Sign-On in Apache for the otrs directory.
   Because I want to enable SSO for agents only, I had to use .htaccess files to specify security. So the only change in the Apache config for OTRS from the default is this:

   
   <Directory "/opt/otrs/bin/cgi-bin/">
       AllowOverride All
       Options +ExecCGI -Includes
       Order allow,deny
       Allow from all
    </Directory>
   
   

   This enables us to comfortable change security settings in a .htaccess file placed in /opt/otrs/bin/cgi-bin. Here it is:
   

   
   <files index.pl>
     AuthType Kerberos
     AuthName "OTRS Kerberos Login"
     KrbMethodNegotiate On
     KrbMethodK5Passwd On
     KrbAuthRealms DOMAIN.LOCAL
     KrbServiceName HTTP/webserver.domain.local
     Krb5KeyTab /etc/krb5.keytab
     KrbSaveCredentials Off
     KrbAuthoritative off
     KrbVerifyKDC off
   #  KerbLocalUserMapping on
     PerlAddVar ntdomain "DOMAIN.LOCAL pdc bdc"
     PerlSetVar defaultdomain DOMAIN.LOCAL
     PerlSetVar splitdomainprefix 1
     require valid-user
    </files>
   
   

   With these settings in place, we only need to adapt OTRS to use Apaches authentication. Add these lines to OTRS's Config/Kernel.pm:
   

   $Self-&gt;{'AuthModule'} = 'Kernel::System::Auth::HTTPBasicAuth';
   $Self-&gt;{'AuthModule::HTTPBasicAuth::ReplaceRegExp'} ='@DOMAIN.LOCAL';

   Despite the manually compiled mod_auth_kerb I still found that Apache passed through the users with the domain name. The easy fix in OTRS is to replace that part via regex, and all is peachy. As of now I'm happy it's working and don't want to spend more time finding out why this fix is required. ;-)
5. Client settings
   This may be obvious to some, but here it is anyway. In Internet Explorer you have to add the webserver.domain.local to the Local Intranet. Also, you have to enable Integrated Windows Authentication in the Advanced Settings. In Firefox you need to make sure to add "domain.local" to the config value network.negotiate-auth.trusted-uris. This will cause firefox to authenticate using SSPI. I use Firefox 5 and this worked like a charm. Specifying webserver.domain.local under network.automatic-ntlm-auth.trusted-uris alone will not work, as this uses NTLM, which is not Kerberos! Also, by using canonical names in Apache, it doesn't matter whether the clients access the serber by hostname alone, or by FQDN. I've had too many users who don't seem to care, so this is a good thing. Of course, when you add SSL to the equation, things look differently. YMMV.
   

/etc/samba/smb.conf

[global]
   workgroup = DOMAIN
   password server = pdc_fqdn bdc_fqdn
   realm = DOMAIN.LOCAL
   security = ads
   template shell = /sbin/nologin
   winbind use default domain = false
   winbind offline logon = false
   use kerberos keytab = true
   winbind enum users = yes
   winbind enum groups = yes
   winbind use default domain = yes

/etc/krb5.conf

 [logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = DOMAIN.LOCAL
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 forwardable = yes

[domain_realm]
 MYSERVER.DOMAIN.LOCAL = DOMAIN
DOMAIN.LOCAL = DOMAIN
.DOMAIN.LOCAL = DOMAIN
 domain.local = DOMAIN
.domain.local = DOMAIN
DOMAIN = DOMAIN
.DOMAIN = DOMAIN
domain = DOMAIN
.domain = DOMAIN

[realms]
 DOMAIN= {
  kdc = pdc_fqdn
  admin_server = pdc_fqdn
 }

 DOMAIN.LOCAL = {
  kdc = pdc_fqdn
  kdc = pdc_fqdn
 }

[appdefaults]
  pam = {
     debug = true
     ticket_lifetime = 36000
     renew_lifetime = 36000
     forwardable = true
#     krb4_convert = true
     krb5_get_tickets = false
  }
[login]
    krb4_convert = true
    krb4_get_tickets = false

Where pdc_fqdn would be somthing like DomainController.domain.local.

/etc/httpd/conf/httpd.conf

These are only the relevant parts, the rest is the default config.

KeepAlive On
ServerName webserver.domain.local
UseCanonicalName On
LogLevel debug

/etc/httpd/conf.d/auth_kerb.conf

LoadModule auth_kerb_module modules/mod_auth_kerb.so
Alias /private/ /path/to/my/folder

<location private="">
  Options Indexes
# SSLRequireSSL
  AuthType Kerberos
  AuthName "webserver Kerberos Login"
  KrbMethodNegotiate On
  KrbMethodK5Passwd On
  KrbAuthRealms DOMAIN.LOCAL DOMAIN
  KrbServiceName HTTP/webserver.domain.local
  Krb5KeyTab /etc/krb5.keytab
  KrbSaveCredentials On
  KrbAuthoritative off
  KrbVerifyKDC off
#  KerbLocalUserMapping on
  require valid-user

 </location>

XenServer iSCSI multipathing

Like many others, I use XenServer with a iSCSI SAN. To leverage a single point of failure, I enabled multipathing. When the IP or path of a host adapter changes, it can be challenging to get XenServer to recognize the new path.

The kb articles that Citrix provides about such issues are good pointers, but some commands are just plain wrong.

So this should serve as my little cheat sheet, and it might help others out there as well.

Check existing iSCSI sessions:

iscsiadm -m session

More verbosive information can be gathered with:

iscsiadm --mode node --loginall=all

If an IP has indeed changed, the old path needs to be forgotten. This can be done with:

iscsiadm –m node –T iqn.2008-01.com.openfiler:store01.lun01 –-portal=ip-address-of-the-storage -u

To reactivate a SR with multipathing one needs to get the SCSI ID from XenCenter, and then issue this:

mpathutil.py add

This step didn't work for me, however. Here's what did work in my case. Discover the targets manually:

iscsiadm -m discovery -t sendtargets --portal=ip-address-of-the-storage

Then, with the iSCSI IQN manually log on to the target:

iscsiadm -m node -T iqn.2008-01.com.openfiler:store01.lun01 --portal=ip-address-of-the-storage -l

This should yield in something like this:

Logging in to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]
Login to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]: successful

The added path should immediately be shown in iscsiadm -m session. If there are still issues, restart the multipath daemon:

 service multipathd restart

It's noteworthy that this doesn't disrupt any services. So it should be safe in a production environment.

If XenCenter still reports wrong number of active/inactive paths, a refresh can be forced by running:

/opt/xensource/sm/mpathcount.py

 

XenServer 5.6 and Dell OpenManage 6.5

Citrix recently released XenServer 5.6 Service Pack 2. While you probably have found out already, that upgrading from XenServer 5.6 FP1 only works through XenCenter and not by booting from the ISOs, those of you running XenServer on Dell boxes might appreciate some pointers in getting OMSA up and running again.

Dell actually recognizes XenServer as an operating system for their servers, so there is an ISO that can be used as supplemental pack when XenServer is installed. Yay! The same ISO can be used to install it on an existing system. Instructions can be found here, at Dell's Enterprise Technology Center. This site also has directions where the latest OMSA download can be found, so check it out!

I had OpenManagae 6.4 installed on all of my XenServers prior to upgrading them to 5.6 SP2. The installer failed because some conflicting packages were already installed. So I basically had to remove OMSA 6.4 before I could install 6.5. To save you the hassle of identifying the individual packages, here's how to get rid of it quickly:

yum remove srvadmin-smweb srvadmin-smcommon srvadmin-omilcore srvadmin-sm srvadmin-iws libsmbios sblim-sfcc

 After running the install.sh script on the downloaded ISO, it is necessary to bounce the srvadmin services by issuing:

/opt/dell/srvadmin/sbin/srvadmin-services stop
/opt/dell/srvadmin/sbin/srvadmin-services start

Thanks to jh for pointing out the mistake. 

CentOS, iperf and monit

I'm quite fond of FAN and so I have several CentOS-based Nagios systems to check various things. I recently needed to check the available bandwith between two servers, so I decided to use check_bandwidth which is based on iperf. So far so good.
Because I need to rely on iperf to be running on both ends, I ended up using monit.

"Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources."

Greg from the brain dive was kind enough to post the steps to set up iperf and monit on CentOS here. These steps almost led me to success, I had to do a few extra steps to make it work:

chmod +x /etc/init.d/iperfd
add "set daemon 60" to /etc/monit.conf

I found monit a bit weird to debug, because it doesn't write anything to it's logfile. So to find out why it's dying, I ran it in foreground with "monit -vI" which gives nice diagnostic output.

SS4200 woes

Well, ever since I had the SS4200 it just froze from time to time. I never had a chance to find out the details, because it mostly froze during the day when I was at work, or at night when I was out drin^H^H^H^H studying.

I always thought it was related to my custom kernel. Now it finally froze in my presence, so I could check it out. I opened the case and found the fans spinning rather slowly, and the CPU and northbridge were very hot. I figured the poor thing overheated.
So in fact I was right, because I enabled thermal management and CPU throtteling in my kernel, and loaded all those modules of course.
Add the fact that the Celeron 420 actually doesn't support SpeedStep, and you've got a few kernel moduels going crazy.
I'll remove the thermal and cpu throtteling modules for the time being and see whether that improves stability.

In the long run I want a NAS that doesn't consume more power than really necessary, so I'll probably switch the CPU to a Celeron E1400 or similiar. Those already got a 2nd core and support SpeedStep. While that's got a higher TDP, I think the power consumption is going to be lower, over all.

Guess we'll have to wait and see.

Linux 2.4 hacking?

I need to hack on a Linux 2.4 kernel for an embedded system. I thought of using one of my Linux boxes to do the work. But not only did the trusted "make menuconfig" bail on me, there were countless compile errors, even on the vanilla kernel! I gathered this is due to a version conflict with ncurses.

I had to resort to an old Debian, that's based on 2.4 and still comes with the old gcc and whatnot. Works like a charm!

I'm sure I am not the only 2.4 hacker using a modern Linux distro. Any hints on how to make it work?

Debian on the SS4200

And debian it is!
I've blatantly plugged the DOM into the IDE socket of a standard PC and fired up the ubuntu server installer. You have to compile a custom kernel however, since the stock ubuntu kernel lacks the IDE drivers for the SS4200.

A custom kernel needs to have IDE_GENERIC, IDE_PIIX and BLK_DEV_IDEDISK for the ide controller to be detected.

The E1000 module has the network driver, and the BLK_DEV_DS is quite nice to have USB storage support.

To compile and install I've stuck to this guide, I've used a vanilla linux kernel (2.6.27.9) though.

The tough part wasn't compiling the kernel though. Its getting the BIOS to always boot from the DOM. The RS232 header didn't work for me as described here. I had to resort to a PCIe x1 graphics adapter and do it the old-fashioned way. I've tried to use a straight through cable without success.