CollabNet Subversion Edge cont'd

I already explained Collabnet Subversion Edge and my experience installing it. Now I want to focus on installing Trac (with some hacks).
I really thought this should be easy, but it turns out that there are a lot of pitfalls. So for any other poor souls who are not fluent in python (and for my own reference) here it goes.

Installing Trac

As you know, trac has some requirements. Namely setuptools and genshii. There are a few packages to download for Windows here. I didn't know which version of python came with subversion edge, so I checked. There is a folder named csvn/python25. So it seems to be python 2.5. But wait - there is a python27.dll in there. and some other files indicating python 2.7. So which is it?

D:\csvn\Python25>python.exe --version
Python 2.7.1

 Alright, that settles it. But why did Collabnet not rename the folder..?
So I downloaded the setuptools package for python 2.7 and win32. The installer failed to run, it sais it couldn't find any python 2.7 installation.

Apparently a regular python installer will create some registry entries so that the system recognizes the python installation. Collabnet subversion edge doesn't do this anymore due some complaints.
So I had to go in there and manually create the relevant entries. They were already present from the old python 2.5 installation, which I wanted to get rid of anyway. So I changed the version to 2.7 and updated the paths to point to D:\csvn\python25. This let me install setuptools, genshii and finally trac too.

Configuring Trac

That part was straight-forward. Like with subversion, I just copied the relevant parts of my VisualSVN server's apache config and pasted it to the main httpd.conf file. Here is the relevant snippet:

<Location /trac>
 SetHandler mod_python
 PythonInterpreter main_interpreter
 PythonHandler trac.web.modpython_frontend
 PythonOption TracEnvParentDir D:\Trac
 PythonOption TracUriRoot /trac

 AuthName "Trac"
 AuthType SSPI
 SSPIAuth On
 SSPIAuthoritative On
 SSPIDomain jerich
 SSPIOfferBasic On
 SSPIOmitDomain On
 require group DOMAIN\Trac-Users
</Location>

All relevant modules are already loaded, so that's it.

Collabnet Subversion Edge

I have been quite fond of VisualSVN Server on Windows, since it was an easy way to setup subversion with a nice means to administer repositories and users.

As of lately, I grew tired because of the way they mangled apache, subversion, python and trac together. While searching for an alternative I found out that Collabnet offers exactly what I need: Apache bundled with Subversion, Python and other stuff. Very nice! This bundle is called Subversion Edge.

I decided to take the plunge and update the production VisualSVN server. After all: How hard can it be?

The installation went fine, and the collabnet subversion edge web-interface greeted me.
I had tweaked the configuration of VisualSVN server to allow single-sign-on with SSPI, so I naturally wanted to copy the relevant parts of the config. Once one tinkers with the config files, the web-interface can no longer be used to make changes to those parts. That's fine by me.

After this was taken care of I fired up Collabnet's Apache service and started to dump my subversion repositories one by one. Instead of manually loading them with svnadmin, the web interface of subversion edge provides takes care of that. Simply point it to the dump file and go ahead. That was easy!

Setting up the server, finetuning apache and migrating my three repositories took less than one hour! ;-)

rancid updates

I'm still happy about rancid, that I blogged about previously. I used one more change by Stefan Kublik that is also present in rancid-git:
Remove memory addresses from "show controllers" to avoid unnecessary change notifications


This is done by changing a line in bin/rancid from 

ProcessHistory("INT","","","!Interface: $INT$1\n") && next;

to

ProcessHistory("INT","","","!Interface: $INT\n") && next;

In other words, just remove the $1 and you will change these lines:

!Interface: FastEthernet0/0, GT96K FE ADDR: 63F52098, FASTSEND: 62084C3C, MCI_INDEX: 0

into these:

!Interface: FastEthernet0/0,

In other news, it turned out to be tricky to have rancid automatically updating a trac repository with post-commit hooks. Trac is very picky when it comes to permissions, and you need write access to the trac environment and to the trac logfile to do that. Try out the "trac-admin changeset" command works before debugging svn-hooks, ok?

discover Rancid

For those of you who don't know Rancid:

RANCID monitors a router's (or more generally a device's) configuration, including software and hardware (cards, serial numbers, etc) and uses CVS (Concurrent Version System) or Subversion to maintain history of changes.

Essentially, this allows to have full coverage over routers, firewalls, network devices configuration and changes. Every network administrator's dream, right? Well, it certainly seems too good to be true..! Let's find out.

I've set it up on a CentOS 5.4 box to tinker with it. Installation is pretty straightforward. Configuration is a bit tricky, but the sample files are well documented. There are a few pitfallse, however.

• "Show VLAN" output has different line-wrapping behaviour on different IOS versions; as diff is line oriented, this leads to unneccesary changes and notifications
• "call-forward all" directives are shown for cisco callmanagers or unified communication managers; if a user forwards his phone, this causes the configuration to change, and notifications to be sent out.
• The notification emails show diff output in plaintext; no colors, no highlighting
• Notificaions are only sent to a single email address; no distribution lists are possible

As rancid is comprised of a set of perl and bash scripts, the above issues shouldn't be too difficult to address, right? Well, there is  fork called rancid-git that provides support for git as well as various other improvements. Most notably is perhaps support for html-mail. So after tinkering a bit with vanilla rancid, I decided to give rancid-git a go. After all it can be used as a drop-in replacement.

Unfortunately it turned out that most of the advertised patches/enhancments are more trouble than they are worth. First of all rancid-git is based on an outdated version of rancid, which means that certain bugfixes and fixes are missing. I didn't see any ill effects with the devices that I monitored (mostly Cisco and proCurve) but ymmv.

So I fired up rancid-git and immedately got a notification with some profound config changes. It was an HTML mail, but there weren't any colors to be seen. I use subversion as repository, so the diff command doesn't show colors. Lucliky there is an easy workaround.

The problem now is, that rancid always sends out notifications, even when there are no changes. So this patch isn't ready for prime-time just yet.

I did like the detailed commit messages of rancid-git, however. So I decided to blend together my own patchset, based on vanilla rancid. Here's what I did:

• Comment out "show vlan", "show vlan-switch" in bin/rancid to get rid of the ever-changing vlan port memberships
• Added "/^ *(call-forward all)/ && next;" to the dog gone cool matches in bin/rancid to filter out unwanted phone forwardings in the notification mails
• Added "/.*coredumpinfo.*/ && next;" to ShowFlash in bin/rancid to get rid of the coredump bug on Cisco ASAs
• Added "svn propset "svn:mime-type" text/x-ios $router" to bin/control_rancid so cisco config files can be properly syntax highlighted in trac
• Added proper commit messages, blatantly copied from rancid-git
• To get syntax highlighting working in trac, install Pygments and this ios lexer; simply put it in the lexers subdirectory and run _mapping.py. If you get an error for a missing attribute, add a line like this "__all__ = ['IOSLexer']" between the imports and the class.
• For trac to recognize the mime-type and the IOS lexer, simply add "text/x-ios:ios" to the mime_map in trac.ini. I've tried to add the mime-type to the IOS lexer itself, but this didn't work.

This way, you can use the trac interface to view diffs and the configs. Instead of notification emails, simply rely on the RSS-feed of trac. Profit!

Apache on Linux and Single-Sign-On with Active Directory

There are many howtos and blog posts that describe how to get Apache on Linux to work against Active Directory with single-sign-on. I found none of them to be current and complete. After tinkering with this topic for a while and getting it to work on my Centos 5.4 box, I wanted to post what I did here.
In my case, I wanted single-sign-on to work for agents in OTRS. 

1. Use samba to join the AD
   No need to create a special user account and tinker with ktpass and the likes on windows. Everything that needs to be done in AD can be done from Linux through samba, and it works fine.
   I've tried to go down the route with the user-service account in AD and it was troublesome, to say the least. There is no resource kit for Windows 7 anymore, so there is no way to get ktpass.exe. The version from the Windows Server 2003 resource kit doesn't work in Windows 7 x64. In the end I had to run ktpass in the domain controller directly, which I'm sure isn't possible in all environments.
   
2. Compile mod_auth_kerb from source, as described by Ben Chapman here. Unfortunately the link he cites is no longer available. While authentication works with the mod_auth_kerb in the centos repositories, authorization seems to be troublesome.
3. Configure Kerberos and Apache accordingly.
   My config files are attached. krb5.conf is complete, the rest are just the relevant parts. Case matters to krb5! I tried to define aliases so it would be less picky about that, but to no avail. Test kerberos functionality with klist and kinit.
   

   kinit username@DOMAIN.LOCAL

   Should prompt for password, and exit silently if all worked. If not, go through your config again. I've set Apaches loglevel to debug, otherwise there is no useful output in case of errors with mod_auth_kerb. I had many issues of
   

   gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, No principal in keytab matches desired name)

   In my Apache error_log. It took me a while to figure them out. Somehow mod_auth_kerb wasn't using the correct name for the kerberos service principal, despite me using canonical names. I had to override it, and this finally worked.
4. Configure OTRS
   I first decided to set up authorization in OTRS via Active Directory through LDAP. The OTRS Wiki provides an excellent article that outlines the necessary changes in Kernel/Config.pm. This will not only authenticate existing users through AD, but also create new OTRS users and sync them from AD. For this to work no special Apache configuration is required. If this works, move on to enable Single-Sign-On in Apache for the otrs directory.
   Because I want to enable SSO for agents only, I had to use .htaccess files to specify security. So the only change in the Apache config for OTRS from the default is this:

   
   <Directory "/opt/otrs/bin/cgi-bin/">
       AllowOverride All
       Options +ExecCGI -Includes
       Order allow,deny
       Allow from all
    </Directory>
   
   

   This enables us to comfortable change security settings in a .htaccess file placed in /opt/otrs/bin/cgi-bin. Here it is:
   

   
   <files index.pl>
     AuthType Kerberos
     AuthName "OTRS Kerberos Login"
     KrbMethodNegotiate On
     KrbMethodK5Passwd On
     KrbAuthRealms DOMAIN.LOCAL
     KrbServiceName HTTP/webserver.domain.local
     Krb5KeyTab /etc/krb5.keytab
     KrbSaveCredentials Off
     KrbAuthoritative off
     KrbVerifyKDC off
   #  KerbLocalUserMapping on
     PerlAddVar ntdomain "DOMAIN.LOCAL pdc bdc"
     PerlSetVar defaultdomain DOMAIN.LOCAL
     PerlSetVar splitdomainprefix 1
     require valid-user
    </files>
   
   

   With these settings in place, we only need to adapt OTRS to use Apaches authentication. Add these lines to OTRS's Config/Kernel.pm:
   

   $Self-&gt;{'AuthModule'} = 'Kernel::System::Auth::HTTPBasicAuth';
   $Self-&gt;{'AuthModule::HTTPBasicAuth::ReplaceRegExp'} ='@DOMAIN.LOCAL';

   Despite the manually compiled mod_auth_kerb I still found that Apache passed through the users with the domain name. The easy fix in OTRS is to replace that part via regex, and all is peachy. As of now I'm happy it's working and don't want to spend more time finding out why this fix is required. ;-)
5. Client settings
   This may be obvious to some, but here it is anyway. In Internet Explorer you have to add the webserver.domain.local to the Local Intranet. Also, you have to enable Integrated Windows Authentication in the Advanced Settings. In Firefox you need to make sure to add "domain.local" to the config value network.negotiate-auth.trusted-uris. This will cause firefox to authenticate using SSPI. I use Firefox 5 and this worked like a charm. Specifying webserver.domain.local under network.automatic-ntlm-auth.trusted-uris alone will not work, as this uses NTLM, which is not Kerberos! Also, by using canonical names in Apache, it doesn't matter whether the clients access the serber by hostname alone, or by FQDN. I've had too many users who don't seem to care, so this is a good thing. Of course, when you add SSL to the equation, things look differently. YMMV.
   

/etc/samba/smb.conf

[global]
   workgroup = DOMAIN
   password server = pdc_fqdn bdc_fqdn
   realm = DOMAIN.LOCAL
   security = ads
   template shell = /sbin/nologin
   winbind use default domain = false
   winbind offline logon = false
   use kerberos keytab = true
   winbind enum users = yes
   winbind enum groups = yes
   winbind use default domain = yes

/etc/krb5.conf

 [logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = DOMAIN.LOCAL
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 forwardable = yes

[domain_realm]
 MYSERVER.DOMAIN.LOCAL = DOMAIN
DOMAIN.LOCAL = DOMAIN
.DOMAIN.LOCAL = DOMAIN
 domain.local = DOMAIN
.domain.local = DOMAIN
DOMAIN = DOMAIN
.DOMAIN = DOMAIN
domain = DOMAIN
.domain = DOMAIN

[realms]
 DOMAIN= {
  kdc = pdc_fqdn
  admin_server = pdc_fqdn
 }

 DOMAIN.LOCAL = {
  kdc = pdc_fqdn
  kdc = pdc_fqdn
 }

[appdefaults]
  pam = {
     debug = true
     ticket_lifetime = 36000
     renew_lifetime = 36000
     forwardable = true
#     krb4_convert = true
     krb5_get_tickets = false
  }
[login]
    krb4_convert = true
    krb4_get_tickets = false

Where pdc_fqdn would be somthing like DomainController.domain.local.

/etc/httpd/conf/httpd.conf

These are only the relevant parts, the rest is the default config.

KeepAlive On
ServerName webserver.domain.local
UseCanonicalName On
LogLevel debug

/etc/httpd/conf.d/auth_kerb.conf

LoadModule auth_kerb_module modules/mod_auth_kerb.so
Alias /private/ /path/to/my/folder

<location private="">
  Options Indexes
# SSLRequireSSL
  AuthType Kerberos
  AuthName "webserver Kerberos Login"
  KrbMethodNegotiate On
  KrbMethodK5Passwd On
  KrbAuthRealms DOMAIN.LOCAL DOMAIN
  KrbServiceName HTTP/webserver.domain.local
  Krb5KeyTab /etc/krb5.keytab
  KrbSaveCredentials On
  KrbAuthoritative off
  KrbVerifyKDC off
#  KerbLocalUserMapping on
  require valid-user

 </location>

XenServer iSCSI multipathing

Like many others, I use XenServer with a iSCSI SAN. To leverage a single point of failure, I enabled multipathing. When the IP or path of a host adapter changes, it can be challenging to get XenServer to recognize the new path.

The kb articles that Citrix provides about such issues are good pointers, but some commands are just plain wrong.

So this should serve as my little cheat sheet, and it might help others out there as well.

Check existing iSCSI sessions:

iscsiadm -m session

More verbosive information can be gathered with:

iscsiadm --mode node --loginall=all

If an IP has indeed changed, the old path needs to be forgotten. This can be done with:

iscsiadm –m node –T iqn.2008-01.com.openfiler:store01.lun01 –-portal=ip-address-of-the-storage -u

To reactivate a SR with multipathing one needs to get the SCSI ID from XenCenter, and then issue this:

mpathutil.py add

This step didn't work for me, however. Here's what did work in my case. Discover the targets manually:

iscsiadm -m discovery -t sendtargets --portal=ip-address-of-the-storage

Then, with the iSCSI IQN manually log on to the target:

iscsiadm -m node -T iqn.2008-01.com.openfiler:store01.lun01 --portal=ip-address-of-the-storage -l

This should yield in something like this:

Logging in to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]
Login to [iface: default, target: iqn.2008-01.com.openfiler:store01.lun01, portal: ip-address-of-the-storage,3260]: successful

The added path should immediately be shown in iscsiadm -m session. If there are still issues, restart the multipath daemon:

 service multipathd restart

It's noteworthy that this doesn't disrupt any services. So it should be safe in a production environment.

If XenCenter still reports wrong number of active/inactive paths, a refresh can be forced by running:

/opt/xensource/sm/mpathcount.py

 

XenServer 5.6 and Dell OpenManage 6.5

Citrix recently released XenServer 5.6 Service Pack 2. While you probably have found out already, that upgrading from XenServer 5.6 FP1 only works through XenCenter and not by booting from the ISOs, those of you running XenServer on Dell boxes might appreciate some pointers in getting OMSA up and running again.

Dell actually recognizes XenServer as an operating system for their servers, so there is an ISO that can be used as supplemental pack when XenServer is installed. Yay! The same ISO can be used to install it on an existing system. Instructions can be found here, at Dell's Enterprise Technology Center. This site also has directions where the latest OMSA download can be found, so check it out!

I had OpenManagae 6.4 installed on all of my XenServers prior to upgrading them to 5.6 SP2. The installer failed because some conflicting packages were already installed. So I basically had to remove OMSA 6.4 before I could install 6.5. To save you the hassle of identifying the individual packages, here's how to get rid of it quickly:

yum remove srvadmin-smweb srvadmin-smcommon srvadmin-omilcore srvadmin-sm srvadmin-iws libsmbios sblim-sfcc

 After running the install.sh script on the downloaded ISO, it is necessary to bounce the srvadmin services by issuing:

/opt/dell/srvadmin/sbin/srvadmin-services stop
/opt/dell/srvadmin/sbin/srvadmin-services start

Thanks to jh for pointing out the mistake.