CollabNet Subversion Edge 5.1.0, Trac and mod_wsgi on Windows

As I've blogged before, I think CollabNet Subversion Edge to be quite handy. Now there's a new version (5.1.0) out that addresses several issues. I found that compiling mod_wsgi isn't necessary anymore. But they still ship a version of mod_wsgi that doesn't work.....

TortoiseSVN 1.8.0 and NTLM Authentication on Windows

Subversion 1.8.0 was released recently, and my favorite svn client was updated, too. I had hoped this update to go down smooth, just like going from 1.6 to 1.7 in the past.
This time around there are some authentication issues it seems. TortoiseSVN has dropped neon and relies now solely on serf to handle HTTP(S) authentication.

The release notes mention some server-side configuration changes for optimal performance, but they fail to mention that serf doesn't play nice with NTLM authentication. It does work well with Kerberos though.

So, if you've set up Apache like I described here or here, just add this line to make Kerberos the default and have  TSVN 1.8.0 play nice with single-sign-on.

SSPIPackage Negotiate

This will change the default authentication schema (NTLM) to Kerberos. This will cause TortoiseSVN < 1.8 to prompt for username and password, because the underlying neon library cannot handle Kerberos. TortoiseSVN 1.8 and later will be fine though.
You can force older clients to use serf for specific servers though. I've previously written about it here.

There is a big discussion going on in the subversion-dev mailing list about how to best fix this behavior. It is expected that TortoiseSVN 1.8.1 will include a fix for this issue.

This issue and the fact that most SVN tools still haven't been built on top of subversion 1.8 means that you should stay on TortoiseSVN 1.7.x for now.

CollabNet Subversion Edge cont'd

I already explained Collabnet Subversion Edge and my experience installing it. Now I want to focus on installing Trac (with some hacks).
I really thought this should be easy, but it turns out that there are a lot of pitfalls. So for any other poor souls who are not fluent in python (and for my own reference) here it goes.

Installing Trac

As you know, trac has some requirements. Namely setuptools and genshii. There are a few packages to download for Windows here. I didn't know which version of python came with subversion edge, so I checked. There is a folder named csvn/python25. So it seems to be python 2.5. But wait - there is a python27.dll in there. and some other files indicating python 2.7. So which is it?

D:\csvn\Python25>python.exe --version
Python 2.7.1

 Alright, that settles it. But why did Collabnet not rename the folder..?
So I downloaded the setuptools package for python 2.7 and win32. The installer failed to run, it sais it couldn't find any python 2.7 installation.

Apparently a regular python installer will create some registry entries so that the system recognizes the python installation. Collabnet subversion edge doesn't do this anymore due some complaints.
So I had to go in there and manually create the relevant entries. They were already present from the old python 2.5 installation, which I wanted to get rid of anyway. So I changed the version to 2.7 and updated the paths to point to D:\csvn\python25. This let me install setuptools, genshii and finally trac too.

Configuring Trac

That part was straight-forward. Like with subversion, I just copied the relevant parts of my VisualSVN server's apache config and pasted it to the main httpd.conf file. Here is the relevant snippet:

<Location /trac>
 SetHandler mod_python
 PythonInterpreter main_interpreter
 PythonHandler trac.web.modpython_frontend
 PythonOption TracEnvParentDir D:\Trac
 PythonOption TracUriRoot /trac

 AuthName "Trac"
 AuthType SSPI
 SSPIAuth On
 SSPIAuthoritative On
 SSPIDomain jerich
 SSPIOfferBasic On
 SSPIOmitDomain On
 require group DOMAIN\Trac-Users
</Location>

All relevant modules are already loaded, so that's it.

Collabnet Subversion Edge

I have been quite fond of VisualSVN Server on Windows, since it was an easy way to setup subversion with a nice means to administer repositories and users.

As of lately, I grew tired because of the way they mangled apache, subversion, python and trac together. While searching for an alternative I found out that Collabnet offers exactly what I need: Apache bundled with Subversion, Python and other stuff. Very nice! This bundle is called Subversion Edge.

I decided to take the plunge and update the production VisualSVN server. After all: How hard can it be?

The installation went fine, and the collabnet subversion edge web-interface greeted me.
I had tweaked the configuration of VisualSVN server to allow single-sign-on with SSPI, so I naturally wanted to copy the relevant parts of the config. Once one tinkers with the config files, the web-interface can no longer be used to make changes to those parts. That's fine by me.

After this was taken care of I fired up Collabnet's Apache service and started to dump my subversion repositories one by one. Instead of manually loading them with svnadmin, the web interface of subversion edge provides takes care of that. Simply point it to the dump file and go ahead. That was easy!

Setting up the server, finetuning apache and migrating my three repositories took less than one hour! ;-)

Apache on Linux and Single-Sign-On with Active Directory

There are many howtos and blog posts that describe how to get Apache on Linux to work against Active Directory with single-sign-on. I found none of them to be current and complete. After tinkering with this topic for a while and getting it to work on my Centos 5.4 box, I wanted to post what I did here.
In my case, I wanted single-sign-on to work for agents in OTRS. 

1. Use samba to join the AD
   No need to create a special user account and tinker with ktpass and the likes on windows. Everything that needs to be done in AD can be done from Linux through samba, and it works fine.
   I've tried to go down the route with the user-service account in AD and it was troublesome, to say the least. There is no resource kit for Windows 7 anymore, so there is no way to get ktpass.exe. The version from the Windows Server 2003 resource kit doesn't work in Windows 7 x64. In the end I had to run ktpass in the domain controller directly, which I'm sure isn't possible in all environments.
   
2. Compile mod_auth_kerb from source, as described by Ben Chapman here. Unfortunately the link he cites is no longer available. While authentication works with the mod_auth_kerb in the centos repositories, authorization seems to be troublesome.
3. Configure Kerberos and Apache accordingly.
   My config files are attached. krb5.conf is complete, the rest are just the relevant parts. Case matters to krb5! I tried to define aliases so it would be less picky about that, but to no avail. Test kerberos functionality with klist and kinit.
   

   kinit username@DOMAIN.LOCAL

   Should prompt for password, and exit silently if all worked. If not, go through your config again. I've set Apaches loglevel to debug, otherwise there is no useful output in case of errors with mod_auth_kerb. I had many issues of
   

   gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, No principal in keytab matches desired name)

   In my Apache error_log. It took me a while to figure them out. Somehow mod_auth_kerb wasn't using the correct name for the kerberos service principal, despite me using canonical names. I had to override it, and this finally worked.
4. Configure OTRS
   I first decided to set up authorization in OTRS via Active Directory through LDAP. The OTRS Wiki provides an excellent article that outlines the necessary changes in Kernel/Config.pm. This will not only authenticate existing users through AD, but also create new OTRS users and sync them from AD. For this to work no special Apache configuration is required. If this works, move on to enable Single-Sign-On in Apache for the otrs directory.
   Because I want to enable SSO for agents only, I had to use .htaccess files to specify security. So the only change in the Apache config for OTRS from the default is this:

   
   <Directory "/opt/otrs/bin/cgi-bin/">
       AllowOverride All
       Options +ExecCGI -Includes
       Order allow,deny
       Allow from all
    </Directory>
   
   

   This enables us to comfortable change security settings in a .htaccess file placed in /opt/otrs/bin/cgi-bin. Here it is:
   

   
   <files index.pl>
     AuthType Kerberos
     AuthName "OTRS Kerberos Login"
     KrbMethodNegotiate On
     KrbMethodK5Passwd On
     KrbAuthRealms DOMAIN.LOCAL
     KrbServiceName HTTP/webserver.domain.local
     Krb5KeyTab /etc/krb5.keytab
     KrbSaveCredentials Off
     KrbAuthoritative off
     KrbVerifyKDC off
   #  KerbLocalUserMapping on
     PerlAddVar ntdomain "DOMAIN.LOCAL pdc bdc"
     PerlSetVar defaultdomain DOMAIN.LOCAL
     PerlSetVar splitdomainprefix 1
     require valid-user
    </files>
   
   

   With these settings in place, we only need to adapt OTRS to use Apaches authentication. Add these lines to OTRS's Config/Kernel.pm:
   

   $Self-&gt;{'AuthModule'} = 'Kernel::System::Auth::HTTPBasicAuth';
   $Self-&gt;{'AuthModule::HTTPBasicAuth::ReplaceRegExp'} ='@DOMAIN.LOCAL';

   Despite the manually compiled mod_auth_kerb I still found that Apache passed through the users with the domain name. The easy fix in OTRS is to replace that part via regex, and all is peachy. As of now I'm happy it's working and don't want to spend more time finding out why this fix is required. ;-)
5. Client settings
   This may be obvious to some, but here it is anyway. In Internet Explorer you have to add the webserver.domain.local to the Local Intranet. Also, you have to enable Integrated Windows Authentication in the Advanced Settings. In Firefox you need to make sure to add "domain.local" to the config value network.negotiate-auth.trusted-uris. This will cause firefox to authenticate using SSPI. I use Firefox 5 and this worked like a charm. Specifying webserver.domain.local under network.automatic-ntlm-auth.trusted-uris alone will not work, as this uses NTLM, which is not Kerberos! Also, by using canonical names in Apache, it doesn't matter whether the clients access the serber by hostname alone, or by FQDN. I've had too many users who don't seem to care, so this is a good thing. Of course, when you add SSL to the equation, things look differently. YMMV.
   

/etc/samba/smb.conf

[global]
   workgroup = DOMAIN
   password server = pdc_fqdn bdc_fqdn
   realm = DOMAIN.LOCAL
   security = ads
   template shell = /sbin/nologin
   winbind use default domain = false
   winbind offline logon = false
   use kerberos keytab = true
   winbind enum users = yes
   winbind enum groups = yes
   winbind use default domain = yes

/etc/krb5.conf

 [logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = DOMAIN.LOCAL
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 forwardable = yes

[domain_realm]
 MYSERVER.DOMAIN.LOCAL = DOMAIN
DOMAIN.LOCAL = DOMAIN
.DOMAIN.LOCAL = DOMAIN
 domain.local = DOMAIN
.domain.local = DOMAIN
DOMAIN = DOMAIN
.DOMAIN = DOMAIN
domain = DOMAIN
.domain = DOMAIN

[realms]
 DOMAIN= {
  kdc = pdc_fqdn
  admin_server = pdc_fqdn
 }

 DOMAIN.LOCAL = {
  kdc = pdc_fqdn
  kdc = pdc_fqdn
 }

[appdefaults]
  pam = {
     debug = true
     ticket_lifetime = 36000
     renew_lifetime = 36000
     forwardable = true
#     krb4_convert = true
     krb5_get_tickets = false
  }
[login]
    krb4_convert = true
    krb4_get_tickets = false

Where pdc_fqdn would be somthing like DomainController.domain.local.

/etc/httpd/conf/httpd.conf

These are only the relevant parts, the rest is the default config.

KeepAlive On
ServerName webserver.domain.local
UseCanonicalName On
LogLevel debug

/etc/httpd/conf.d/auth_kerb.conf

LoadModule auth_kerb_module modules/mod_auth_kerb.so
Alias /private/ /path/to/my/folder

<location private="">
  Options Indexes
# SSLRequireSSL
  AuthType Kerberos
  AuthName "webserver Kerberos Login"
  KrbMethodNegotiate On
  KrbMethodK5Passwd On
  KrbAuthRealms DOMAIN.LOCAL DOMAIN
  KrbServiceName HTTP/webserver.domain.local
  Krb5KeyTab /etc/krb5.keytab
  KrbSaveCredentials On
  KrbAuthoritative off
  KrbVerifyKDC off
#  KerbLocalUserMapping on
  require valid-user

 </location>

VisualSVN + Trac = Love

VisualSVN is pretty well known for providing a decent subversion server on windows-platforms thats also easy to administer. Even for those of us who want to use integrated windows authentication. However, the built-in support is lacking seamless single-sign-on from those applications that support it.

There exists a special version of VisualSVN thats packed with trac. While this install isn't quite as straight-forward as the standard package - whoever has installed python on a production server knows what I'm talking about - once setup it works remarkably well. Except for true single-sign-on.

To address this issue I figured that VisualSVN is just comprised of apache plus some svn modules. So it should be easy to hack in support to auth against an Active Directory. There are quite a few options when it comes to this. I found only one working stable: mod_auth_sspi.

To get the correct modules and dependancies I installed a stand-alone apache (the exact version that ships with VisualSVN + Trac), and went from there. I blatantly copied any missing deps into the visualsvn/bin folder.

Here's my httpd.conf:

#
# This is configuration file for Apache HTTP server which is
# part of VisualSVN Server.
#
# DO NOT EDIT THIS FILE IT WILL BE REGENERATED AUTOMATICALLY BY VISUALSVN SERVER
#
# Contact with VisualSVN Ltd. (support@visualsvn.com) if you have suggestions and
# improvements.
#

ThreadsPerChild 64
MaxMemFree 100
MaxRequestsPerChild 0

ServerRoot "C:/Program Files/VisualSVN Server"
ServerName "hostname.mydomain:443"
ServerSignature Off
ServerTokens Prod
DocumentRoot "htdocs"

PidFile "C:/Repositories//server.pid"

Listen "443"

LoadModule auth_basic_module bin/mod_auth_basic.so
LoadModule auth_digest_module bin/mod_auth_digest.so
LoadModule authn_file_module bin/mod_authn_file.so
LoadModule authn_visualsvn_module bin/mod_authnz_visualsvn.so
LoadModule authz_visualsvn_module bin/mod_authnz_visualsvn.so
LoadModule authz_svn_module bin/mod_authz_svn.so

LoadModule dir_module bin/mod_dir.so
LoadModule deflate_module bin/mod_deflate.so
LoadModule mime_module bin/mod_mime.so
LoadModule setenvif_module bin/mod_setenvif.so

LoadModule dav_module bin/mod_dav.so
LoadModule dav_svn_module bin/mod_dav_svn.so
LoadModule ssl_module bin/mod_ssl.so

LoadModule sspi_auth_module bin/mod_auth_sspi.so


Options FollowSymLinks
AllowOverride None



DAV svn

SVNListParentPath on
SVNParentPath "C:/Repositories/"
SVNIndexXSLT "/svnindex.xsl"
SVNPathAuthz on

AuthName "Subversion Repositories"
AuthType Basic

AuthType SSPI
SSPIAuth On
SSPIAuthoritative On
SSPIDomain jerich
SSPIOfferBasic On
SSPIOmitDomain On

require group MYDOMAIN\Subversion



DirectoryIndex index.html


ErrorLog "C:/Repositories//server.log"

LogLevel error

DefaultType text/plain


TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl



SSLEngine on

SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLPassPhraseDialog builtin
SSLSessionCache "shm:C:/Repositories//ssl_scache"
SSLSessionCacheTimeout 300
SSLMutex default
SSLCertificateFile conf/hostname.cer
SSLCertificateKeyFile conf/hostname.key.insecure



Include conf/httpd-custom.conf

And the httpd-custom.conf which includes the trac configuration:

LoadModule python_module "trac/python/mod_python_so.pyd"
LoadModule authz_user_module bin/mod_authz_user.so

SetHandler mod_python
PythonInterpreter main_interpreter
PythonHandler trac.web.modpython_frontend
PythonOption TracEnvParentDir C:\Trac
PythonOption TracUriRoot /trac

AuthName "Trac"
AuthType Basic

AuthType SSPI
SSPIAuth On
SSPIAuthoritative On
SSPIDomain mydomain
SSPIOfferBasic On
SSPIOmitDomain On

require group MYDOMAIN\Subversion

Everyone in the AD-group "Subversion" is granted access to SVN and trac also. This could be changed of course to allow different groups access. The access levels in trac can be configured from within the admin interface.
I decided to run a https server. To avoid nasty Internet Explorer warnings about certificates and whatnot, you can either disable the checks for certificates, or create a proper certificate if you are lucky and you've got a CA server in your domain. Here's how to create a proper cert using openSSL and a windows CA. Copy them onto your svn server and that's it.