Yesterday support for pluggable e-mail backends has landed in Django’s trunk and today I’m happy to announce two e-mail backend implementations to be used with Django-based App Engine applications.

The e-mail backends allow you to use native Django functions for sending e-mails like django.core.mail.send_mail() on App Engine.

To use pluggable e-mail backends you’ll have to use a recent version of Django. Support for pluggable e-mail backends was introduced in rev11709 and will be released in Django 1.2.

Using the e-mail backend

The e-mail backends are available as a single package on bitbucket. To check out the most recent sources run:

$ hg clone http://bitbucket.org/andialbrecht/appengine_emailbackends/

or got to the downloads page and grab a zip file.

To use the e-mail backend for App Engine copy the appengine_emailbackend directory to the top-level directory of you App Engine application and add the following line to your settings.py:

EMAIL_BACKEND = 'appengine_emailbackend'

If you prefer asynchronous e-mail delivery use this line in your settings.py instead:

EMAIL_BACKEND = 'appengine_emailbackend.async'

When using the async backend e-mails will not be sent immediately but delivered via a taskqueue.

Running the demo application

There’s also a demo application included in this repository. Run this application using

$ dev_appserver.py .

and open http://localhost:8080 in your web browser. Remember to include a current checkout of the Django SVN repository in the top-level directory of the demo application.

If you’re a regular Dominion player like me, you’re always in search for new card sets. Fortunately I’ve had a little time this weekend to write a simple App Engine application that gives me a random card set on demand.

You can see it in action at http://dominionsets.appspot.com

That’s it, just a short post on this, now back to the next match :)

P.s.: Some technical stuff… There’s Django with it’s i18n support under the hood. The UI is powered by JQuery.

A new bugfix release for CrunchyFrog is out in the wild. There are not much visible changes, but a few bug fixes under the hood. One visible change is the option to give the SQL editor much more space by hiding all other UI elements in the main window. Here’s a screenshot:

SQL editor eating up all space

The second screenshot shows the new highlighting of errors in the editor. In this case there’s no table called “actor”. When the execution of a statement results in a SQL error, CrunchyFrog now tries to find the position of the error in the SQL editor, highlights the corresponding part in your SQL statement and moves the cursor to the right place. I found it very handy to have the cursor in place to correct the error and run the statement again without searching the error before.

ups... an error occurred

If you’re interested in the other, non-visible changes, please read the full list of changes or just go straight to the project page and give it a try :)

Thanks to everyone who contributed to this release by sending patches and submitting issues on the tracker!

The quote in the title is something that Dana Colley said after giving credits to the members of Orchestra Morphine on some show circulating through the internet about nine years ago. I think this show is still available on the HI-N-DRY website and I strongly recommend to buy it, it’s a fascinating recording.

I’ve choosen this quote because it expresses what I’m thinking today, the 10th anniversary of Mark Sandman’s passing. His music shaped me more than any other music I’m listening too. He had that idea of a unique sound, not to mention the great lyrics. It’s not easy to explain to someone who never listened to Morphine what this idea is about. It’s not even easy to explain it to people that had to hear quite a lot of Morphine the past ten (and more) years (many, many thanks to my family for their patience with my musical taste they call “a bit strange” from time to time :)…

There’s always something good about unique and innovative ideas: They make their way through time. Without a doubt, Mark died way too early. But his musical concept is still present today. Just have a look at Twinemen, A.K.A.C.O.D., Orchestra Morphine and many other artists and bands doing covers and claiming Marks musical ideas as a major influence for their own ideas.

I discoverd Morphine through a good friend of mine. He visited me after a travel to Canada somewhere in the late 90’s and had those records in his hand, just saying “Listen to this.” That fundamentally changed the way I was listening to music up to today. It was this mixture of blues, jazz, rock and poetry carried through a suprising soundscape of a 2-string slide bass, baritone saxophone and a straight and driving drum. Unfortunately I’ve never had the chance to see them live. We were about to see them in Brugge in July ‘99, but that show never happened.

I’ve met a few people since that who found the Morphine sound and lyrics “so charmingly depressive”. But, hey, I’m not sure if they really listened… :) One thing I really like about the lyrics in Sandman songs is that there’s always something that spontaneously comes to your mind in certain situations. Here, it is to just say “Yes” (in terms of “get in your go-cart and go, little sister”).

A funny side-note regarding the lyrics: A few years ago I wrote my thesis about fortune in medieval Arthurian literature and after a busy day reading manuscripts, making notes und trying to understand the different concepts of fortune that can be found, I’ve put the B-Sides into my player and found myself laughing out loud, Mark summarizes them all in “Lucky Day”:

Now I’m down a little in fact I’m down a lot
I’m on a roller coaster ride that I can’t stop
My luck has changed but she’ll come back
That’s the beauty of a game of chance
I can’t loose forever but I’m doomed to try
Keep on hearing a voice inside
Players win and winners play, have a lucky day

There’s both the concept of being at almighty Fortune’s mercy and the concept of getting rid of her by using your own virtue condensed in a few lines while I was reading verse over verse to find a subtle phrase for it.

BTW, the version of “Lucky Day” found on the B-Sides is one of my favorites. The instrumental part preceeding these lines is incredibly powerfull and one of the most astonishing bits of music I’ve ever heard.

So, let me close with saying thanks too. Thanks to the folks at HI-N-DRY and the Mark Sandman Music Project for spreading a musical idea over years now, keeping up the memory of Mark Sandman and of course for the honest and good music they make. And thanks to Mark for giving his potion of uniqueness and sharing his thoughts and insights in an fascinating musical way. I’m looking forward to the first downloads on HI-N-DRY today featuring covers of Sandman songs, poetry and unreleased songs from the depths of the HI-N-DRY archive!

With mistakes yea mistakes and sudden inspirations
Edges corners explosions convections
All fast through a slow motion landscape

P.S.: The lyrics are copied from The Other Side, kudos to Ian Hadfield for keeping this site up!

Yesterday I’ve taken over maintenance of hgsvn from it’s original author Antoine Pitrou. Many thanks to Antoine at this point for doing a great job on this tool!

In case you don’t know, hgsvn is “a set of scripts to work locally on Subversion checkouts using Mercurial.”

My aim is not to make these scripts a full-blown Subversion extensions for Mercurial. hgsubversion or the convert extension will probably do a much better job with tighter integration as it’s in the scope of hgsvn. So the main focus is on fixing bugs and to finish and release the hgpushsvn script that commits changes in your Mercurial repository back to Subversion.

The reason why I decided to work on this project is pretty simple: I’ve made good experiences with it and I still want to keep using it.

With hgpushsvn the tool provides three scripts for the most basic tasks:

hgimportsvn initializes a hg repository and fetches the sources from svn, optionally starting at a specific revision.

hgpullsvn pulls new change sets from svn to your local hg repository.

hgpushsvn pushes back your local commits to the svn repository.

Using hgsvn for a while now, I’m feeling very comfortable with these scripts and they fit very well into my workflow.

The next things I’m about to do are:

• fix the unittests, at least one is broken ATM
• setup a buildbot to have some automated tests
• finish hgpullsvn

If you’re using hgsvn, please file bug reports or submit patches on the issue tracker.

There was a Simpsons episode, I can’t recall correctly, but I think Bart recorded Lisa when her heart breaked and he watched it in slow motion to stop exactly at that point.

I thought of this episode yesterday while playing around with my laptop’s webcam and a Python shell. Finally I wrote a little fun script that does almost the same: Just register it as a hg hook and it takes a picture of you exactly at the unique moment when merging fails and it sends it directly and without any further questions to Twitpic and Twitter:

#!/usr/bin/env python

import os
import sys
import tempfile
import time

from CVtypes import cv
from twitpic import TwitPicAPI

DEVICE = 0
TWITTER_USER = 'xxx'  # CHANGE THIS!
TWITTER_PWD = 'xxx'   # CHANGE THIS!
# This is the time in seconds you need to realize that the merge has
# failed. When setting this consider that it already takes about a second
# for the camera to take the picture. "0" means no delay ;-)
EMOTIONAL_SLUGGISHNESS_RATE = 0.0

def grab_image(fname):
  camera = cv.CreateCameraCapture(DEVICE)
  frame = cv.QueryFrame(camera)
  cv.SaveImage(fname, frame)

def how_do_you_look():
  failed = bool(os.environ.get('HG_ERROR', 0))
  if not failed:
    return  # hmpf, maybe next time...
  fd, fname = tempfile.mkstemp('.jpg')
  if EMOTIONAL_SLUGGISHNESS_RATE > 0:
    time.sleep(EMOTIONAL_SLUGGISHNESS_RATE)
  grab_image(fname)
  twit = TwitPicAPI(TWITTER_USER, TWITTER_PWD)
  retcode = twit.upload(fname, post_to_twitter=True,
                        message='Another merge failed.')
  os.remove(fname)

if __name__ == '__main__':
  how_do_you_look()

You’ll need the CVtypes OpenCV wrapper and this Twitpic Python module. I’ve patched the twitpic module to support messages. Have a look at this issue if it’s already supported, otherwise a diff that adds the message keyword is attached to the issue. To use it as a Mercurial hook just add to .hg/hgrc:

[hooks]
update =  /path/to/the/above/script.py

and make the script executable.

The results are pretty good :)

Have fun!

BTW, the way how to access the camera is inspired by this nice blog post about face recognition using OpenCV.

Edit (2009-05-12): It was Ralph, not Lisa. Thanks Florian!

Edit 2009-09-04: Starting with App Engine SDK 1.2.5 released today there’s now a much smarter approach. See the docstring of the ereporter module for details!

Django’s error reporting via e-mail is something I’ve missed for my App Engine applications. If you’ve followed the instructions on how to use Django on App Engine errors are already logged. But honestly, I don’t check the logs very often and sometimes it’s just a silly typo that’s fixed in a minute ;-)

So if you’ve followed the instructions you can simply extend the log_exception function to send an email after the exception is logged:

def log_exception(*args, **kwds):
  """Django signal handler to log an exception."""
  excinfo = sys.exc_info()
  cls, err = excinfo[:2]
  subject = 'Exception in request: %s: %s' % (cls.__name__, err)
  logging.exception(subject)

  # Send an email to the admins
  if 'request' in kwds:
    try:
      repr_request = repr(kwds['request'])
    except:
      repr_request = 'Request repr() not available.'
  else:
    repr_request = 'Request not available.'
  msg = ('Application: %s\nVersion: %s\n\n%s\n\n%s'
         % (os.getenv('APPLICATION_ID'), os.getenv('CURRENT_VERSION_ID'),
            ''.join(traceback.format_exception(*excinfo)),
            repr_request))
  subject = '[%s] %s' % (os.getenv('APPLICATION_ID'), subject)
  mail.send_mail_to_admins('APP_ADMIN@EXAMPLE.COM',
                           subject,
                           msg)

Make sure to replace “APP_ADMIN@EXAMPLE.COM” with the email address of an admin for this application and to add these two imports somewhere at the top of your main.py:

import traceback
from google.appengine.api import mail

You don’t need to comment this when developing your app to avoid lots of useless mails. The dev server doesn’t fully implement send_mail_to_admins() most likely as there’s no real concept of an designated admin user in the SDK. It just writes a short message to the log, even if the sendmail option is enabled.

P.s.: If you use webapp instead of Django have a look at the seventh slide of this presentation by Ken Ashcraft. He shows a similar implementation for webapp based applications.

After I’ve described some basics in part I let’s have a closer look to the actual Python module called sqlparse. First off, have a look at the project page on how to download and install this module in case you’re interested. But now, let’s have some fun…

The API of the module is pretty simple. It provides three top-level functions on module level: sqlparse.split(sql) splits sql into separate statements, sqlparse.parse(sql) parses sql and returns a tree-like structure and sqlparse.format(sql, **kwds) returns a beautified version of sql according to kwds.

As mentioned in my previous post, what effort needs to be done to build the return values depends on what lexing and parsing work is needed to find the result. For example sqlparse.split() does the following:

• Generate a token-type/value stream basically with a copy of Pygments lexer
• Apply a filter (in terms of a Pygments stream filter) to find statements
• Serialize the token stream back to unicode

sqlparse.parse() does all the grouping work and sqlparse.format() runs through those groups, modifies them according to the given formatting rules and finally converts it back to unicode.

Here’s an example session in a Python shell:

>>> import sqlparse
>>> # Splitting statements:
>>> sql = 'select * from foo; select * from bar;'
>>> sqlparse.split(sql)
<> # Formatting statemtents:
>>> sql = 'select * from foo where id in (select id from bar);'
>>> print sqlparse.format(sql, reindent=True, keyword_case='upper')
SELECT *
FROM foo
WHERE id IN
  (SELECT id
   FROM bar);

>>> # Parsing
>>> sql = 'select * from "someschema"."mytable" where id = 1'
>>> res = sqlparse.parse(sql)
>>> res
<<>> stmt = res[0]
>>> stmt.to_unicode()  # converting it back to unicode
<<<u>>> # This is how the internal representation looks like:
>>> stmt.tokens
<<>>

Now, how does the grouping work? Grouping is done with a set of simple functions. Each function searches for a simple pattern and if it finds one a new group is built. Let’s have a look at the function that finds the WHERE clauses.

def group_where(tlist):
    [group_where(sgroup) for sgroup in tlist.get_sublists()
     if not isinstance(sgroup, Where)]
    idx = 0
    token = tlist.token_next_match(idx, T.Keyword, 'WHERE')
    stopwords = ('ORDER', 'GROUP', 'LIMIT', 'UNION')
    while token:
        tidx = tlist.token_index(token)
        end = tlist.token_next_match(tidx+1, T.Keyword, stopwords)
        if end is None:  # WHERE is at the end of the statement
            end = tlist.tokens[-1]
        else:
            end = tlist.tokens[tlist.token_index(end)-1]
        group = tlist.group_tokens(Where, tlist.tokens_between(token, end))
        idx = tlist.token_index(group)
        token = tlist.token_next_match(idx, T.Keyword, 'WHERE')

tlist is a list of tokens, possible subgroups are handled first (bottom-up approach). Then it grabs the first occuring “WHERE” and looks for the next matching stop word. I’m pretty unsure if the stop words approach is right here, but at least it works for now… If it finds a stop word, a group using the class Where is created, the tokens between WHERE and the stop word are attached to it and – still within the while loop – the next WHERE keyword is used as the next starting point.

So why not use a grammar here? At first, this piece of code is pretty simple and easy to maintain. But it can also handle grammatically incorrect statements more lazily, e.g. it’s no problem to add an if clause that – when for example an unexpected token occurs – the function just jumps to the next occurance of WHERE without changing anything or even raising an exception. To achieve this the token classes provide helper functions to inspect the surroundings of an occurrence (in fact, just simple list operations). There’s no limitation what a grouping function can do with the given token list, so you could even “guess” a proper group with some nasty algorithm.

The current problem with this approach is performance. Here are some numbers:

Most of the performance is lost when giving up the stream-oriented approach in the parsing phase. The numbers are based on the first unrevised working version. I expect performance improvents especially in the way how token lists are handled behind the scenes with upcoming versions. For real life statements the parser behaves quite well. BTW, the Pygments lexer takes about 6 seconds (compared to 5.5 secs. for splitting) for the 1.9MB of SQL.

The non-validating approach is a disadvantage too. You’ll never know if a statement is valid. You can even parse middle high german phrases and receive a result:

>>> sqlparse.parse('swer an rehte güete wendet sîn gemüete')[0].tokens
<<>>

It’s up to the user of this module to provide suitable input and to interpret the output. Furthermore the parser only supports not every nifty edge of an SQL dialect. Currently it’s mostly ANSI-SQL with some PostgreSQL specific stuff. But it should be easy to implement further grouping functions to provide more SQL varieties.

The splitting feature is currently used by CrunchyFrog and does a pretty good job there. I assume that SQL splitting works stable and reliable in most cases. Beautifiying and parsing is very new in the module and full functionality needs to be proven with time. Luckily the top-level API with it’s three functions is damn simple and keeps the doors open for significant changes behind the scenes if they’re needed.

The sources of the sqlparse module are currently hosted on github.com but may move to Google Code anytime soon. Refer to the project page on Google Code for downloads and how to access the sources.

In addition there’s a simple AppEngine application that exposes the formatting features as an online service.

Some time ago I was in search for some kind of SQL parser module for Python. As you can guess, my search wasn’t really successfull. pyparsing needs a grammar, but I’m not really interested in writing a full-blown grammar for SQL with it’s various dialects. AFAICT Gadfly implements a customized SQL parser, but I was not able to figure out how to use it. And there’s sqlliterals, but as the name suggests it’s just for identifying literals within statements.

I expect such a module to do the following:

• It should be fast ;-)
• scalable – what the parser needs to know about a string containing SQL statements depends on what I want to do with it. If I just want to split that string in separate statements I don’t need to know as much as when I want to know what identifiers occur in it.
• non-validating -  Parsing shouldn’t fail if the statement is syntactically incorrect as I want to use it for an SQL editor where the statements are (naturally) incorrect most of the time.
• some beautifying is a nice-to-have – We all know SQL generated by a script and copied from a logging output to a database front-end – it always looks ugly and is pretty unreadable.

The only thing I’ve found that comes close to my needs is Pygments (yes, the syntax highlighter). It does a really good job highlighting SQL, so it “understands” at least something. To be concrete, it has a rock-solid and fast lexer. There’s a FAQ entry if it’s possible to use Pygments for progamming language processing. The answer is:

The Pygments lexing machinery is quite powerful can be used to build lexers for basically all languages. However, parsing them is not possible, though some lexers go some steps in this direction in order to e.g. highlight function names differently.

Also, error reporting is not the scope of Pygments. It focuses on correctly highlighting syntactically valid documents, not finding and compensating errors.

There’s a nice distinction between lexers and parser in this answer. The former does lexical and the latter syntactical analysis of a given stream of characters. For my needs this led to a two step process: first to do the lexing using Pygments mechanism (BTW, read “focuses on [...] syntactically valid documents, not finding [...] errors” as “non-validating”) and then to add parsing on top of the token stream.

So I stripped the lexing mechanism for SQL out of Pygments to get rid of some overhead not needed for my purposes (e.g. loading of extensions). The only changes to the Pygments tokenizer was to replace a huge regex for finding keywords by a dictionary-based lookup and to add a few new tokens to make live easier in the second processing stage. The achieved performance improvement by replacing the regex doesn’t play a significant role for Pygments, it just speeds up the lexing a bit.

In the second step the very efficient token-type/value stream generated in step 1 is grouped together in nested lists of simple classes. Instantiating a bunch of classes results in a performance loss, but with the benefit of some helpful methods for analyzing the now classified tokens. As the classes are pretty simple most of the performance loss was recaptured by using slots.

The grouping mechanism is again non-validating. It tries to find patterns like identifiers and their aliases or WHERE clauses and their conditions in the token stream, but it just leaves the tokens unchanged if it doesn’t find what it’s looking for. So it actually defines some sort of “grammar”, but very unrestrictive. That means that the quality of answers to questions like “Is there a WHERE clause?” or “Which tables are affected?” heavily depends on the syntactical clearness of the given statement at the time when it’s processed and it’s up to the user or application using the module to intrepet the answers right. As the module is seen as a helper during the develoment process of SQL statements it doesn’t insist on syntactical correctness as the targeted database system would do when the statement is executed.

All in all these are the two (simple) processing steps:

1. Lexing a given string using regular expressions and generate a token-type/value stream (in fact, this is the Pygments lexer).
2. Parsing
   1. Instantiate a simple token class for each item in the stream to ease syntactical analysis.
   2. Group those tokens in nested lists by using simple functions by finding patterns.

Depending on what should be done with the statements, only the first step is required. For example, splitting statements or simple formatting rules don’t require parsing.

Pt. II of this blog post will cover the two remaining points: scalibality and the beautifiying goodie. And it will give a closer look at the  module I came up with, including some performance stats. For the impatient: the source code of this module is available here.

This blog post describes how to generate a simple man page during build time for Python applications using distutils and optparse. Technically this post describes how to write a custom HelpFormatter and a custom build command.

For GUI applications command line options usually bother me not much. Just in some rare cases, mainly when debugging or feeling unlucky with the startup behavior, I’m interested in these options. Even then I prefer to use “–help”. BTW, for the “real” command line tools like sed, ls or grep this is totally different.

So, due to my lack of interest in man pages for GUI applications I’ve shipped the previous release of CrunchyFrog along with a totally out-dated man page. Moreover I really don’t like writing man pages or keeping them in sync with the actual command line options already documented by OptionParser’s help features…

Did I say “already documented”? I’m not a friend of doing things twice, reusing that “already documented” should be very pleasant to go to get rid of my problem.

What To Put On A Man Page?

The short answer is: It should mention the application’s name, it’s purpose, how to start it, who wrote it, where to find additional information and what command line options it provides. Or, in terms of man page sections: “NAME”, “SYNOPSIS”, “DESCRIPTION”, “OPTIONS”, “SEE ALSO” and “AUTHOR” (as you may already have noticed, I’m focussing just on man page (1) in this post). Of course, there are more possible sections, but after inspecting some randomly choosen man pages for GUI applications on my system, it seems that those sections are the most common ones.

(If you are interested in the full answer, read the Linux Man Page Howto.)

Now back to our Python application that uses optparse to provide command line options.

optparse comes with a pretty good out-of-the-box help feature. As stated in the docs the primary purpose is to assist in creating user-friendly command-line interfaces and not to generate help documents (like man pages). But the ability to write a custom HelpFormatter makes it easy to generate such documents.

Writing a Custom HelpFormatter

Let’s have a look at the custom formatter:

class ManPageFormatter(optparse.HelpFormatter):

    def __init__(self, indent_increment=2, max_help_position=24,
	         width=None, short_first=1):
        """Constructor. Unfortunately HelpFormatter is no new-style class."""
        optparse.HelpFormatter.__init__(self, indent_increment,
                                        max_help_position, width, short_first)

    def _markup(self, txt):
        """Prepares txt to be used in man pages."""
        return txt.replace('-', '\\-')

    def format_usage(self, usage):
        """Formate the usage/synopsis line."""
        return self._markup(usage)

    def format_heading(self, heading):
        """"Format a heading.
        If level is 0 return an empty string. This usually is the string "Options".
        """"
        if self.level == 0:
            return ''
        return '.TP\n%s\n' % self._markup(heading.upper())

    def format_option(self, option):
        """Format a single option.
        The base class takes care to replace custom optparse values.
        """"
        result = []
        opts = self.option_strings[option]
        result.append('.TP\n.B %s\n' % self._markup(opts))
        if option.help:
            help_text = '%s\n' % self._markup(self.expand_default(option))
            result.append(help_text)
        return ''.join(result)

The three methods starting with format_* do all the formatting for us. The base class has additional formatting methods, but there’s no need to overwrite them as they do nothing special except wrapping lines to the desired width.

To use a custom formatter either give it as an parameter to the OptionParser constructor or use

parser.formatter = ManPageFormatter()
parser.formatter.set_parser(parser)

if your option parser is already instantiated.

Now, when you call parser.format_option_help() you’ll see a man page fragment with all options (including nicely formatted option groups). Maybe you wonder why not use parser.print_help() here, which normally adds a nice usage line and the description if given to the output. The reason for that is quite simple: Many of the documentation features of optparse are optional. If there’s no usage defined, no usage line will be formatted. So using it for the SYNOPSIS section and eventually the header of the man page isn’t a good idea. Everything surrounding the OPTIONS section of the man page should be generated by something else, that takes care to include a proper header and footer. We are using the custom formatter just for the options themself.

Adding a Custom Build Command

To write the rest of the man page we’re using a custom build command for distutils. This command is executed when you run python setup.py build or install that depends on build. There’s a nice side-effect when generating the man page on build time: You can simply skip this step when the setup.py commands are run on an OS that doesn’t know what a man pages is ,-)

The skeleton of a distutils command looks like this:

# -*- coding: utf-8 -*-

"""build_manpage command -- Generate man page from optparse/dist metadata."""

from distutils.command.build import build
from distutils.core import Command

class build_manpage(Command):

    description = 'Generate man page.'

    user_options = []

    def initialize_options(self):
	pass

    def finalize_options(self):
	pass

    def run(self):
	# Do something useful!

build.sub_commands.append(('build_manpage', None))

As you can see, “build_manpage” is added as a sub-command to the build command and the build_manpage class has some methods for option handling and a run() method.

To use this command add the following to your setup.py:

from distutils.core import setup
from build_manpage import build_manpage
setup(
    [...]
    cmdclass={'build_manpage': build_manpage}
)

Adding Command Options

Generating the man page needs some options. The build_manpage commands needs to know where to write the generated man page and which option parser it should use. To achieve this we have to modify the user_options attribute and the two corresponding methods in our skeleton class:

    user_options = [
        ('output=', 'O', 'output file'),
        ('parser=', None, 'module path to optparser (e.g. mymod:func'),
        ]                                                                       

    def initialize_options(self):
        self.output = None
        self.parser = None                                                      

    def finalize_options(self):
        if self.output is None:
            raise DistutilsOptionError('\'output\' option is required')
        if self.parser is None:
            raise DistutilsOptionError('\'parser\' option is required')
        mod_name, func_name = self.parser.split(':')
        fromlist = mod_name.split('.')
        try:
            mod = __import__(mod_name, fromlist=fromlist)
            self._parser = getattr(mod, func_name)()
        except ImportError, err:
            raise
        self._parser.formatter = ManPageFormatter()
        self._parser.formatter.set_parser(self._parser)
        self.announce('Writing man page %s' % self.output)
        self._today = datetime.date.today()

user_options is a list of 3-tuples (long option, short option, help),  initialize_options() sets default values and finalize_options() makes sure that the given options are sane (nb, the Command class has some validation functions, unfortunately you’ll have to read the source to find them…). Run setup.py build_manpage --help to see how to use your options on the command line. Besides setting these options on the command line you can set them in a file called setup.cfg too:

[build_manpage]
output=data/mymanpage.1
parser=myapp.somemod:get_parser

Both options are required. output tells the command to which file it should write the generated man page, parser is a string pointing to a function that returns an OptionParser instance (a dotted module path with a function name separated by a single colon).

Putting It All Together

This is done by the run() method, of course. In finalize_options() the option parser was imported and our ManPageFormatter was attached to it. The run() method now does the following:

    def run(self):                                                             
        manpage = []                                                           
        manpage.append(self._write_header())                                   
        manpage.append(self._write_options())                                  
        manpage.append(self._write_footer())                                   
        stream = open(self.output, 'w')                                        
        stream.write(''.join(manpage))                                         
        stream.close()

Let’s have a look at the _write_header() helper function (_write_options() is basically a call to the formatter and _write_footer() just adds the author and a link to the homepage):

    def _write_header(self):                                                   
        appname = self.distribution.get_name()                                 
        ret = []                                                               
        ret.append('.TH %s 1 %s\n' % (self._markup(appname),                   
                                      self._today.strftime('%Y\\-%m\\-%d')))   
        description = self.distribution.get_description()                      
        if description:                                                        
            name = self._markup('%s - %s' % (self._markup(appname),            
                                             description.splitlines()[0]))     
        else:                                                                  
            name = self._markup(appname)                                       
        ret.append('.SH NAME\n%s\n' % name)                                    
        synopsis = self._parser.get_usage()                                    
        if synopsis:                                                           
            synopsis = synopsis.replace('%s ' % appname, '')                   
            ret.append('.SH SYNOPSIS\n.B %s\n%s\n' % (self._markup(appname),   
                                                      synopsis))               
        long_desc = self.distribution.get_long_description()                   
        if long_desc:                                                          
            ret.append('.SH DESCRIPTION\n%s\n' % self._markup(long_desc))      
        return ''.join(ret)

As you can see, this method makes use of another advantage of using a distutils command. It fetches some information not covered by optparse from distutils’ meta data and in addition takes care of optparse’s optional usage and description attributes.

The complete implementation of the build_manpage command is available here.

This is how the resulting man page for CrunchyFrog looks like:

Generated Man Page for CrunchyFrog

I’ll have to admit, it’s a very simple approach to have a man page without writing it and – even worse – maintaining it. But at least it solved my problem to distribute a GUI application with an up-to-date man page. This approach has no extra requirements as everything needed can be found in Python’s stdlib and the man page is generated when needed. Luckily the build command is extensible, so it’s no problem to add additional sections (what about connection the script to a bug tracker and add a hopefully not too long BUGS section?).

P.S.: In case your interested in another formatter example, I’ve recently written another HelpFormatter that creates a Google Code Wiki page for Rietveld’s upload.py. The source are available here.

Edit (2009-03-25): There’s a nice patch written by René Neumann that adds a SEE ALSO option to the distutils command.