rob's Weblog

| Private |
| Technology |
| World |
 $ pipeline blogposts:all task:blogpost2html concat:-

Krita on Kickstarter

My digital drawing & painting program of choice these days is the open source software, Krita. It’s got great brush engines, works fantastically with my Wacom tablet under Linux (my primary criteria), and they also recently launched a well supported Windows version.

They recently started a Kickstarter campaign to fund some full-time developers who’ve already made significant contributions to the project.

If you haven’t seen it before, check out:

If you’re interested in supporting its development:

— by Robert Thomson, created 15th Jun, 2014, last modified 16th Jun, 2014 | Tags: Tech, World

Emulate EC2's for your own VMs

Assuming you have your bridge configured as br0, and are are running a webserver on port 9880 (either unbound or bound the primary IP of the bridge ‘br0’), the following should allow a KVM VM, LXC container, etc. to connect to just like they can in Amazon EC2. This has to be run on the server hosting the VMs, of course.

# sysctl net.bridge.bridge-nf-call-iptables=1 
# sysctl net.bridge.bridge-nf-call-arptables=1
# ip addr add dev br0 
# iptables -t nat -A PREROUTING -d -p tcp --dport 80 -j REDIRECT --to-port 9880
or using DNAT:
# ip addr add dev lo
# iptables -t nat -A PREROUTING -j DNAT -d -p tcp --dport 80 --to

Your webserver should probably ensure that the request is coming from the IP of one of the VMs currently running on the server itself.

— by Robert Thomson, created 14th Feb, 2014, last modified 14th Feb, 2014 | Tags: Tech

I love software that "just works", and is one of them. It builds disk images in various image formats and for various cloud providers.

With a small JSON configuration file, a kickstart file, and a set of provisioning scripts, I can have a QEMU image automatically built from the install CD and customised as I wish.

With a different config file, I can have the same thing for EC2 images.

And because it's JSON, I can dynamically generate custom configuration files quickly and easily.

There seems to be a positive trend of self-contained, single-purpose and well designed software coming from the Golang camp. Keep at it! :-)

— by Robert Thomson, created 9th Jan, 2014, last modified 9th Jan, 2014 | Tags: Tech


Following up from my post about Ansible, I decided to look into SaltStack.

SaltStack looks like one of the more promising tools to appear recently on the system management landscape.

It's primary features are:

This is an impressive set of features for a single solution, but there are some areas of concern:

Some specific technical issues appear to be:

I hold out hope that these issues will be addressed. In the meantime, I won't be recommending it as a general purpose solution for a large organisation.

— by Robert Thomson, created 15th Oct, 2013, last modified 15th Oct, 2013 | Tags: Tech

Service Orchestration

I've been musing a bit about complicated service orchestration of late. For some situations, it's as simple as SSH'ing into one or more boxes in a predefined order and executing some actions; sometimes it involves copying files around, calling random APIs, adding/removing nodes from load balancing proxies, launching new VMs in a cloud and waiting for them to be ready or bringing them down again.

This has of course led me down the road of workflows and state machines, and has me wondering about the kind of interface that I would like to have for designing orchestrations as well as for for executing them. I think it'd be nice to kick a flow off from the command line and then query it's current progress or 'tail' it. You could also execute flows via a web-based job scheduler and have visual feedback.

As for design, I'm less interested in formal BPM workflows since they seem a bit of an overkill, although they can also be applied to service orchestration. A state machine model might be a better fit. I could imagine something similar to Puppet's model, having orchestration steps instead of resources, and it running until the dependencies for the finished state have been satisfied. There are a lot of possibilities, and I'm still looking for a model that would work for me.

As far as I've thought about it, a given flow would comprise of a set of discrete steps executed under certain conditions. There would have to be some state associated with the flow itself, but the conditions wouldn't have to limit themselves to that state (ie. a conditional could call a custom function which looks up external state). The discrete steps could be code, builtin functions and decision logic, or other flows. I'd like to be able to express these steps succinctly using a simple data structure, but turn to a real programming language to do the more complicated work.

— by Robert Thomson, created 5th Apr, 2013, last modified 5th Apr, 2013 | Tags: Tech

Python pipelines

I recently realised that one of my most common tasks is to deal with streams of structured data, perform some task on or with the data, and spit out the results. The traditional UNIX way is to do line-based processing, piping through awk, sed, grep, perl one-liners, and storing the results in temporary files which are then used as input to other commands. However, having pre-structured data makes more sense for many scenarios.

So I’m currently writing a set of python libraries for working with streams of structured data. It uses a push-based pipeline, where a stream processor can either modify, filter, or add new items to the stream before passing it onto the next processor in the pipeline.

Perhaps some command line examples would show the idea best:

$ pipeline in:hosts filter:location=~ec2 parallel:16 task:ping filter:+success ssh:“uptime -a” out:host_uptimes
$ pipeline blogposts:all filter:tags=~tech task:blogpost2html concat:-
$ pipeline in:shares task:get_latest_share_price task:analyse_share_value csv:share,value,profit

All of this can also be done programmatically with relative ease.

With the command line tool, if no output processor is provided, the default will be used which saves the structured data to “last”, using a default “database”.

— by Robert Thomson, created 3rd Apr, 2013, last modified 3rd Apr, 2013 | Tags: Tech

German Copyright Law & Employment Contracts

Like many IT geeks, my work is also my hobby. This often means that my personal fields of interest are often in the same problem domain as that of my workplace. Recently I decided to thoroughly investigate German copyright law and my work contract after a colleague suggested that I had crossed a line with an open source project of mine. I’m happy to say that after my efforts, I’m quite confident that I’m OK. I am going to share what I learnt, however.

Firstly – a clarification for those who don’t know or get confused easily. Copyright law applies to specific implementations, not to ideas. Also, I am not a lawyer, so ask a lawyer for a legal opinion. :-)

In Germany, copyright always belongs to the individual or individuals who wrote a piece of software. That means that any software I would write for work would have my name associated with the copyright. However, there are different rights associated with copyright. German (and most European) copyright law aims to ensure that the author retains the inalienable moral rights. The second set of rights are the economic rights, which govern how and if the software will be made available to others.

German copyright law states, in UrhG. 69b: “Where a computer program is created by an employee in the execution of his duties or following the instructions given by his employer, the employer exclusively shall be entitled to exercise all the economic rights in the program, unless otherwise agreed.” (auf Deutsch: “Wird ein Computerprogramm von einem Arbeitnehmer in Wahrnehmung seiner Aufgaben oder nach den Anweisungen seines Arbeitgebers geschaffen, so ist ausschließlich der Arbeitgeber zur Ausübung aller vermögensrechtlichen Befugnisse an dem Computerprogramm berechtigt, sofern nichts anderes vereinbart ist.”)

I interpret this to mean that so long as I don’t write something for work, in order to facilitate my work, or at the instruction of my employer, then I retain the economic rights. Very straight-forward and sensible, isn’t it?

I also assume, although I haven’t seen it stated here, that you shouldn’t use work resources for personal projects, or work on personal projects during work hours. Maybe this is more to do with the typical employment contract than copyright, but I could imagine it forming part of a copyright claim’s justification.

In the event that there is a dispute, however, it is a civil matter that can be dealt with in a number of ways. The suggested way would be to discuss it and reach an amicable solution, rather than involve expensive lawyers and courts. The best solution is to avoid potential disagreements in the first place, by making your company aware of any private projects which could cause problems beforehand. You don’t require their approval, nor do they have the right to deny it, but you should have them acknowledge it, preferably in writing.

The next thing to consider is your employment contract. In Germany, they assign a lot of weight to contracts and quite a lot is allowed (although there are more restrictions when it comes to employment relationships). My work contract, for example, considers work in a certain field to be somewhat “protected”, and for that field I should seek approval from the company before openly releasing any work to the public. Luckily I have little interest in that particular field.

As a general rule, you shouldn’t publish anything openly which could cause harm to your company. It has nothing to do with copyright – but as in any contract, you have a fiduciary duty to protect the interests of the other party, plus it’s just common sense to maintain a good working relationship with your company.

So my friends in Germany, go forth and write software!! :-)

— by Robert Thomson, created 2nd Apr, 2013, last modified 3rd Apr, 2013 | Tags: Tech, World

Ansible for System Orchestration

I recently stumbled upon Ansible, which has an interesting approach to managing systems. It's not what I'd choose for a large scale organisation with lots of sysadmins and developers needing to maintain standards, but for the lone admin or small skilled group with good version control habits, it might be the ideal choice.

The primary features of Ansible are:

In every large organisation where I've worked, we've always used a system management database of some sort (usually home rolled) to keep track of the systems and usually their (top-level) configuration. In my current position, I have access to several several thousand systems and occasionally have to run commands on a subset of them. Until now I've been using an inventory-search script that I wrote to extract system lists, and GNU parallel to execute commands in parallel, but I've found it limiting at times (and a mess of hostlists, and failure scenarios are problematic)

In Ansible, I found the external inventory better suited to small-scale system management, but it didn't take me long to work around the limitation with my own wrapper script.

As an example, I can now run "ans app=foo dc=lon01 env=prod -a uptime" and it will run the uptime command on all "prod"uction systems configured to run the "foo" app in the "lon01" datacenter. I could also run "ans '*.foo.lon01.*' -a uptime" to match based on the FQDN. ALL the matching is now done by my inventory script, and ansible just uses the results en-masse. My script is a lot more efficient than Ansible's grouping functions.

At home, I store information about my local systems/VMs and cloud-based VMs in a stateful cluster manager, and I have an inventory script for that too. I'm also more likely to use playbooks at home, as Ansible is lighter weight than running Puppet or Chef agents, but has much of the same functionality.

— by Robert Thomson, created 3rd Mar, 2013, last modified 3rd Mar, 2013 | 2 comments | Tags: Tech

What *isn't* Puppet?

What isn’t Puppet?

  1. Puppet is not a scripting language.
  2. Puppet is not an orchestration system.
    * Puppet is not well suited to VM orchestration.
    * Puppet is not well suited to application release management/complex upgrades.
  3. Puppet is not a user management system.
  4. Puppet is not a good configuration database.

Of course you can do all of these with Puppet.. but it’s just not the best solution for them. Sometimes you should put down the hammer, and go look for a wrench.

— by Robert Thomson, created 6th Mar, 2012, last modified 11th Mar, 2013 | Tags: Tech

An Extensible Puppet ENC with class & parameter overrides (& more)

Puppet ('s a great tool for system automation - out of the box it comes with most of the functionality that you'll ever need, and a DSL that's easy enough to let junior sysadmins and developers maintain their own recipes. But, depending on the kind of workhorse you want puppet to be, it's possible to hit a wall. For my company, we had too many developers, many system administrators, multiple releases in production simultaneously, and a change control to rule them all (but managed nothing). We also had the odd typo in node definitions causing incidents.

One possible answer to this is to employ an External Node Classifier, which in simple terms means an external script that Puppet calls to get the node definition, or at least the classes and parameters (and the environment, but don't rely on that due to an outstanding bug). An ENC cannot call definitions or puppet functions directly however - it can just pass in classes, class parameters, and global parameters. It's therefore up to you to make your ENC more intelligent.

I think Dan Bode of Puppet Labs gave me the idea of using YAML at first, over a year ago, and I whipped up a basic version in a couple of hours.

Over my long weekend, I decided to write a new ENC library from scratch that would provide the basics that one might want in an ENC, but allows it to be extended with plugins. You instantiate it for a node with some seed data, pass in a loader object, and then include one or more input/override files. It will take care of class, parameter, and substitution-variable overrides, and will spit out a nice Puppet 2.6+ ENC-compatible hash.

The test_puppetenc.rb file is the best source of documentation.

The Code:

— by Robert Thomson, created 24th Jan, 2012, last modified 24th Jan, 2012 | Tags: Tech

AMQP command line clients in C

At work, AMQP looks to be the message queueing protocol that will be targeted in the future. One team there implemented some simple clients in C, that would send messages, and also listen for messages and launch a program on receipt, passing in the routing key and pointing to a temporary file containing the message. I didn't get to see their code, but was given RPMs with unnecessary dependencies that I couldn't be bothered to get running on my Ubuntu box, but I thought that low resource shell-scriptable AMQP send and receive programs would be a good idea for the world at large, so over the course of a couple of weekends I extended RabbitMQ-C's examples to have the same functionality. I compile librabbitmq statically into the binary, and the resident memory footprint when running is 500kb.

Let me present rmt's AMQP Tools:

For security, you could cryptographically sign the messages, or just rely on AMQP's security. libRabbitMQ-C only supports the plaintext SASL authentication method at the moment, and SSL isn't supported (try stunnel).

— by Robert Thomson, created 24th Oct, 2010, last modified 24th Oct, 2010 | Tags: Tech

Awesome window manager

The Awesome window manager has me slightly impressed.. who knows whether I will continue to be impressed in a few days, but for now, I am. I found myself somewhat frustrated with the standard GNOME desktop when using multiple monitors in a Xinerama layout (the default if you just plug a screen in). Awesome is a tiling window manager.. in the past, I haven't been inclined to use one, because I work mostly on a single screen, or I have enough screen space for all my current work to be focused. My current work environment, however, involves a bit of task switching, and having a big monitor on my desk as well as the small laptop screen means I want better control of my windows.

The key feature for me is the ability to have independent virtual screens on each monitor, despite it being Xinerama. Most of the time I want the Windows VM showing on the laptop screen, but not always. In my current work environment, having tiling windows is actually proving quite nice, and it's helping me to be a little more organised. I split my tasks between virtual screens, and keep an eye on the Windows VM & outlook on the laptop, so context switching is easy.

Of course, some windows should be floating, and Awesome supports that just fine, as well as allowing each virtual screen to have its own layout style, and easily switch between them.

— by Robert Thomson, created 20th Apr, 2010, last modified 20th Apr, 2010 | 1 comment | Tags: Tech

Python interface to Linux's input_event and uinput

A little while back I bought an ASRock 330HT as a media PC. Aside from the fan being a little noisier than I hoped, the remote control had no Linux driver so I bought a Hama MCE remote control after reading that it works well under Linux. Unfortunately, while it works fine under Linux, it's got hardcoded key values and not every key is unique. As an example the "i" info button is actually a prolonged right click, a couple of the keys send *exactly* the same keycodes as others (Play and Pause), and there are a bunch of buttons which generate Control-Shift-foo and Alt-bar and so on. I guess under Windows Media Center these all serve a specific purpose. Under Linux and its media center programs (such as XBMC) they don't (and XBMC's input layer is a bit restrictive).

I had my toys, but things weren't working exactly as I wanted. What's a guy to do? I got hacking, of course.

The Linux Input Subsystem is pretty clean nowadays and detecting the correct device and reading a few bytes from /dev/input/eventXX isn't all that hard. Within a couple of hours I was reading the bytes from the remote and pretty printing them as I received them. Unfortunately, this lead to the realisation (as shown above) that the remote's a little screwy and that a few hacks would be necessary to make things work as I wanted. But I was having fun and I was learning something.

The next step was deciding how to use these key events. XBMC has an interface to LIRC and another chap had written a python script making the HAMA MCE remote produce LIRC events. I tried it, fixed some bugs, and it worked.. but it was unsatisfying.. the MCE remote also has a mouse cursor (there are two input devices generated - a mouse and a keyboard - but the mouse device also sends a few key events that I care about as well as the all-important info button).. I started to explore the uinput device which lets you create a fake input device and write to it. Figuring out how to initialise it was a little tricky but I found some similar code and worked it out.. after that it's just writing input events to the device.

My initial idea of just reading all input, munging a few things, and passing the remaining events directly through proved somewhat problematic. If you leave a stranded Control or Alt key pressed down then you have a serious problem and in my tests I had to kill my X session a few times. A better way is to read all the input and have a virtual keyboard that handles the MOD keys, does the processing of normal keystrokes, does the translations, and then converts the translated events back into input events for uinput, ensuring there're no dangling modkeys.

The final step was to create a versatile translated step. I decided that I wanted a configuration file, instead of coding the translations all in python.

Here's some snippets from my config file:

# simple 1-1 translation
KEY_HOMEPAGE-down = send KEY_ESC-down
# the info button/right click magic
BTN_RIGHT-down = set rightclick time.time()
BTN_RIGHT-up = if (time.time()-rightclick) > 0.3; send KEY_I; else; send BTN_RIGHT
# compound key sequence example
Alt-KEY_F4-down = wait
Alt-KEY_F4-down KEY_F4-up = send KEY_ESC; clear

The wait action means to add the current key event to the match criteria for the next keystroke. clear then clear's the list.

Code will be online soon at

— by Robert Thomson, created 3rd Feb, 2010, last modified 4th Feb, 2010 | 2 comments | Tags: Tech

Python's SimpleHTTPServer and unnecessarily fragmented HTTP sessions..

A couple of months ago I discovered the joys of an Internet Explorer bug revolving around HTTP.. If a HTTP redirect is returned from a server but the Location: header is not in the same TCP packet, Internet Explorer would have some serious issues. This seems to affect MSIE6, MSIE7, & MSIE8. This suggests some poor separation of the network layers in Windows & IE, but it also highlighted the strangeness of SimpleHTTPServer's implementation, which is used by paste's default server.

Every time it sends a header, it flushes the buffer.. the end result is that you often get one TCP packet for each header, and then the data afterwards.. obviously this isn't the most efficient use of the network. Headers and data should be buffered before sending.

The solution was to switch to using FLUP & FastCGI for both the testing and production environments. We're using Nginx, and this combination proved very robust. Wireshark certainly showed much nicer results.

— by Robert Thomson, created 19th Dec, 2009, last modified 19th Dec, 2009 | Tags: Tech

Django: Forcing a Multiple Choice Widget into a delimited CharField

This works, but is there a better way, like only modifying a Widget or a Field?

class TextCheckboxSelectMultiple(widgets.CheckboxSelectMultiple):
    Set checked values based on a comma separated list instead of a python list
    def render(self, name, value, **kwargs):
        if isinstance(value, basestring):
            value = value.split(",")
        return super(TextCheckboxSelectMultiple, self).render(name, value, **kwargs)

class TextMultiField(forms.MultipleChoiceField):
    Work in conjunction with TextCheckboxSelectMultiple to store a
    comma separated list of multiple choice values in a CharField/TextField
    widget = TextCheckboxSelectMultiple
    def clean(self, value):
        val = super(TextMultiField, self).clean(value)
        return ",".join(val)
— by Rob, created 15th Sep, 2009, last modified 15th Sep, 2009 | 2 comments | Tags: Tech

Grok/Zope: Redirect from Traverser

class RedirectModel(grok.Model):
    RedirectModel + RedirectView allow us to do redirects from Traverser's easily
    def __init__(self, request, url):
        self.redirecturl = url

class RedirectView(grok.View):
    def update(self):
    def render(self):

Then from your Traverser's traverse():

return RedirectModel(self.request, "/newURL/")
— by Robert Thomson, created 29th Jul, 2009, last modified 29th Jul, 2009 | Tags: Tech

SQLAlchemy, PyMSSQL, encoding problem .. FIXED

The Problem: MSSQL database with "LATIN1" (case-sensitive) encoding, SQLAlchemy and PyMSSQL. Fields with non-ascii characters were sometimes being returned double encoded.. The actual case is that SQLAlchemy passes parameters to the DBAPI as however you pass them to SQLAlchemy.. the convert_unicode option doesn't seem to change the encoding of unicode parameters when saving, only the SQL string.. which seems kind of inconsistent. I'm actually not 100% certain of the interactions here, and who is to blame - PyMSSQL or SQLAlchemy - but let's split the difference and blame Microsoft, they're always a good target.

The Solution: A custom type:

class EncodedString(types.TypeDecorator):
    impl = types.String
    def process_bind_param(self, value, dialect):
        if type(value) == unicode:
            return value.encode(dialect.encoding, 'replace')
        return value
    def process_result_value(self, value, dialect):
        if value and type(value) == str:
            return value.decode(dialect.encoding)
        return value

and instead of defining a Column with types.String or types.Unicode, use EncodedString.

It looks like types.Unicode should do exactly this, but for some reason it's not working for me.

— by Robert Thomson, created 16th Jul, 2009, last modified 16th Jul, 2009 | Tags: Tech

Offline Web Apps on Maemo with Tear

It's possible via a slightly non-standard method to have offline web apps on the Maemo platform. Tear is a webkit based browser, and when using a recent webkit (more info here) you have access to the local storage facilities as specified in HTML5. The problem is that full AppCache functionality doesn't exist/work yet..

However, you can host your apps locally on your maemo device, and by setting "document.domain = your.domain" in javascript, you can use the security context of the remote domain when making requests.. When you're offline, you just return information from the local data stores, but when you're online, you can get new information easily. Neat, eh?

Still, I'm looking forward to AppCache support so I don't have to save pages locally.

— by Robert Thomson, created 10th Jul, 2009, last modified 10th Jul, 2009 | 1 comment | Tags: Tech

Back from Berlin..

LinuxTAG was interesting.. I got a better grasp of the Linux and Open Source crowd in Berlin (and in Germany in general). I think my geek-side will be satisfied there. I've decided to check out Sugar-on-a-stick after chatting to them there (and pointed one of the developers in the direction of Numptyphysics, which just may appear as a Sugar package at some point). I'm looking forward to settling down in a proper apartment again, and setting up a media center for myself - A nice ATOM dual core + NVidia (ION) server combination (TEO-X had one there) should provide all the power that I need (with lower power consumption) for XBMC & a retro (& not so retro) gaming setup. :-)

Also interesting was Büro 2.0, which involves a shared workspace and services for Open Source companies and freelancers in Berlin. If I decide to be a freelancer there, this might be an option.

— by Robert Thomson, created 30th Jun, 2009, last modified 7th Jul, 2009 | Tags: Tech, World

Seantis Questionnaire released

Announcing Seantis Questionnaire, a Django Questionnaire/Survey application that I developed for Seantis GmbH in Switzerland.


Talk to Seantis GmbH about custom development.

Download Seantis Questionnaire from GitHub. It is released under the same license as Django itself.

— by Robert Thomson, created 15th Jun, 2009, last modified 18th Jun, 2009 | Tags: Tech

Directly implementing Zope 3's IAuthentication

Zope 3's Pluggable Authentication Utility doesn't automatically store a cookie saying that you're logged in if you've been authenticated once. Instead, it uses credentials and authentication plugins. For each credentials plugin, it passes the credential's output to each of the authentication plugins, and if any succeed, then it returns a Principal. It sounds logical enough, except that it does that every time, instead of storing a variable in my session stating that I'm already authenticated. Since I'll be authenticating against an external system, I didn't like the idea of checking the password on every request.. It wasn't immediately obvious to me how I could simply achieve this within the Pluggable Authentication framework, and I wasn't sure I needed the power of it anyway, so I decided to create my own IAuthentication implementation, and forget about the PAU altogether.

What it does

This implementation stores all the IPrincipal information (id, name, and description) as a signed cookie in the user's browser. By default, the session will expire after 6 hours of non-use, and the cookie's timestamp will be updated every 5 minutes with a new timestamp.

The Code

The signed string code is located in, and the authentication code in You also need to implement the authentication in your login form, and add it as a local utility to your application/site.

# coding: utf-8
import md5, random

SECRET='SomeRandomStringThatYouShouldNotShare' # CHANGE THIS

def make_sstring(question, string, r = None):
if r is None:
r = ''.join([random.choice('1234567890abcdef') for x in '12345678'])
m = md5.md5(SECRET + question + ":" + string + r)
return "%s|%s|%s" % (string, r, m.hexdigest())

def get_sstring(question, securestring):
string, r, md5 = securestring.split('|',2)
if make_sstring(question, string, r) == securestring:
return string
return False

# coding: utf-8
from import IAuthentication, IUnauthenticatedPrincipal, PrincipalLookupError, IPrincipal, ILogout
from zope import interface, schema, security
from securestring import make_sstring, get_sstring
from import hooks
from zope.traversing.browser.absoluteurl import absoluteURL
import time
from urllib import urlencode

class Principal(object):

    def __init__(self, id, title, description): = id
        self.title = title
        self.description = description

    def __str__(self):
        return "<Principal: %s>" % self.title

def make_authenticated(request, principal):
    id =
    title = (principal.title or '').replace("::","..")
    description = (principal.description or '').replace("::","..")
    tm = int(time.time())
    sstring = "%d::%s::%s::%s" % (tm, id, title, description)
    sstring = make_sstring('z3c_sstring_login', sstring)
    request.response.setCookie('z3c_sstring_login', sstring, path="/")
    return principal

class SStringAuthenticator(object):
    interface.implements(IAuthentication, ILogout)
    loginpagename = 'login'
    timeout_in_seconds = 60*60*6 # 6 hours
    update_timeout = 60*5 # how often to update the cookie

    def logout(self, request):
        request.response.expireCookie('z3c_sstring_login', path="/")

    def authenticate(self, request):
        sstring = request.cookies.get('z3c_sstring_login', None)
        if sstring is None:
            return None
        sstring = get_sstring('z3c_sstring_login', sstring)
        if not sstring:
            return None
            tm, id, title, description = sstring.split('::',3)
            tm = int(tm)
            now = int(time.time())
            if (now - tm) < self.timeout_in_seconds:
                principal = Principal(id, title, description)
                if (now-tm) > self.update_timeout:
                    make_authenticated(request, principal)
                return principal

    def unauthenticatedPrincipal(self):
        # not really sure what to do here, but it doesnt seem to hurt
        return None

    def unauthorized(self, id, request):
        site = hooks.getSite()
        stack = request.getTraversalStack()
        query = request.get('QUERY_STRING')
        camefrom = '/'.join([request.getURL(path_only=True)] + stack)
        if query:
            camefrom = camefrom + '?' + query
        url = '%s/@@%s?%s' % (absoluteURL(site, request),
            urlencode({'camefrom': camefrom}))

    def getPrincipal(self, id):
        principal = Principal(id, id, id)
        interface.directlyProvides(principal, IUnauthenticatedPrincipal)
        return principal

Install it as a local utility

In your application, you can add it as a local utility. Since I'm using Grok, I'll give a Grok example:

class MyApp(grok.Application)
    grok.local_utility(SStringAuthenticator, provides=IAuthentication)

You could of course also provide a setup function, to modify the string.

Local Utilities will only appear on NEW objects, so your existing applications/sites won't make use of it.

Authenticate from your login form

I wasn't sure the best way to do the actual authentication here, so I thought I'd just leave it up to the login form. If successful, the login form should just call auth.make_authenticated(request, principal) where Principal is an instance of an IPrincipal (you can use auth.Principal if you like, but there's no doubt a better way).

The Alternative using Pluggable Auth

I realised afterwards that it would be possible to do the same thing within the pluggable auth framework. You could do it almost exactly the same way, except that the credentials plugin should just return the cookie if it's set (credentials just returns a dict, so you're not limited to just a login/password), and the authentication plugin can just check if it's valid. The credentials plugin can still redirect unauthorized users to a login page. should be able to give you an idea about how to implement Pluggable Authentication.

— by Robert Thomson, created 9th Jun, 2009, last modified 18th Jun, 2009 | Tags: Tech

IPv6 enabled

My Xen server is now IPv6 enabled, and I have 4bn or so routable IPs. Yay!
And with a little bit of playing, I have IPv6 over the openvpn connection to my laptop, so now I have my very own public IPv6 network at home. And of course, yay for ip6tables!

— by Robert Thomson, created 22nd May, 2009, last modified 22nd May, 2009 | Tags: Tech


I got busy with work just after the N810 arrived, but nevertheless I’ve enjoyed having it and I find it quite a cool gadget. It’s screen is readable in all but direct sunlight, which helps to make it a great device when traveling. I found myself using it mostly as an E-Book reader when on the road, and loving Project Gutenberg and the Baen Library. I still have quite a few application ideas for it – largely revolving around making online content easily searchable and then available offline (eg. train timetable searches, hostel/hotel contact information & reviews).

As a side note, I suspect that the next device I buy will come with a PixelQi screen for it’s easy readability, low power, and versatility, but it will still be a few months until the first commercial device using it makes it to market.

I’ve developed a reasonably complex Django application for questionnaires/surveys for a medical study, which I hope to release publically in the near future, after the internal project is released. We’re still working on a name, as django-questionnaire and django-survey are both taken.

— by Robert Thomson, created 11th May, 2009, last modified 11th May, 2009 | Tags: Tech, World

N810, found some work, etc.

I received my N810 about 3 weeks ago, and despite the fact it says "N00" instead of "N810" on the front (N00 is the normal number for prototype models from Nokia, but its internal labeling and hardware specs are that of a N810), I am quite taken with it. Surprisingly, I use it mostly as an Ebook reader (it turns out that I really enjoy reading, I just hate carrying books around). As far as Internet goes, I mostly use the weather widget, and receive email updates on it too. Maemo Mapper is quite good to have around also, when one gets lost in a city.. but the GPS doesn't work that well. One of my primary reasons for geting the device itself was that I wanted something that I could hack on and program for easily.. it certainly fits the bill. Some of the standard software release's libraries are a little old, but they're easy enough to update. Python works fine, but I'm partial to using Vala for efficiency. All I need now is a good phone and data plan to go along with it.. but I think that will have to wait until I'm settled in Germany.

On the work front, I've found some programming work with a small Swiss company. I'm currently in Luzern, enjoying the weather and the lovely water views. I miss having the water nearby.. I'm doing some Django and Zope work, and will continue working from home from next week. Sophia's coming up on Thursday and we'll meet some friends in Switzerland and see some of the sights.

— by Robert Thomson, created 7th Apr, 2009, last modified 7th Apr, 2009 | Tags: Tech

Ordered N810

Yesterday I placed an order for the N810 Internet tablet. They've been talking about a successor to the N810 for months now (RX-51 / N900), but frankly, the N810 will probably suffice for years whether or not a new (and more expensive) model appears in the next few months.

— by Robert Thomson, created 13th Mar, 2009, last modified 13th Mar, 2009 | Tags: Tech

Parrot, Squaak, Perl6

I’ve been playing around with Parrot again lately, since it’s approaching a 1.0 release. I’m going through the tutorials for the Parrot Compiler Toolkit, and I’m quite impressed. Sure, I keep shooting myself in the foot since I don’t understand how everything interacts yet, but it does work. I like it because it makes it easy to figure out what’s really going on and how the conversion processes work, and it seems to be reasonably well documented.

One of my longer term goals is to have a suitable nice language for MUD world programming (which assumes that the programmers can’t be trusted), and I think I’d like to modify an existing language to fulfill the requirements. Of course, having almost no idea about compiler design seriously limits how much I can tweak without wreaking havoc.

— by Robert Thomson, created 10th Mar, 2009, last modified 10th Mar, 2009 | Tags: Tech

The Scala Programming Language

From the website: “Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages. It is also fully interoperable with Java.”

I quite like it.. I’ve never been a fan of Java, but I have been of good VMs and their well tested highly tuned libraries. Java and the CLR being the two common ones these days. I did briefly look at the website a few months ago, but disregarded it because the funny syntax didn’t appeal to me at the time.. I shouldn’t have. I recommend the book “Programming in Scala” (available as an PDF download from the publisher Artima or from a bookshop near you) in preference to the online tutorials and documentation that exist today.

Of course, given my tendency to get bored easily, I could be learning another language next week. :-)

— by Robert Thomson, created 17th Jan, 2009, last modified 19th Jun, 2009 | Tags: Tech

Updating Nokia firmware in Vmware under Linux

It was a bit of a struggle to get everything working... but here's how I did it.

1. Install VMWare 2.0.0 under Ubuntu Linux (due to kernel changes, the older version of vmware could no longer compile the kernel modules (asm/semaphore.h changed to linux/semaphore.h and the kill_proc function was removed from the kernel). 1. Browse to vmware's localhost web page and install the Firefox/Mozilla plugin for it, restart firefox. 1. Upgrade VMWare hardware version of my pre-existing Windows vmware image, add USB hardware if not there, let Windows hardware detect and install drivers as necessary, reboot if needed. 1. `mount -t usbfs none /proc/bus/usb -o devgid=500,busgid=500,busmode=664,devmode=664` (I set the devgid and busgid to my uid, which was the admin account in vmware. Maybe you use root and don't need to do something, or maybe it was just strange for me.. no idea.) 1. Browse to Nokia's website and install the Nokia Software Updater. Reboot windows. 1. Plug in the Nokia phone, select Nokia mode (as opposed to storage mode or another mode).. mount the device in the VMware image. Any read/write errors here are probably permission related in regard to /proc/bus/usb/. 1. Shutdown Windows, find the *vmware.log* file and do `"grep path:" vmware.log`.. find the entry corresponding to the Nokia phone. Copy & Paste the Numbers/Numbers after the path: bit. (maybe it's 1/2/3 for you or just 1/2) 1. Edit the .vmx file in the same directory, add `usb.autoConnect.device0 = "path:THOSE/NUMBERS"`. I also added `usb.generic.skipsetconfig = "true"` but I'm not sure that's necessary. You are actually binding that particular USB port on the host to that VMware image. You may want to undo this afterwards. 1. Start up the Windows guest again. The phone should now connect automatically. 1. Run the Nokia Software Updater.. cross your fingers.. if it doesn't work, uninstall and reinstall the software. Apparently the USB device drivers get screwed up often enough and this fixes them - at least, it did for me. 1. The Nokia Software Updater SHOULD now detect your phone.. if it doesn't, it's probably the device driver.. uninstall and reinstall Nokia Software Updater. It should find any new firmware for your phone and commence installing it.

The reason why you need to use usb autoconnect with the path option is because (it would seem) that the phone reboots into a firmware update mode and changes its identity.. and if it doesn't respond fast enough, then the Nokia Software Updater aborts (causing you to wonder whether you just turned your phone into a brick - luckily it wasn't the case for me).

Good luck!

(For the record, I ended up embarking on this journey when the Nokia Software Updater just stalled at the start screen on my girlfriend's laptop - a common problem, google would suggest.. Please Nokia, give us a Linux firmware updater!)

— by Robert Thomson, created 12th Dec, 2008, last modified 12th Dec, 2008 | 1 comment | Tags: Tech

gnokii --reset hard != reboot

I held a short memorial service for my SMS archive. At least I'd backed up my phonebook before doing it. Seriously, it looks innocuous if you read the gnokii manpage.

Update: Somebody on IRC said that it should be harmless. For some reason it wasn't on my phone.. but after the "reset" and the subsequent firmware update, everything works.

— by Robert Thomson, created 11th Dec, 2008, last modified 10th Mar, 2009 | Tags: Tech

Why git is better than X prompted me to take more than a superficial look at git (Version Control System), and I'm very impressed. I think I'm a convert, forever destined to look down on other VCS' with a slight air of disgust.

— by Robert Thomson, created 7th Dec, 2008, last modified 8th Dec, 2008 | 2 comments | Tags: Tech

Internet in Perugia

We have achieved that which we thought impossible. We now have Internet at our apartment in Perugia! It is a miracle! After over a month of wrangling with technical difficulties (which they never mentioned when we called to find out about our order - instead claiming that we were not customers, and trying to sell us something, thus leading us to assume our order was dropped, and to us placing a second order).. And so it is, that we now have Internet in our apartment. We probably could have had it a couple of weeks ago, had the modem (or instructions, since we had a spare modem) arrived sooner. But now we are officially customers, and we shall cherish our new-found status.

— by Robert Thomson, created 9th Oct, 2008, last modified 9th Oct, 2008 | Tags: Tech

Java, Android, XML

Does it irk other people as much as me that we are expected to read & write XML in order to build and run programs? What ever happened to real configuration files? How about editing tools for XML? Did half the world forget that XML is intended for machines, not humans? I always get a bit annoyed when I see end user documentation telling humans to edit XML. Sure, sometimes the tools come later, but quite often not, or they eat up 200 MB of RAM and require a GUI to edit a 30 line configuration file. EOR.

— by Robert Thomson, created 22nd Jul, 2008, last modified 5th Aug, 2008 | 1 comment | Tags: Tech

iRiver Clix2 arrived..

The iRiver Clix2 that I ordered finally arrived. It was supposed to be here after 2–3 weeks, but instead it took 8 weeks… in the meantime, I was given a free 1GB iShuffle by T-Systems (my new contracting client following Shell’s outsourcing), which I also quite like due to its small size and the clip, making it perfect for when I’m on my bike.

I’m happy with the Clix2. Right now, I’m using it to listen to the radio (and switching stations whenever they start talking their horrible language). It came configured in USB mass storage device mode, instead of the alternative MTP mode.. I’ve installed some new themes and some flash lite games on it.. I’ve transcoded some TV shows for it (320×240 OLED screen) .. wouldn’t want to watch a full length movie on it though.

In other news, Sophia and I are going to Perugia, Italy, later this month, to see if we can find some accommodation for her Erasmus year and my extended-holiday, and generally check out the place. We’re eagerly counting down the days until we’re both in Perugia!

— by Robert Thomson, created 13th Jul, 2008, last modified 13th Jul, 2008 | Tags: Tech

C++ gives me static

Defective C++

Scripting languages, managed runtimes and similar (CLR, Parrot and D) make me smile.

— by Robert Thomson, created 24th Apr, 2008, last modified 18th Jun, 2009 | Tags: Tech

Fix for strange colours when playing movies

I’ve an NVidia card with driver version 169. Sometimes when playing movies with Totem, the colours would be off. It took me a while to figure it out, thinking it was maybe a bug I couldn’t do anything about except for downgrading the NVidia driver (which seemed like too much work with Fedora).. Not sure exactly where the bug lies, but the hue setting for XV changes sometimes, resulting in odd colours. I think gstreamer may reset it sometimes. The quick fix is to install xvattr and run the following:

xvattr -a XV_HUE -v 0

— by Robert Thomson, created 2nd Feb, 2008, last modified 2nd Feb, 2008 | Tags: Tech

Laptop news..

I wiped Vista with a fresh copy of Fedora 8. I'm normally a Debian & Ubuntu person, but I decided to give Fedora a go for a change. It installed without any major problems. It would've been nice if it had started NetworkManager for me, so I could easily configure my wireless network.

Yesterday I decided to install a copy of Windows XP, which I had lying around. Once again, I'm not a Windows person and booted into Windows maybe twice a year on my old laptop, but since I have a license, I may as well put it on. There were issues with the boot CD and it not going any further. Recommended BIOS tweaks for the HDD to be detected did nothing either. So I am now trying an alternative method. I installed Windows XP under VMWare, installed all the drivers for the laptop that I could, and will try to copy the vmware image to the real partition, resize using ntfsresize, modify the boot sector and the boot.ini, and then see what happens. I will of course document my steps if I'm successful. Sounds neat, no? :-)

— by RobertThomson, created 26th Nov, 2007, last modified 26th Nov, 2007 | Tags: Tech

Ordered a Dell Inspiron 1520

I just ordered myself a new toy – a [[ Dell Inspiron 1520]]. The specs are:

My current laptop (Compal PowerNote CL50) has served me well for 4 years now, and I hope that this new laptop will serve me just as well. Choosing a new laptop is a tough decision and unfortunately having to make compromises is par for the course. I think I’ve made a good choice… not too small, not too big, and powerful enough to last me for a few years. It’s hard to find similarly spec’d laptops here with the same resolution.

For a case I ordered the [[–6459_7–31898142.html Kensington Contour Cargo Notebook Messenger]]

For your information, the other main contender was: the Acer Aspire 5920G ([[,39050489,40634980p-2,00.htm mixed review]] – a few differences in the EU market though)

So.. I bet you’re all wondering what I plan to do with a spankin’ new laptop, and what I couldn’t do on my current beloved laptop. Here’s what:

¹ That’s a joke, folks. I’ll be installing Linux ASAP. :-)

### Update – 9 Nov 2007

Still in preproduction. :-( Why, oh why, must I wait so long? Newsflash Dell – it’s not Just In Time if you have to wait days or weeks for parts. I’m a child of the Internet age – I want instant satisfaction. Hop Hop.

— by RobertThomson, created 3rd Nov, 2007, last modified 18th Jun, 2009 | Tags: Tech

Fedora's Build System - Mock and Koji now working

I’m helping to revamp the build infrastructure here. Until now, we’ve been using a bunch of home-rolled scripts and [[ Test::AutoBuild]].

We decided to look at Fedora’s build system and processes, since we’re developing RHEL and using RHN Satellite, didn’t want to re-invent the wheel, and Red Hat would similarly be trying to keep closely aligned with Fedora’s processes also. Yum repository support is official in RHEL5 and supported by RHN Satellite 5.

Mock was relatively simple. Create a Yum repository (extract latest version information from Satellite and copy the RPMS and comps.xml to a directory, then run createrepo), create a group and the directory structures, the configuration files for our build targets (x86_64 and i386) and it’s a go.

Koji’s a different story. Koji doesn’t support straight Username/Password logins – it uses SSL certificates and/or Kerberos to authenticate. I tried to follow the instructions for the [[ Koji Server HowTo]] but my setup is a bit different in that I’m installing everything on one machine. For me it meant that I didn’t need (and couldn’t use) a separate cert for kojiweb and kojihub..

For the ClientCert’s, you need a combined PEM file containing both the cert and the private key. If you don’t have that, you will see a python error with PEM something-or-other in it. This applies for ~/.fedora.cert (whatever path’s mentioned in $HOME/.koji/config for the cert), as well as for the WebCert in /etc/httpd/conf.d/kojiweb.conf, that used by kojid (cert= in /etc/kojid/kojid.conf), and of course that used by kojira (as explained in the [[ Koji Server HowTo]].)

The “Python ProxyDNs” option should correspond to the DN of the web host (or maybe hub, but for me it’s the same) .. if you get this wrong, don’t worry, you will (when the rest is correctly setup) receive an error from the web interface with the proper string.

Now that I can login, I just have to learn how to use Koji. :-)

Update: Koji turned out to be too much of a pain. It really is a complete distribution management solution, and we already use RHN Satellite for that.. so I built my own distributed build system based around Mock. It works quite well, although it lacks a lot of the pizazz of Koji.

— by Robert Thomson, created 25th Oct, 2007, last modified 18th Jun, 2009 | Tags: Tech

Cheap/free phone calls

I signed up with the VSP (Virtual Service Provider) VoipBuster. It's actually one of the many VOIP websites run by BetaMax, a German company. After paying €11.90 (€10 + 19% VAT) I can now make calls freely or cheaply to many countries for 120 days or until I run out of credit. From home, where I have a VOIP adapter, I can call Australia or Dutch numbers gratis. With my mobile, I can call a landline number in the Netherlands (relatively cheap, plus it's a work mobile) and then use the VoipBuster service to dial to Australia or Germany for 1¢/minute (OK, 1.19¢ with VAT). German mobiles cost just 20¢/minute, thereby undercutting the new EU roaming tariffs, if you ignore the local phonecall required from my mobile (as I do).

VoipBuster and the similar websites do occasionally switch which countries are free to call to, and rates do vary between the websites ( has 1¢ SMS', whereas VoiceBuster has 5¢), but overall it is (they are?) still one of the cheaper VSPs that exist. The multiple websites and the requirement to register using the Windows client turned me off at first (and the client requirement still does) but I can accept that the multiple websites are largely just a marketing game to generate buzz, differentiate themselves from the competition, and get around the one-size-fits-all model to exploit niches.

— by Robert Thomson, created 24th May, 2007, last modified 18th Jun, 2009 | Tags: Tech

Finally entering the 20th century

I’ve just purchased a Linksys SPA 3102 SIP gateway.. now I’ll just have to wait for delivery.

It will allow me to use my standard home phone as a land-line, and also route calls to/from PSTN & SIP through it.

I shall configure it to use my voxalot account ( but also register my landline number on E.164.

Not certain which SIP provider(s) I’ll go with yet.. I just know that I really didn’t want to give KPN (the local telco monopoly here) any more than the required €10/month copper-tax here.

Update: The SPA 3102 works quite well, but the configuration was a bit confusing for a first time telephony user – the supplied documentation is virtually non-existent. I’ve also configured it as a PSTN-to-SIP gateway, so that I can call home then make an international call through my VOIP accounts (and land-line calls are cheap). The quality is quite good. Now the only task now is to encourage enough other people to switch to VOIP also, and therefore remove all middlemen. I suspect, unfortunately, that it’s not as easy as one would hope to configure a VOIP adapter or phone, so getting sweet old Aunt Betty online may require a personal visit. Now, all I need is a cheap SMS gateway, since most of the time I’m near a computer and I could save considerable amounts and not be subject to the whims of bastard telcos

— by RobertThomson, created 27th Apr, 2007, last modified 18th Jun, 2009 | Tags: Tech

I have the contract..

I'm told that the paperwork will be sorted out by the end of today, but I've given the recruiter permission to accept on my behalf based on our discussed rates and start-dates. Now - to find accommodation. I may be one of the better paid homeless in Den Haag.

I'm actually going to stay with Thom May for a couple/few days when I first arrive.. but I don't want to impose so I'll do some frantic apartment hunting. I'll also use this time to get my SOFI (tax file) number, a bank account, and a dutch phone number.

It's quite sad to be leaving Passau, but Passau has no IT industry to speak of, and this is very much the kind of opportunity I was looking for last time I was looking for work in Europe.. but at that time I wasn't prepared to move too far from Passau. I'll probably keep my apartment here for now - I may sublease it, or just give up the lease.. but I've got a reasonable amount of stuff to move, so I'll have to come back and sort through things soon enough.

Update (9 Jan 2007)

I just heard, I start on Monday, and I get the rate I hoped for. I'm dusting off my suitcase as we speak. Farewell dinner/drinks on Wednesday.. probably train on Thursday.

Update (11 Jan 2007)

Leaving tomorrow morning at 11:22am, arriving around 8:30pm.. changing trains once in Frankfurt, and once in Utrecht in the Netherlands. I'm sure I'll be up for a good meal and a beer after that journey.

— by Robert Thomson, created 8th Jan, 2007, last modified 18th Jun, 2009 | Tags: Passau, Tech, World

New Bliki using Django

A few weeks ago I thought I'd jump into the 21st century and check out a web framework or two. I looked at RoR first, and after a little mind-bending I understood it reasonably well. It really requires you to be one with RoR to grok it fully. Shortly after that, however, I noticed the Netherlands contract advertised, which was of course for a Python developer. Since things were going well there, I decided to focus once again on Python.

There are a few interesting web frameworks for Python.. but my eyes fell on Django and I gave it a go. I must say, I'm impressed. It's far less hyped than Rails, and it's really quite simple. I wrote this entire website with under 350 lines of Python code, plus HTML templates. The administrative interface is quite neat, although I wish it were a little more customisable and a little less tied to the database structure. The basics required only 200 lines of code -- comment spam checking, email integration, and custom captcha code took up the rest. It took quite a while to get here, though, as I was learning on the way.

I'm using SQLite, because I really don't need or want anything more. In fact, I still think that a database is overkill for a blog/wiki. I am, however, mildly impressed with the result. I'm also using Markdown (modified) for a more wiki-style of editing.

— by Robert Thomson, created 3rd Jan, 2007, last modified 18th Jun, 2009 | Tags: Tech
... now get off my lawn!