Updating the Python Job Board

Have you ever looked at the Python.org Job Board? It lists in chronological order postings sent by recruiters and employers looking for developers with skills and interests in Python.

The Job Board has been around since at least 1997 (I checked using the WayBack Machine just to see what it looked like) and not too much has changed since then. New postings are sent by email and a volunteer cuts and pastes them into the head of a text file, fixes the formatting and checks it into Subversion. Instead of the original HTML template, now an ReStructuredText template is used which causes all sorts of problems because few people know what ReST is or have any idea how to format their posting. Eventually, the posting (if accepted) turns up on the web page and later in the RSS feed and on Twitter.

Here are some stats: Currently, there are approximately 40 jobs posted per month. The web page is in the top 10 entry pages for the site month after month gathering around 80k hits per month. The RSS feed gets around twice that and some third party sites re-brand the content and redistribute it which we have no stats for. The Twitter page has around 230+ followers.

Everything on the page is static and the Python infrastructure team has been against dynamic content for good reason but in a time when Google's appengine allows the the easy construction of a dynamic website with additional server capacity on demand, maybe it is time to update the Jobs Board. Two things to change are the process by which jobs are added and the lack of useful search or visualization ("where are the jobs?", "where are the job-seekers?").

The largest part of the work is converting the incoming posting into valid ReST and checking the page after pasting the new posting in. That's it. Emails are answered with a templated response (for both additions and removals) and checking into Subversion takes seconds.

There is good progress being made on a web front-end that will allow employers to directly submit postings and see how they look in their final HTML form. These will be held pending approval. The use of the browser will allow postings to be added and updated from anywhere instead of only from a machine that has the development environment currently required. There is also some intent to support submission by email but that has details yet to be worked out.

All this will be ready for some friendly testing soon (mid-November) and hopefully will reduce the response time and the amount of work needed to add postings as well as adding feedback to employers about how their ad is performing.

If you have comments or suggestions or would like to help, please let me know!

one of those days..

I have not written anything in here for a while because:

1) I'm too busy
2) I don't have anything interesting to say
3) No-one reads this, right?

A funny thing happened today.. I found this post from Steve Yegge that brushed aside my dejected writer's block and reminded me why I started blogging in the first place - partly to help me remember interesting things and partly to improve my writing skills. Or lack of.

If you do read this, leave me a comment.

Moving House

Not literally moving house but I decided it was time to upgrade my python.org local build machine from Redhat ES4 to something more recent so I fired up a Fedora Core 11 VM that I haven't been using so far, copied over my SSH keys and started checking out the website svn repository.

There was nothing wrong particularly with ES4 but Xemacs is a little long in the tooth now and didn't want to mess the machine config around to install Komodo or Eclipse when that VM was really for testing releases at work.

The checkout finished without a hitch (can't get PEPs but that was expected). Now to install docutils, yaml and pyramid.

I am using an RPM-based system so I have a choice of installing the packages from source, installing from easy_install or RPM. My preference is RPM if only to make sure that I have a single place cataloging all software installed on system paths. If I wanted to get into virtualenv or some other sandbox arrangement (not today) then I could go hog-wild and install all sorts of crazy stuff.

I install docutils and run make again and it fails looking for yaml. There's only one YAML available from Yum that is Python related but is it the one that pydotorg is using? A quick search of the wiki and a broken link or two later and I decide to give it a go.

The README in the checkout root directory tells me I need mako (a templating library) as well. Yum adds in python-beaker (a caching library or WSGI middleware layer) and libyaml and away we go.

The next make command succeeds so start the server and presto! Everything is working, time to shut down the ES4 image.

Start to finish.. around 15 minutes. Lots faster than the first time or two when the site went to using Pyramid.

Python Job Board Tag Cloud

Using TagCrowd (http://www.tagcrowd.com), I created this visualisation of the most common words on the Python Job Board (after removing meaningless, high-frequency words like contact, description, e-mail etc).

ability (90) able (47) agency (43) analysis (40) applications (214) apply (41) architecture (40) automation (39) based (71) build (101) business (48) ca (39) candidate (98) client (66) code (90) com (308) communication (56) company (80) computer (87) create (42) css (56) currently (38) data (98) database (55) degree (43) design (143) django (111) email (56) engineer (106) environment (110) equivalent (38) etc (45) excellent (48) following (37) frameworks (73) highly (44) html (59) http (159) ideal (43) implement (45) including (96) information (44) interest (62) java (46) javascript (84) job (261) join (39) knowledge (140) language (69) lead (49) linux (106) looking (130) maintain (45) management (108) mysql (65) network (39) open (58) opportunity (51) performance (57) php (50) platform (40) please (71) plus (76) position (77) posted (186) problems (49) product (46) programming (127) project (80) python (488) related (38) requirements (85) responsibilities (72) role (42) science (75) security (42) senior (42) services (76) site (47) skills (207) software (266) source (57) sql (41) strong (123) support (58) systems (105) team (185) technical (83) technologies (111) telecommuting (87) testing (117) tools (60) uk (57) understanding (53) unix (42) usa (87) used (85) web (362) www (99) years (115)

created at TagCrowd.com

Using Git Behind a Firewall

I am doing a lot of work involving the Samba code base right now and it has frustrated me that I cannot check out the trunk using git because I am behind a firewall.. then again the tarballs for the stable releases have been adequate until I realised that my tinker toy modifications needed to be made into patches and that was tricky..

Long story short:
if you try 'git clone git://git.samba.org/samba.git samba' and it fails because of a firewall issue then change the scheme from git:// to http:// and presto!

I think this only works if the repository you are checking out has a gitweb installation.

Python in a web page

"Gestalt is a library released by MIX Online Labs that allows you to write Ruby, Python & XAML code in your (X)HTML pages. It enables you to build richer and more powerful web applications by marrying the benefits of expressive languages, modern compilers, AJAX & RIAs with the write » save » refresh development model of the web."

Python on a web page! In Silverlight .. which means it is time to learn Mono and .Net and all that stuff.

Update: I did nothing to learn any of that stuff then one day I walked into the office and they said "oh, we need you to work on a C# project.. on a Windows box"

SQLite Gotcha 2

Creating a table with a column of type INT PRIMARY KEY will not result in automatic assignment of a unique value into that column when new rows are added. Why not? In part because INT is not a type in SQLite (but you can use it and the column will have integer affinity but you have to provide the value) and partly because rowid assignment only occurs on columns defined as INTEGER PRIMARY KEY.

Take a look at the documentation on SQLite data types (http://www.sqlite.org/datatype3.html). You may also be surprised to discover that you can declare a column of one type then store data of a different type in it.

SQLite Gotcha

ROWID is a reserved word in SQLite which seems reasonable.. but it has a synonym OID so if you declare a table with a column called OID then select from that table without using the table name as a prefix, you will not get what you expect.

If you are using an ORM to abstract away the database then you may never realise what is going on..

Unicode Gotcha

What will this give you?

>>> astring = None
>>> print unicode(astring)

If you said None then you will be surprised to find the answer is u"None" which is not at all the same thing. Really messed up my day..

Bing Sucks

Bing, the new search engine from Microsoft, sucks. This conclusion is based on one simple test:
Enter "internet search" as your search term and the results do not contain Yahoo, Google, Bing or LiveSearch. Dogpile is the number one hit.

Remove the word internet and now you get Yahoo listed top.

Try the same test on Yahoo and it lists Bing first in a sponsored results box, followed by Yahoo and Google .. and then Dogpile. Huh?

New Things

I am trying to be more organized on my Mac. As part of this I have added some new apps: Cyber Duck (a pretty good FTP client) and Syrinx for twitter. They work, they don't get in the way and they look they are part of the system.

Regex Exploration

The Question

I was asked a question on the way out of the office at the end of the week: what is the difference between a regex match and a regex search?

It didn't seem like a difficult question but it stumped me for a little while. Both a search and a match should use a regular expression and evaluate it against a text. Perhaps a regex search traverses the input string and returns one result at a time and a regex match does.. hm.. or maybe a regex match is used for validation and a regex search is more exploratory?

The individuals asking the question were looking at the Python regex documentation (or rather AMK's very excellent regex howto ) which says "match() function only checks if the RE matches at the beginning of the string while search() will scan forward through the string for a match" which is interesting because it means that the Python regex match effectively inserts a leading anchor before attempting evaluation. Both functions return a MatchObject.

Wait a minute, you may say, you have the documentation to answer the original question! Not exactly because this question was not inspired by an abstract search for knowledge but a performance problem. The other interesting part was that Python documentation was being studied but the code is in C++ with no Python involved.

If you need a regex library for C++ then a good place to look is at the Boost home page which is “...one of the most highly regarded and expertly designed C++ library projects in the world” and has libraries for all sorts of things like graph theory, linear algebra or interprocess communication. So what does the Boost Regex library say about match and search? How about this: "the result is true only if the expression matches the whole of the input sequence. If you want to search for an expression somewhere within the sequence then use regex_search". Of note is the fact that the return type of match is boolean.

Did you get that? The Python regex match/search answer is a matter of the start position of the match and the Boost C++ answer is whether the input sequence is consumed.

Performance Evaluation

Another alternative for a regex library for the C++ developer is the Perl Compatible Regular Expression library (see the website) which offers a single execute function instead of a search or match function, with a flag parameter modifying behaviour. One suggestion I heard was that the PCRE library performs faster than the Boost library because "maybe the Boost library sucks?" but as Carl Sagan said "Extraordinary claims require extraordinary evidence" or, in other words, how do you figure?

I wrote a small app that generates multiple random strings by appending together words randomly picked from a small dictionary and then evaluates a number of regular expressions. Here's the results (slightly re-arranged):

Boost regex search, pattern: yellowgreensapphireruby.*bluered
12.50 s

Boost regex search with continuous flag, pattern: yellowgreensapphireruby.*bluered
0.07 s

Boost regex search with any flag, pattern: yellowgreensapphireruby.*bluered
12.51 s

Boost regex match, pattern: yellowgreensapphireruby.*bluered
0.05 s

PCRE regex, pattern: yellowgreensapphireruby.*bluered
1.47 s

Boost regex search, pattern: ^yellowgreensapphireruby.*bluered
6.59 s

Boost regex search with continuous flag, pattern: ^yellowgreensapphireruby.*bluered
0.07 s

Boost regex search with any flag, pattern: ^yellowgreensapphireruby.*bluered
6.57 s

Boost regex match, pattern: ^yellowgreensapphireruby.*bluered
0.05 s

PCRE regex, pattern: ^yellowgreensapphireruby.*bluered
0.02 s

Boost regex search, pattern: yellowgreensapphireruby.*bluered$
14.16 s

Boost regex search with continuous flag, pattern: yellowgreensapphireruby.*bluered$
0.07 s

Boost regex search with any flag, pattern: yellowgreensapphireruby.*bluered$
14.15 s

Boost regex match, pattern: yellowgreensapphireruby.*bluered$
0.05 s

PCRE regex, pattern: yellowgreensapphireruby.*bluered$
2.01 s

Boost regex search, pattern: ^yellowgreensapphireruby.*bluered$
6.59 s

Boost regex search with continuous flag, pattern: ^yellowgreensapphireruby.*bluered$
0.07 s

Boost regex search with any flag, pattern: ^yellowgreensapphireruby.*bluered$
6.59 s

Boost regex match, pattern: ^yellowgreensapphireruby.*bluered$
0.05 s

PCRE regex, pattern: ^yellowgreensapphireruby.*bluered$
0.02 s


Eric's Law

If you think the compiler is wrong, you are 99.999% certain to be the problem.

*This number may vary according to the platform and the compiler.

Web2py rocks more than the alternatives?

I need to find some spare time, perhaps just some of that discretionary sleeping time, to take a look at web2py. I have a small project in mind - updating my friends website to have a simple calendar and appointment system - but I have been getting slammed at the office porting an application to AIX (yes, it's still in use) and busy at home with the twins.

Linus comes to his senses

According to a link on Slashdot, Linus Torvalds has finally come to his senses and now agrees with me that Gnome is a better choice for a desktop than KDE.

Best Mac Downloads of 2008

Brad added this link for the best Mac downloads of 2008 to his del.icio.us account and it has several interesting entries including Timemachine backups over the network, Dropbox (file synchronization over the net) and XBMC (a very nice looking cross platform media center).

Writable NTFS on Mac

I use an external HD to to carry large files between home and office but the Mac (or at least Leopard) doesn't support NTFS as a writable filesystem so the arrangement is only useful when all the new data is at the office and nothing needs to be changed at home. Obviously this is not ideal.

MacFUSE allows the addition of new fileystems to the Mac including NTFS-3g which originated on Linux (I believe). Installation is quick and simple (install MacFUSE, install NTFS-3g) and after a reboot, NTFS partitions are writable without any configuration required.

A quick search turns up a number of other filesystems that could also be used through MacFUSE incuding two that will encrypt a partition and one that will allow the se of a Gmail account as storage.

Illegal Use of Linux in the Classroom

A teacher in Austin, TX did not like what she saw when she found a student in her class experimenting with a liveCD from HeliOS.. so she confiscated the CD, reprimanded the student and then wrote a letter to his after explaining he was potentially liable to a civil suit and that the kids should be using Microsoft products which they (MS) would be happy to supply for free..

Wow.

Breathlessly awaiting the followup after the teacher and the parent meet in the superintendent's office.

More Uses For An OLPC

Video player for toddler.

Meghan likes watching DVDs in the minivan and she likes banging on the keys of the OLPC so I combined the two. I used Handbrake to rip one of her favorite DVDs then loaded mplayer onto the OLPC and played back the movie from a a thumb-drive.

VLC and mplayer had similar problems with freezing before announcing the file was longer available. I tried turning off power management with no improvement. It wasn't until I disabled the radio that playback was successful .. then I found the defect mentioned earlier.

Perl is Dead! Long Live Perl!

This somewhat risible survey reveals that Perl has almost fallen out of the top 10 programming langauges in use. Given the many different ways that statistics can be mashed together and the loose method of collecting the numbers, I think it is unlikely that they can claim 3 decimal places of accuracy for any of this.

On the other hand, using activity as a barometer of interest in a langugae and perhaps what to brush up on to stay/get employed, take a look at stats on Ohloh. You can pick a basket of languages and compare check-ins and TLOC and decide that actionscript is not about to overtake C++ any time soon.

This page lets you look at a ramge of stats for a language off their long list while this will let you compare a wide range of languages with a pretty chart: take a look at this comparison showing that the volume of Python checkins is growing faster than Perl and Ruby together. Or maybe it is showing the rate of Python projects being added to Ohloh? Who can say?