articles in 2007

From the very excellent (UK Based) Media Molecule, a Lionhead spin-off company. I had an interview with Lionhead many years ago, long story short, I didn’t get it :(

Maybe youv’e noticed, maybe you haven’t, but quite a lot has changed at this site in the last few days. Its all behind the scenes work though, with the exception of a few style changes and new pages here and there. So what’s new;

• Switched from Typo to the much better, Mephisto
• Microformats have been applied to all entries (hAtom)
• My updated footer profile and all comments have also got the hCard treatment
• The entire layout is a bit more liquid (in more ways than one) Its now fluid rather than fixed – (drag your browser width) and built on liquid templates (part of Mephisto) There is a min-width defined – which (of course) falls over on narrow IE6 pages :(
• Gone are the old AJAX comment submissions (in Typo) and AJAX on ‘search-as-you-type’ has also been removed.
• I’ve added my own code to handle Gravatar caching, which was very easy to bolt into Mephisto. I’ll probably do another post on how this was done soon.
• I’ve upgraded the server to the latest Lighttpd 1.5 RC. With the new mod-proxy-core module, it means I no longer have to work with a mongrel-pound-lighttpd stack. The site is just served straight-up, from Lighttpd to 3 Mongrels.
• As usual the newer ‘more about me’ page still needs more work.

A few weeks ago I took on some work theming ZenCart, a popular open source PHP/MYSQL shopping cart system. After investiagting a number of solutions (including my much prefferred choice, Shopify), the client decided on something that would offer a better financial solution, and would work directly with the RealEx payment engine.

As an ecommerce sites go, ZenCart is perfectly suitable for the end user. It has the usual features and structure you would find in other online shops (recent products, related purchases, categorisation, notifcations, stock management etc.) The problem comes when you need to theme it. Here’s just a few of the issues that came up;

• Abolutely no attempt has been made to sperate PHP code and HTML presentation. Opening up a templated page, means trawling through mountains of includes, function calls and PHP case statements to find the HTML you are looking for.
• ZenCart claims to allow a table-less layout to be easily customised, and while there are no tables, the xHTML markup is some of the worst I have seen; semantically incorrect at best, grossly bloated and invalid at worst.
• The promise of using a few CSS files to theme the shop is simply not true. In many pages with similar layouts you will find completely different class and id attributes for exactly the same elements.
• The xHTML is inconsistent, in some pages a product list might be shown in a table, in others as an ordered-list. Combined with bad CSS identifiers, the size of your stylesheet grows and grows.
• The admin panel seems to feature not one, but two built-in templating systems that (along with being confusing) allow raw-HTML to be entered by non-technical staff without any validation. I chose to avoid using them, to keep things simple.
• For a complete re-theme, there are literally hundreds of files that need attention in the themes/ folder; and many of these contain application code.

Now I don’t know all of the history behind ZenCart, but I have to say I think it over promises on simplicity (a lot). Rather than a ‘zen’ like experience, working with it will probably cause your hair to fall out.

Shopify from JadedPixel on the other hand, is an outstanding piece of software. I followed its development closely (via their blog) from early last year. They make use of the Liquid templating system, and provide a large selection of well designed themes to choose from. They have gone as far as developing an offline application, Vision allowing developers and designers to build the required theme templates without having to setup a development box. It is a great approach to building a simpler e-commerce solution, with the minimum of fuss.

The disadvantages being its closed source, and a hosted-only solution. Setup is free, but requires a credit card to activate – and you must agree to a 3% commission on all sales.

Today, I got my (almost daily) ‘hot products’ from Amazon, and here’s the email they sent.

Now picture your average (non-technical) home Windows user, faced with this choice in their in-box. Do you go for the Ultimate, since it sounds the best? Ultimate Warrior, you could shout! – ‘Premium’ sounds like you’re being ripped off – ‘Basic’ insults your intelligence, ‘Business’ could be the right one, but you’re not wearing a blue-tooth headset and talking out of your ass today.My initial answer choice would be ‘none of the above’, there are actually 7 different product names (!)

And what’s that Vista Resource Centre all about? Is it an optional extra, like alloy wheels you say? Well its NEW! so it must be good. And don’t get me started on the number of different Office versions too, no wonder Amazon have chosen to include 5 different books in the email as well. Lets face it, you’ll need at least 3 of them to make a decision.

Along with these ridiculous product names, Microsoft’s seem to be pushing their entire marketing campaign on the fact that their task manager, goes a little bit 3D. See the featured screen-shots in this BBC news article, and the WOW ad’s on TV.

My rant is over. Now decide which CEO goes to talks about creating passionate users?, Vista – you are coming to a sad realisation, CANCEL or ALLOW?

I’m only seeing this now on DVD, watch the trailer here; its a great show, and not a bad soundtrack either.

The local elections in Northern Ireland (read, ‘political hi-jinks’) are coming soon. I can imagine the thousands of bearded posters lining the streets of Belfast now. I hope and dream of a future where our country can move forward; life-like beard-hair stapled to every election poster! What do we need to make sense of all this nonsense? The Northern Ireland elections wiki by Mr. Stewart

Contribute away and help George and the gang to victory. Stray Taoist has some Tales from the homeland on the matter.

Some photos from the campaign trail have turned into a caption competition on flickr, here, here and oh, here too.

I think IRC is a vastly under-rated tool. It is somthing I used a lot in the mid/late 90’s and in my first few years at University on the old DEC Workstations. Back before Skype and before the web was getting ready for its next version number, IRC was the best way to chat online.

For a long time I forgot about IRC, until it became obvious that (with a channel bot) it can be useful for all sorts of things, beyond just regular chat. With a bot parsing and watching RSS feeds, your channel can include notifications for just about anything you want, SVN commits, Trac tickets, even exceptions thrown from your code (you’ll have to build a feed for that yourself).

So I looked into setting up a channel bot myself (something I haven’t done before). Looking around I played with infobot (Perl) and eggdrop (C/TCL) – before (rather predicatably) going with RBot (Ruby) I had a little hassle getting the bot to work on Debian. It requires the Berkley Database (and Ruby’s binding to it).

Of course the BDB that comes with GNU/Debian Linux (apt-get) is out of date, and you will need to compile the most recent version instead. If you already installed Berkeley DB on your Debian Box, uninstall it to prevent conflicts. Then get it from here (./configure, make, make install). Choose BDB 4.4 or lower since the Ruby BDB bindings are not compatible with the new 4.5) – I couldn’t find all of those caveats documented together in the one place, so that may be of use to someone.

Next grab Ruby’s BDB bindings – and finally make sure you can call the bdb library in irb. If you can do,

:~$ irb
>> require 'bdb'
=> true

- without throwing an error, then download and install RBot. Its fairly simple to setup and there are a large number of plugins to play around with. Rbot will install a some of these by default (including rss.rb) and you can extend them, or write your own in the bot’s local plugin directory.

To persist the bot, use a simple ‘keep alive’ script;

#!/bin/bash

proc=`ps -fu $LOGNAME | grep name_of_your_bot | grep -v grep | wc -l`
if test $proc -eq 0
then
 nohup rbot /home/username/.name_of_your_bot > /dev/null &
fi

And put it in a cron job to check its running (e.g. every 55 minutes);

# make sure rbot is running all the time
*/55 * * * * /home/username/rbotchk.sh

My work in progress Rbot (kafka) is sitting in #komura on irc.perl.org – expect it to be broken most of time as I add a few plugins and try stuff out.

While there are hundreds of IRC clients available, I have stuck to using IRSSI (a terminal client) its old-fashioned (I know), but simple and I can access it from anywhere (keeping it running on a screen). I’d also recommend Colloquy a decent graphical client for OSX.

Here are some links that may be of use to anyone getting started with REST Resources in Rails.

• Documentation on the orginal RESTFul plugin
• NESTED CRUD Resources in Rails
• REST and Rails no reason to be scared
• REST on Rails
• Simply RESTful Support – And How to Use It
• Rails nested resource scaffold

Some years ago I made use of a Windows based (and free) packet analsyer tool Packetyzer, from Network Chemistry) It did the job and proved to be invaluable for testing and building different packet headers – for projects just like this – but as a fan of Ruby, I had to see if this could be done in the language of my choice.

Just a few months ago I began doing a little research into GPRS packets for a potential ‘location-based’ mobile web service. To do this I was hoping to use Ruby and through pCap I was able to.

To start monitoring packets, you’ll first have to install libpcap – a low level packet capture library (authored in C) – from LBNL’s Network Research Group. The libpcap library grabs packets from any network adaptor on your system, giving you the raw data to work with. Doing this is immensely powerful, you could write listeners to monitor anything from email traffic to instant message conversations (provided the traffic isn’t encrypted of course).

Next, to allow us to write Ruby code to interface with this library, you’ll need to install pcaplet – This Ruby binding library makes writing capture programs easy. Its as simple as using require ‘pcaplet’ at the head of your Ruby code and your off.

I was able to use these libraries to examine TCP/IP packets from my mobile device (sent via GPRS over TCP/IP) to my server. The following Ruby script extracted the information I was interested in (data addressed to HTTP) using a regular expression;

#!/usr/bin/env ruby

require 'pcaplet'

# create a sniffer that grabs the first 1500 bytes of each packet
$network = Pcaplet.new('-s 1500')

# create a filter that uses our query string and the sniffer we just made
$filter = Pcap::Filter.new('tcp and dst port 80', $network.capture)
$network.add_filter($filter)

# the packet sniffer loop
for p in $network
  # if the packet matches the filter and the regexp...
  if $filter =~ p and p.tcp_data =~ /GET(.*)HTTP.*/
    # print all packet data for each packet that matches
     puts p.tcp_data
  end
end

Then save this and run the script with something like;

ruby packet_filter_regex.rb 
# you may need to run this as root if you have permission problems

The add_filter function, with ‘tcp and dst port 80’ – means all tcp traffic directed at port 80 on the adapter. This code will be a useful tool for me in another project -looking at streaming video and audio.

Another article at Linux.Ars roughly describes what is behind this and shows some more example Ruby scripts for packet monitoring (including an aim chat sniffer). There are also a few more example filters on the pcaplet site.

Apr-07 UPDATE – Sylvain Sarmejeanne has just released Scruby a Ruby application that works on UNIX providing a shell where you can perform packet creation, sending and sniffing functions in a Ruby-esque fashion. It requires libpcap and uses PCapRub (a mininal PCAP wrapper) rather than using PCAP directly. The package is well documented and has an irb-like interface, accepting command line input for debugging.

Following on from Carisenda’s ‘book reading shame’, here is a list of what I got through in 2006. In 2005 I finished every Stephen Ambrose book he’d published and generally overdosed on WWII factual history a bit too much. Men of Valour is worth a read if you’re interested in getting introduced to that type of thing.

Last year I read into the history of programming, hackers and Apple a bit —and while visiting Japan I was introduced to Haruki Murakami, without a doubt the best fictional work I have read in years. His books remind me of stories you’d expect from a Studio Ghibli production.

(I know some people can read this many books in a couple of months, I tell myself, reading slowly, I’m getting more out of them …)

• Programming Ruby – The Pickaxe, worth its weight in gold
• Out of the Inner Circle: A Hacker’s Guide to Computer Security – describes some very old hacking techniques used in the early days of the internet
• Hackers – a great book documenting the history of hacking and the advances in computing from MIT mainframes to modern day (well, circa 1989) home computers. Including some interesting true stories on Wozniak, Gates and the rise and fall of Seirra games
• Cuckoo’s Egg: Tracking a Spy Through the Maze of Computer Espionage – astronomer Clifford Stoll at Berkeley, tracks a hacker from western Europe for several months.
• Getting Real – 37 Signals, on building simple, focused software that does just what you need and nothing you don’t.
• iWoz – How Apple came to be, Wozniak’s story.
• The Victors – Stephen Ambrose
• Don’t Make Me Think! A Common Sense Approach to Web Usability – short and to the point, essential reading
• Why Software Projects Fail – an excellent read for anyone trying to argue the benefits of agile methods
• No Silver Bullet – Brooks (PDF)
• The Wind Up Bird Chronicle – Haruki Murakami
• Kafka on the Shore – Haruki Murakami

Anyone else got any books to recommend ?