Matthew Hutchinson

My commute home from work last Tuesday. From Tottenham Court Road, heading west towards Notting Hill, via Mayfair, MarbleArch, Bayswater Road, Holland Park Road and then south towards Brook Green. Recorded with Hyperlapse at 6x regular speed for some added drama.

TLDR; I setup Hubot from scratch with a bunch of different CM tools, Ansible, Babushka, Chef, Puppet, Salt, and finally plain old Bash

Show (without the tell)

A few weeks ago I began evaluating various tools for server configuration management. Before I start, I would like to point out that this exercise was inspired by the very excellent Taste Test by Matt Jaynes.

I do not want this post to detract in anyway from the hard work he has put into his guide. If you want an in depth look at the pro’s and con’s of each tool, including insight into their different approaches, I highly recommend you grab a copy. Please consider this post more of a “show” (without the “tell”). I’ll basically be pointing to the code and letting it do the talking.

Why explain this with Hubot?

To date, I’ve dabbled a little with Puppet and I’m no stranger to what we now call ‘the devops’. In the not so distant past I’ve rolled my own one-time scripts with Bash or Ruby. Trawling past repo’s on GitHub, you might even have seen my attempt to create a CM tool from scratch, Catapult (a mash-up of Capistrano, Ruby, Bash and some best practice docs from Slicehost).

More recently I’ve had a need to evaluate the best CM tools currently available, and I’ve been waiting for any excuse to play with Hubot (GitHub’s open source extensible chat-bot). So why not combine the two?

Of course, Hubot has been carefully crafted to be easily installed on Heroku, but from scratch, it has just the right combination of setup steps to exercise the more common tasks you’d want to know for any CM tool. In summary they are;

• Installing dependencies with a package manager
• Installing other (not so common) packages
• Cloning your code/project to the machine (e.g. with Git)
• Setting up your code/project with a user/group
• Adding configuration variables for your code/project
• Setting up log rotation (using a templated logrotate file)
• Service management for your code/project (e.g. with upstart)
• Starting your code/project service running

Show me the tools!

We basically want each CM tool to install and configure Hubot (including dependencies) and start the chat-bot service running. You can find my Hubot (named Henshall), here. If you are following along at home, I’d suggest creating your own bot to work with and pushing it to GitHub.

The best place to start is with a working shell script that spells out exactly what steps are needed. Using that as a guide I can then figure out how to express each step in the many dialects and dsl’s each tool provides. I should mention that Digital Ocean, Linode or a local Vagrant setup are great playgrounds for this sort of thing.

Without further ado, I’ve placed the nessecary files and README’s right here. Each README explains the “golden path” to setting up Hubot. I should mention that a fresh Ubuntu Server (12.10 x64) was used for all machines, and at the time of writing, these setup steps were correct with the latest versions of each tool. Ymmv with other distributions and as time marches on.

Bash

A bare bones shell script. Configuration via environment variables, logrotate and upstart templates are included inline. Of course this script is not designed to be ran more than once, it is a fire and forget approach to setting up Hubot on a fresh machine.

Ansible

Ansible (along with Babushka) was one of the easiest tools to use. All that is needed is a simple SSH connection to the remote Hubot machine. Simple yml files define ‘playbooks’ that configure your servers. A collection of built-in modules provide the playbook commands and templates are written with Jinja2 (a simple Python templating language).

Babushka

Babushka is another simple Ruby based CM tool. It takes a test driven approach to server management, and forces you to think about the setup tasks in a nested dependency tree of ‘deps’ that each do one thing well. It a fresh opinionated approach to CM that was a joy to use.

Chef

Chef is a powerful, fully featured CM tool. Unfortunately using Chef to setup one machine, requires using 3, a chef-master, chef knife and the Hubot machine itself. It is possible to use a chef-solo machine to avoid this (but that requires learning yet another way to set things up). A Ruby DSL is used to define cookbooks and recipes. Templating is available with Ruby and Erb. Certificate signing is required before machines can talk to each other.

Puppet

Puppet uses manifests and Erb templates to configure machines with a Ruby based DSL. A library of puppet modules provide the nessecary commands for manifest files. Puppet manifests are deployed via a puppet master server and again, certificate signing is required before machines can talk to each other.

Salt

Salt (like Ansible) uses a simple yml syntax to define ‘sls’ files. Again templates are written with Jinja2 (a simple Python templating language). Salt applies configuration to ‘minions’ via SSH and a salt master server.

Summary

All these tools have their own strengths, weaknesses and hurdles to overcome. I found both Ansible and Babushka the simplest to get started with, followed by Salt, Puppet and finally Chef. Setting up Hubot gave a great insight into each tool and I’ve chosen to move forward with both Puppet and Babushka in future projects. Babushka’s approach felt like a breath of fresh air and was in tune with a test-driven approach that has real added value. Puppet has a more established community and module library. It is a tremendously powerful tool and I feel like I’ve only scratched the surface.

Please throw any questions in the comments below (or via GitHub issues). Wow, I think I managed to get all the way through that without mentioning the word “Cloud”!

With some downtime over Christmas, I decided to give ActsAsTextcaptcha some love and release v4.0. For those not familiar with the gem;

ActsAsTextcaptcha provides spam protection for your Rails models using logic questions from the excellent TextCAPTCHA web service.

This was a much needed update and necessary to provide a fix for an urgent issue – kindly identified and reported by Jeffrey Lin (thanks again!)

A bot (using something like Mechanize) could by-pass spam protection using a fake question and answer. This was down to the gem exposing the possible answers in a hidden form field. Encryption of the answers only really helped prevent against attacks from the dumbest of spam bots. With a valid question and possible answer combo, POST requests could be constructed to appear valid and submit spam.

Using the built-in Rails session to hold possible answers between requests was a no-go too. Let’s say your app used the default ActionDispatch::Session::CookieStore. An attacker could grab a valid cookie that contained some possible answers to a known question and send that same cookie data (repeatedly) with POST requests containing spam. CSRF won’t help here either (with Mechanize) a valid token can be requested before each POST request.

So it was obvious the possible answers needed to be stored server side, and any reference to associate form requests with the answers needed to be random and dynamic (much like Rails own form_authenticity_token). I settled on using the Rails.cache, it’s available by default and easy to configure. The cache key mutates with each request and cache values only live for 10 minutes.

With this new release I took some time to refactor things a little, update the README and improve test coverage. As a nice side effect code-climate rates the gem with a 4.0 GPA.

Finally, I released v4.1 removing various hooks for Rails 2. It is 2014 now (over 4 years since Rails 3 was released) I think its time to move on.

A couple of months ago I embarked on a side project to add animated gif capturing to the popular Lolcommits gem. A couple of months later and I’ve refreshed my Objective-C skills, learnt more about building command line apps with Ruby and discovered the nuances of video capturing on a Mac.

And I’m happy to announce lolcommits v0.5 was released last week and now features animated gifs! (MacOSX only for now). For those not familiar with the gem;

lolcommits takes a snapshot with your webcam every time you git commit code, and archives a lolcat style image with it. Git blame has never been so much fun.

No longer should you git commit without expressing your current emotions in the form of some physical action! After squashing that bug feel free to get creative or just opt for the default self-five. Your animated masterpiece will be captured for eternity and can be published for all to see (to lolcommits.com or via another configurable lolcommit plugin)

Developing this feature I had hoped to use wacaw a command line video capturing tool for the Mac. Unfortunately its an old SourceForge project (that doesn’t seem to be maintained) and it fails to work with some of the latest Mac hardware. After a little effort I released VideoSnap an alternative using the latest QTKit framework.

Finally, thanks to Matthew Rothernberg I’m now an official maintainer for this gem! I’ve started contributing to PR’s, investigating issues and tidied up the wiki a little. I hope you enjoy this lolcommits release as much as I enjoyed developing it. Happy coding!

I’ve just released my first attempt at an Objective-C project, VideoSnap. It’s a simple OSX command line tool for recording video and audio from any attached QuickTime capture device.

VideoSnap was inspired by ImageSnap from @rharder, and I started working on it after finding problems with the older (carbon based) wacaw command (which no longer works with some of the latest Mac/OSX hardware). I hope to use VideoSnap in a new version of lolcommits, which i’m working on.

With VideoSnap you can specify which device to capture from, the duration, size/quality, a delay period (before capturing starts) and optionally turn off audio. The only required argument is a file path. By default VideoSnap will capture 6 seconds of video and audio from the default capture device in H.264(SD480)/AAC encoding to ‘movie.mov’. You can also use VideoSnap to list attached QuickTime capture devices by name.

VideoSnap is coded with Objective-C and uses the Cocoa and QTKit frameworks. It has been an interesting project so far and a great introduction to both Objective-C and Xcode. If you happen to give it a try, please let me know of any issues.

DRY isn’t only a good practice in software development. Developers should always be looking for better ways to not repeat themselves. Everything we do more than once could (and probably should) be a candidate for optimisation. From simple repetitive tasks like, connecting to remote servers to setting up a fresh development environment (and everything in between).

At HouseTrip we have amassed a collection of shortcuts, scripts, and rake tasks to make our lives that little bit easier. In an effort share them throughout the team and make some of them more well-known, I decided to create an all new ht command.

Helpers and configuration

Sub has some great conventions, but to allow us to direct commands at any one of our (many) staging and production servers, I forked it and added some helpers and configuration options of our own. Our staging servers can be temporary things, so its important we can easily change where we want to direct and auto-complete commands to.

Our server connection information lies in a simple yml file for each Rails environment we have. So we can issue a single command like this;

ht console staging100 # or ht c staging100

and have a Rails console (running on the remote staging100 machine) in no time. In the future I’ll try to create a pull-request with these helpers (to sub).

These helpers also include methods for handling and logging command output (even speaking output with the built-in OSX ‘say’ command). So far the ht command has been a hit, a trusty friend (with a voice) to call on for saving time.

Command Ideas

To give you some idea of what I’ve implemented so far (or plan to soon);

• `ht-ssh (env)` – connect to any server
• `ht-console (env)` – ssh and ans start a Rails console on a remote server
• `ht-dump (db-name)` – grab a fresh (anonymised) database dump and import it to your development env
• `ht-cache-clear (env)` – clear the Rails.cache on a remote server
• `ht-be-admin (env) (username)` – convert an existing remote user to an admin
• `ht-jobs (env)` – get some basic stats on job queues
• `ht-bump-job (env) (id)` – bump the priority of a remote job
• `ht-booking (env)` – show stats on the last few live bookings
• `ht-gif (keyword)` – fetch an animated gif into your paste buffer (with giphy.com)
• `ht-git-visual (time-ago) (repo)` – visualise git repository activity (with Gource)
• `ht-add-server (config)` – add new server details to your local ht config file
• `ht-mugshot (keyword)` – search and grab a photo from our team intranet page
• `ht-laptop` – kick off a setup script that installs and configures your laptop for HouseTrip development
• `ht-hammer` – replay a realistic log file of traffic requests to a staging server (with httperf)

Our ht command isn’t open source just yet, but it will be soon!

A few months ago, I was lucky enough to attend the very excellent La Conf with a few colleagues from HouseTrip. More on the actual conference in another post…

At some point during our trip I cooked up an idea for the popular lolcommits gem.

lolcommits takes a snapshot with your webcam every time you git commit code, and archives a lolcat style image with it. Git blame has never been so much fun.

My crazy plan was to try and make this even more fun, capturing video and converting that to an animated lolcommit gif!

Scope it!

So my change would add one new command line option. Calling lolcommits with `—animate=3`, a 3 second video would be captured, converted and annotated with your commit message. Without `—animate`, or `—animate=0` lolcommits would operate as normal, snapping a photo.

I could have added more options; video width, height, frame rate, optimisation params etc. But I wanted to keep the implementation (and my pull-request) simple, so I chose sensible defaults for these values and moved on.

Lolcommits has a framework for plugins where additional actions can be made on captured images (share with twitter, upload to a server etc.) I wanted this feature to stay compatible with plugins, and since I was dealing with a change to the initial capturing process, it didn’t make sense to to code this as a “post capture” plugin.

Also since my time was limited, I would only be targeting Mac/OSX for now (I could always add support for Linux or Windows in a future PR).

Code it!

After forking and cloning the repository, I found `bin/lolcommits` the best place to start (the main executable). Written in Ruby this code is well structured and easy to follow. Choice is used to handle command line arguments, so its pretty clear how each option executes.

After option handling in `bin/lolcommits` execution moves to the runner where a `Capture` class is instantiated and `capture` is called.

The gem organises capturing into `Capture` classes for each platform it supports. Taking the same approach I chose to implement my own, `CaptureMacAnimated`.

I followed the gem’s convention for running and logging system calls (with Methadone::CLILogging) and then (on the command line) I started experimenting with different ways to create an animated gif. The approach I eventually settled on was this;

• Created a new command line app to capture video, VideoSnap (uses the QTKit framework)
• Capture video in 320×240 AVI format with no audio
• Use ffmpeg (already an optional requirement for this gem) to build a folder full of png frames from this video
• Use ImageMagick to build an animated gif from these frames (choosing every 3rd frame) and optimising the gif, to keep file size low
• Pass the generated animated gif onto lolcommits in the `snapshot_location` to be annotated with the commit message

Developing this feature I had hoped to use wacaw a command line video capturing tool for the Mac. Unfortunately its an old SourceForge project (that doesn’t seem to be maintained) and it fails to work with some of the latest Mac hardware. After a little effort I released VideoSnap an alternative using the latest QTKit framework. It accepts a variety of options and is built in the Unix style of doing one thing well. It even has a `-d` option that maps nicely to an existing lolcommits option `—device`.

You can see this all implemented here, lib/lolcommits/capture_mac_animated.rb

Some more notes

• I use the seq command to pass every 3rd frame to convert
• Different browsers (and OSX preview) will play animated gifs at different rates. Choosing every 3rd frame and a delay of `9` gave a reasonable playback speed on the browsers I chose to test with
• Lolcommits already had a precedent for bundling executables; which meant including the wacaw binary was trivial

Ship it!

After updating the README, adding some plumbing for option handling, I ran the tests and finally submitted the PR just a few days ago! Fingers crossed it will be reviewed or accepted soon!

Get your animated lolcommits right now!

mroth has been reviewing the plugin and with a few tweaks from me, it could be merged for a new gem release soon. Lolcommits v0.5.0 has now been released! Simply follow the README to enable and configure animated gif capturing.

Finally, use the fork!

Depending on your machine, generating animated gifs is much slower than snapping a photo. And since this gem hooks into all post-commits, this change could quickly result in some angry impatient developers. To help with this I’ve submitted another PR that adds a `—fork` option.

With `—fork` you get your prompt back immediately after git commit, and image capturing/processing happens in a background thread.

If you made it this far, I hope you enjoyed this walk through. If you start using animated lolcommits in your repositories let me know!

This is a step by step guide on how to setup your local development environment to serve a Rails (or any Rack) app with Pow and NGINX over HTTPS.

To begin i’m going to assume you’re using OSX (probably Mountain Lion), HomeBrew and rbenv. For other setups ymmv.

Why use Pow?

• Easily host multiple Rack apps on your local machine under different domains e.g http://your-app.dev
• Configure local apps to run under SSL (explained below)
• Use the xip.io domain to visit your app from other devices on your local network
• Serve requests with multiple Pow workers
• Easy to configure, customise and works with multiple Rubies (via rbenv or RVM) and Bundler

Installing Pow

Install Pow with this command;

curl get.pow.cx | sh

Next create a symlink in ~/.pow to your app’s base directory like so;

ln -s /full/path/to/your-app ~/.pow/your-app

Zsh Rbenv users

If you are running zsh with rbenv you may need to follow these instructions and add a PATH export to your ~/.powconfig file like so;

export PATH=`brew --prefix rbenv`/shims:`brew --prefix rbenv`/bin:$PATH

Then restart the pow process with;

touch ~/.pow/restart.txt

This should be enough for you to see your app at http://your-app.dev. The next steps assume you have this working.

Installing & configuring NGINX

Install NGINX via brew;

brew install nginx

By default brew will install and configure NGINX to listen on port 8080. We need to run it on port 443 (decrypting SSL and proxy-ing all requests through to our Pow server).

Using this config file we can set up NGINX with some good defaults, and tell it to look for sites in `/usr/local/etc/nginx/sites-enabled`.

mkdir -p /usr/local/etc/nginx/sites-enabled
mkdir -p /usr/local/etc/nginx/sites-available

curl -0 https://gist.github.com/matthutchinson/5815393/raw/9845b99433a0e1ebd2763b264643fe308ea74b4f/nginx.conf > /usr/local/etc/nginx/nginx.conf

Next we create our site configuration in `/usr/local/etc/nginx/sites-available`

curl -0 https://gist.github.com/matthutchinson/5822750/raw/4790d7030d55a955b3c3a90fe2669b81235b95d2/your-app.dev > /usr/local/etc/nginx/sites-available/your-app.dev

Edit this file, setting the root (public) directory and replacing `your-app.dev` throughout. Finally symlink it into sites-enabled;

ln -s /usr/local/etc/nginx/sites-available/your-app.dev /usr/local/etc/nginx/sites-enabled/your-app.dev

Generating an SSL Cert

You might have noticed that the config file you just edited referenced an SSL cert that we have not yet created.

In a tmp directory, let’s use this handy gist to generate it and move the cert files into place;

curl https://gist.github.com/matthutchinson/5815498/raw/9da28acd6bf0ce1666f39cc0351dd5eee764be8b/nginx_gen_cert.rb > /tmp/nginx_gen_cert.rb
ruby /tmp/nginx_gen_cert.rb your-app.dev
rm /tmp/nginx_gen_cert.rb

You should now have SSL cert files for your app properly configured and contained in `/usr/local/etc/nginx/ssl`.

Trying it out

Thats it! To start NGINX (since we are listing on port 443) you need to run it with sudo;

sudo nginx

Visit https://your-app.dev/ now to see your app served via HTTPS.

Controlling things

The web app can be restarted by running `touch tmp/restart.txt` in the base directory. And you can control NGINX from the command line with flags like this;

sudo nginx -s stop
sudo nginx -s reload

Debugging with pry-remote

Since your app is now running in Pow’s own worker processes, to operate a live debugger you will need to use something like pry-remote.

First add the pry and pry-remote gems to your Gemfile (and `bundle install`). Then to introduce a breakpoint use this in your code;

binding.remote_pry

Fire off a request and when it stalls, run this command from your app’s base directory;

bundle exec pry-remote

A connection to the running worker process is established and you should be presented with a regular pry prompt. You can read more about pry-remote and pry here.

Further steps

Your browser may complain about not trusting your new SSL cert — we can fix that!

Restart or open Safari and visit https://your-app.dev. Click ‘Show Certificate’ from the SSL warning dialog. Choose the ‘Trust’ drop-down and select ‘Always Trust’. This adds your newly generated cert to the OSX keychain.

Setting up more sites is easy, just add them with a similar NGINX site config, generate an SSL cert (using the helper script again) and symlink things into place.

You can play with Pow’s configuration (e.g timeouts, workers) by defining ENV variables in ~/.powconfig, for example;

export POW_DOMAINS=dev,test
export POW_DST_PORT=80
export POW_TIMEOUT=300
export POW_WORKERS=3

Any change to ~/.powconfig needs a Pow restart;

touch ~/.pow/restart.txt

I hope this guide has been useful. Comments or questions are always welcome. (Pow artwork by Jamie Dihiansan)

It’s been a while. Before we catch up, let me sing you the song of my contributions.

Update – wow, i’ve just realised that this embedded iframe will play the tune on every visit to this page (a bit annoying). So I’ve created a quick PR for the project to add a “▸ Click to play” button. If (or when) it gets merged, I’ll throw the embedded music box back in here.

Update – the power of open source, thanks to ajacksified for creating this and merging!

Thanks to Simon at StandByProductions for putting this all together.