articles tagged with osx

DRY isn’t only a good practice in software development. Developers should always be looking for better ways to not repeat themselves. Everything we do more than once could (and probably should) be a candidate for optimisation. From simple repetitive tasks like, connecting to remote servers to setting up a fresh development environment (and everything in between).

At HouseTrip we have amassed a collection of shortcuts, scripts, and rake tasks to make our lives that little bit easier. In an effort share them throughout the team and make some of them more well-known, I decided to create an all new ht command.

Helpers and configuration

Sub has some great conventions, but to allow us to direct commands at any one of our (many) staging and production servers, I forked it and added some helpers and configuration options of our own. Our staging servers can be temporary things, so its important we can easily change where we want to direct and auto-complete commands to.

Our server connection information lies in a simple yml file for each Rails environment we have. So we can issue a single command like this;

ht console staging100 # or ht c staging100

and have a Rails console (running on the remote staging100 machine) in no time. In the future I’ll try to create a pull-request with these helpers (to sub).

These helpers also include methods for handling and logging command output (even speaking output with the built-in OSX ‘say’ command). So far the ht command has been a hit, a trusty friend (with a voice) to call on for saving time.

Command Ideas

To give you some idea of what I’ve implemented so far (or plan to soon);

• `ht-ssh (env)` – connect to any server
• `ht-console (env)` – ssh and ans start a Rails console on a remote server
• `ht-dump (db-name)` – grab a fresh (anonymised) database dump and import it to your development env
• `ht-cache-clear (env)` – clear the Rails.cache on a remote server
• `ht-be-admin (env) (username)` – convert an existing remote user to an admin
• `ht-jobs (env)` – get some basic stats on job queues
• `ht-bump-job (env) (id)` – bump the priority of a remote job
• `ht-booking (env)` – show stats on the last few live bookings
• `ht-gif (keyword)` – fetch an animated gif into your paste buffer (with giphy.com)
• `ht-git-visual (time-ago) (repo)` – visualise git repository activity (with Gource)
• `ht-add-server (config)` – add new server details to your local ht config file
• `ht-mugshot (keyword)` – search and grab a photo from our team intranet page
• `ht-laptop` – kick off a setup script that installs and configures your laptop for HouseTrip development
• `ht-hammer` – replay a realistic log file of traffic requests to a staging server (with httperf)

Our ht command isn’t open source just yet, but it will be soon!

This is a step by step guide on how to setup your local development environment to serve a Rails (or any Rack) app with Pow and NGINX over HTTPS.

To begin i’m going to assume you’re using OSX (probably Mountain Lion), HomeBrew and rbenv. For other setups ymmv.

Why use Pow?

• Easily host multiple Rack apps on your local machine under different domains e.g http://your-app.dev
• Configure local apps to run under SSL (explained below)
• Use the xip.io domain to visit your app from other devices on your local network
• Serve requests with multiple Pow workers
• Easy to configure, customise and works with multiple Rubies (via rbenv or RVM) and Bundler

Installing Pow

Install Pow with this command;

curl get.pow.cx | sh

Next create a symlink in ~/.pow to your app’s base directory like so;

ln -s /full/path/to/your-app ~/.pow/your-app

Zsh Rbenv users

If you are running zsh with rbenv you may need to follow these instructions and add a PATH export to your ~/.powconfig file like so;

export PATH=`brew --prefix rbenv`/shims:`brew --prefix rbenv`/bin:$PATH

Then restart the pow process with;

touch ~/.pow/restart.txt

This should be enough for you to see your app at http://your-app.dev. The next steps assume you have this working.

Installing & configuring NGINX

Install NGINX via brew;

brew install nginx

By default brew will install and configure NGINX to listen on port 8080. We need to run it on port 443 (decrypting SSL and proxy-ing all requests through to our Pow server).

Using this config file we can set up NGINX with some good defaults, and tell it to look for sites in `/usr/local/etc/nginx/sites-enabled`.

mkdir -p /usr/local/etc/nginx/sites-enabled
mkdir -p /usr/local/etc/nginx/sites-available

curl -0 https://gist.github.com/matthutchinson/5815393/raw/9845b99433a0e1ebd2763b264643fe308ea74b4f/nginx.conf > /usr/local/etc/nginx/nginx.conf

Next we create our site configuration in `/usr/local/etc/nginx/sites-available`

curl -0 https://gist.github.com/matthutchinson/5822750/raw/4790d7030d55a955b3c3a90fe2669b81235b95d2/your-app.dev > /usr/local/etc/nginx/sites-available/your-app.dev

Edit this file, setting the root (public) directory and replacing `your-app.dev` throughout. Finally symlink it into sites-enabled;

ln -s /usr/local/etc/nginx/sites-available/your-app.dev /usr/local/etc/nginx/sites-enabled/your-app.dev

Generating an SSL Cert

You might have noticed that the config file you just edited referenced an SSL cert that we have not yet created.

In a tmp directory, let’s use this handy gist to generate it and move the cert files into place;

curl https://gist.github.com/matthutchinson/5815498/raw/9da28acd6bf0ce1666f39cc0351dd5eee764be8b/nginx_gen_cert.rb > /tmp/nginx_gen_cert.rb
ruby /tmp/nginx_gen_cert.rb your-app.dev
rm /tmp/nginx_gen_cert.rb

You should now have SSL cert files for your app properly configured and contained in `/usr/local/etc/nginx/ssl`.

Trying it out

Thats it! To start NGINX (since we are listing on port 443) you need to run it with sudo;

sudo nginx

Visit https://your-app.dev/ now to see your app served via HTTPS.

Controlling things

The web app can be restarted by running `touch tmp/restart.txt` in the base directory. And you can control NGINX from the command line with flags like this;

sudo nginx -s stop
sudo nginx -s reload

Debugging with pry-remote

Since your app is now running in Pow’s own worker processes, to operate a live debugger you will need to use something like pry-remote.

First add the pry and pry-remote gems to your Gemfile (and `bundle install`). Then to introduce a breakpoint use this in your code;

binding.remote_pry

Fire off a request and when it stalls, run this command from your app’s base directory;

bundle exec pry-remote

A connection to the running worker process is established and you should be presented with a regular pry prompt. You can read more about pry-remote and pry here.

Further steps

Your browser may complain about not trusting your new SSL cert — we can fix that!

Restart or open Safari and visit https://your-app.dev. Click ‘Show Certificate’ from the SSL warning dialog. Choose the ‘Trust’ drop-down and select ‘Always Trust’. This adds your newly generated cert to the OSX keychain.

Setting up more sites is easy, just add them with a similar NGINX site config, generate an SSL cert (using the helper script again) and symlink things into place.

You can play with Pow’s configuration (e.g timeouts, workers) by defining ENV variables in ~/.powconfig, for example;

export POW_DOMAINS=dev,test
export POW_DST_PORT=80
export POW_TIMEOUT=300
export POW_WORKERS=3

Any change to ~/.powconfig needs a Pow restart;

touch ~/.pow/restart.txt

I hope this guide has been useful. Comments or questions are always welcome. (Pow artwork by Jamie Dihiansan)

Occasionally you’ll find me in IRC using the old school Irssi client. Its a simple and powerful IRC client that most importantly, I can launch on any unix based server and leave running in a screen session. A little something like this;

screen irssi
# Ctrl+A,D to unhook this screen
# screen -x to jump back into it (when you've only 1 screen open)

Doing this on a remote server, means I’m always connected to IRC. I can easily jump back in to previous conversations by ssh’ing to the server and switching to the screen.

What I wanted was a growl notification every time my name (nick) was mentioned in a room, or I received a private message. I stumbled on a few blog posts explaining how to do this, but came across a few gotchas on OSX, so here goes another explanation.

fnotify

First, on the remote server (where irssi will run under screen), add the fnotify script to your own ~/.irssi/scripts folder. This is a handy perl script that will write any messages referencing your nick, to a local file (by default ~/.irssi/fnotify). fnotify is a good, simple example of irssi perl scripting.

Don’t forget (as I did) to load this script in irssi with the following command (or setup irssi to auto load it on startup, some instructions here)

/script load fnotify.pl

Check this is working by having someone mention your name in a chat room (or pm you). You should see the ~/.irssi/fnotify file fill with these messages.

Growl

Next on OSX, install growl (if you haven’t already) and be sure to install the growlnotify script (make it executable and in your PATH e.g. /usr/local/bin) Test its working in your OSX terminal with something like;

growlnotify -m 'it works!'

SSH & connection script

Now if you haven’t already got one, create an SSH user with a key to login to your remote server. The key can have a pass-phrase. If you do choose to set one, for the following script to run without interruption, you’ll have to let OSX remember the pass-phrase in its key-chain.

Finally the client script, in OSX create the following script in ~/irssi_growler

#!/bin/sh

(ssh ssh_username@your_server.com -o PermitLocalCommand=no  \
  ": > .irssi/fnotify ; tail -f .irssi/fnotify " |  \
while read heading message; do                      \
  growlnotify -s -t "${heading}" -m "${message}";      \
  say "${heading} says, ${message}";                \
done)&

Replace your ssh_username and your_server.com details, make it executable (644) and run it. This script does the following;

• opens an ssh connection to your remote machine
• runs the tail -f command to remotely tail the ~/.irssi/fnotify command
• the output of which is piped to a read while loop
• in this loop each message is sent to growlnotify, and (for an audio bonus) to the say command too :)
• the growl message is set to be sticky (-s) and will stay onscreen until you click to hide it

Launch at startup

After numerous attempts (on Snow Leopard) trying to get this script automatically launching at startup (using launchctl, .plist’s, automator etc. etc.) I opted to just add the script to my login items. This does leave a Terminal window open every-time I login, but I can live with that for now. If someone can explain how properly add a bash script to launch on startup (as a daemon) on the latest Snow Leopard, please let me know.

The final caveat I hit was my firewall. It was smart enough to close any open SSH session after a period of inactivity. This kept killing the irssi_growler script. To prevent this I changed my ssh client config (in /etc/ssh_config) with the following ‘keep alive’ settings;

ServerAliveInterval 150 # can be adjusted higher or lower
ServerAliveCountMax 3

In the future I might re-write fnotify to do more that just write to file. It could potentially send the message data via tcp/ip to the a growl client (set to accept incoming connections).

I have jumped on board the RVM bandwagon for managing my development environments. I recently sold my Mac Pro on eBay and have a fresh new 27" iMac on the way. So I thought I’d document this part of the install process and a couple of gotcha’s.

First, grab the Postgres one-click installer and run it. Or you can build & compile it from source. Then for RVM and the latest REE follow these instructions;

# install readline for OSX if needed - check if it already exists in /usr/local/lib
wget ftp://ftp.cwru.edu/pub/bash/readline-6.1.tar.gz
tar -xvzf readline-6.1.tar.gz
cd readline-6.1
./configure && make && sudo make install

# install RVM from github
bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )
 
# edit your .bash_login (or profile) and add this at the bottom
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

# check RVM has installed OK, look for 'is a function'
. ~/.bash_login
type rvm | head -n1
rvm notes

# install REE, or whatever Ruby you'd like
rvm install ree -C --enable-shared,--with-readline-dir=/usr/local

# show what rubies are installed
rvm list

# set REE as the default and current ruby
rvm --default ree

# check your working directories
which ruby && which gem

# In general I use Bundler for apps, but for .irbrc and some scripts I need these too
gem install wirble awesome_print hirb bundler mysql heroku

# and finally for the postgres (pg) gem
sudo env ARCHFLAGS='-arch i386' gem install pg

I’m surprised by how few steps it takes these days (to get Ruby & Rails up and running with a database), four years ago this was a different story altogether. With tools like RVM and Bundler, it couldn’t be simpler.

Some months ago I played around with code_swarm by Michael Ogawa – partly for fun and partly to see what all the fuss was about with the Processing framework (something I have yet to really investigate). Last week I came across Gource another source code visualisation tool this time using 3D rendering.

Software projects are displayed by Gource as an animated tree with the root directory of the project at its centre. Directories appear as branches with files as leaves. Developers can be seen working on the tree at the times they contributed to the project.

Some recent commits to the project fixed build issues on OSX. Despite these fixes I still had trouble compiling. So in summary here’s what I did to get it working. Note that I did resort to installing mac ports (something I’d rather NOT do) – after many attempts to download and manually compile the prerequisites (FTGL 2.1.3~rc5-2 kept giving me problems)

# get mac ports
http://www.macports.org/install.php

# get all the ports you need for gource
sudo port install pcre libsdl libsdl_image ftgl

# get and build Gource from github and follow the instructions in INSTALL
git clone git://github.com/acaudwell/Gource.git
cd Gource
autoreconf -f -i
./configure && make && sudo make install

# navigate to your project directory and type;
gource

# have a look at all the options
man gource

# here are the settings I used (to get a large user icon and change the speed/size
gource ./ -s 0.5 -b 000000 --user-image-dir ~/images/avatars/ --user-scale 2.0 -800x600

# video it in h264 using ffmpeg, first install ffmpeg via mac ports (with codecs)
sudo port install ffmpeg +gpl +lame +x264 +xvid

# pipe PPM images to ffmpeg to generate a h264 encoded movie
gource ./ -s 0.5 -b 000000 --user-image-dir ~/images/avatars/ --user-scale 2.0 -800x600 --output-ppm-stream - | ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -vpre hq -crf 28 -threads 0 bugle.mp4

# finally, if you want to add some audio to the video (from an mp3)
ffmpeg -i audio.mp3 -i bugle.mp4 -vcodec libx264 -vpre hq -crf 28 -threads 0 bugle-with-audio.mp4
(this output will be the length of the video/audio track, whichever is longer)

# phew! - now go here to see how to remove macports when you're done :)
http://trac.macports.org/wiki/FAQ#uninstall

Bugle did exist (for a time) as an open source project on github before I moved it to be privately hosted. So here is the result; Bugle’s git log from April ’09 to the present parsed through Gource, with just one committer (me).

In summary you can see real bursts of activity at the start followed by some long periods of inactivity, and coming toward the present date renewed development and work going on. Gource is more impressive when visualising big projects with multiple committers over long periods, like the history of Git itself for example.

Working on the rails day in and out now, I’ve found the following aliases to come in handy;

# General Commands
alias ls='ls -al'

# TextMate, mate all of current dir and crucial rails folders only
alias et='mate . &'
alias ett='mate app config lib db public test vendor/plugins &'

# RAILS,  (run these from your rails folder)

# rails scripts
alias ss='./script/server'
alias sc='./script/console'
alias sg='./script/generate'
alias sp='./script/plugin'
alias mr='mongrel_rails start'

# rails testing 
alias att='autotest'
alias tu='rake test:units'
alias tf='rake test:functionals'    

# tail logs
alias tl='tail -f ./log/development.log'
alias tt='tail -f ./log/test.log'      

# clean the logs
alias ctl='cp /dev/null ./log/test.log'
alias cdl='cp /dev/null ./log/development.log'

I should credit Peep Code for the idea. To use, (e.g. in OSX) place the above in a ~/.bash_aliases file and in ~/.bash_profile, load it in with this command;

if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases ; fi

Also, (and before I forget it myself) – here’s a quick session cleaner command to put in your cron, (for day old, mysql session clearage action); rather than build a rake task, or extra controller to clean them out.

#!/bin/bash

cd /u/apps/matthewhutchinson.net/current
echo "<== CRON TASK ==> clear day old sessions data on matthewhutchinson.net"
ruby script/runner -e production "ActiveRecord::Base.connection.delete(\"DELETE FROM sessions WHERE updated_at < NOW() - INTERVAL 1 DAY\")"

After deciding to take a break from freelance work for the next couple of months, Ive had some time to sort a few things out here on this blog (and in doing so, learnt a thing or to).

Very soon I’ll be moving this site to my shiny new VPS server at Rimuhost, now armed with root access – all manner of crazy things can happen. Ive spent some free time over the last couple of weeks doing the following;

• Tidied up all the CSS here
• Moved all my Rails development from Windows to OSX
• Setup a new VPS Debian server with Rimuhosting
• Upgraded to Typo 4.0 and migrated this site’s theme into it
• Switched my photos back to a Flickr Pro account from 23HQ
• Moved this site to using SVN / Capistrano for deployment
• Changed from using Apache FCGI, to a full Lighttpd, Pound and Mongrel stack.
• Survived the London heat

So be prepared for all sorts of nerd like ‘how-to’ posts – as I attempt to share the knowledge and gotchas I experienced during all this (including the London heat) None of the changes to this site have been rolled out yet, its still sitting on Dreamhost running under Apache/FCGI.

Why bother? – The whole process has been a bit of learning exercise for myself and it means I can now quickly develop, deploy and host rails applications, faster and more reliably – which is good -you see.