articles tagged with ruby

ActsAsTextcaptcha v4.0 released!

no comments yet, post one now

With some downtime over Christmas, I decided to give ActsAsTextcaptcha some love and release v4.0. For those not familiar with the gem;

ActsAsTextcaptcha provides spam protection for your Rails models using logic questions from the excellent TextCAPTCHA web service.

This was a much needed update and necessary to provide a fix for an urgent issue – kindly identified and reported by Jeffrey Lin (thanks again!)

A bot (using something like Mechanize) could by-pass spam protection using a fake question and answer. This was down to the gem exposing the possible answers in a hidden form field. Encryption of the answers only really helped prevent against attacks from the dumbest of spam bots. With a valid question and possible answer combo, POST requests could be constructed to appear valid and submit spam.

Using the built-in Rails session to hold possible answers between requests was a no-go too. Let’s say your app used the default ActionDispatch::Session::CookieStore. An attacker could grab a valid cookie that contained some possible answers to a known question and send that same cookie data (repeatedly) with POST requests containing spam. CSRF won’t help here either (with Mechanize) a valid token can be requested before each POST request.

So it was obvious the possible answers needed to be stored server side, and any reference to associate form requests with the answers needed to be random and dynamic (much like Rails own form_authenticity_token). I settled on using the Rails.cache, it’s available by default and easy to configure. The cache key mutates with each request and cache values only live for 10 minutes.

With this new release I took some time to refactor things a little, update the README and improve test coverage. As a nice side effect code-climate rates the gem with a 4.0 GPA.

Finally, I released v4.1 removing various hooks for Rails 2. It is 2014 now (over 4 years since Rails 3 was released) I think its time to move on.

Lolcommits v0.5 released!

no comments yet, post one now

Lolcommits Logo

A couple of months ago I embarked on a side project to add animated gif capturing to the popular Lolcommits gem. A couple of months later and I’ve refreshed my Objective-C skills, learnt more about building command line apps with Ruby and discovered the nuances of video capturing on a Mac.

And I’m happy to announce lolcommits v0.5 was released last week and now features animated gifs! (MacOSX only for now). For those not familiar with the gem;

lolcommits takes a snapshot with your webcam every time you git commit code, and archives a lolcat style image with it. Git blame has never been so much fun.

No longer should you git commit without expressing your current emotions in the form of some physical action! After squashing that bug feel free to get creative or just opt for the default self-five. Your animated masterpiece will be captured for eternity and can be published for all to see (to lolcommits.com or via another configurable lolcommit plugin)

Developing this feature I had hoped to use wacaw a command line video capturing tool for the Mac. Unfortunately its an old SourceForge project (that doesn’t seem to be maintained) and it fails to work with some of the latest Mac hardware. After a little effort I released VideoSnap an alternative using the latest QTKit framework.

Finally, thanks to Matthew Rothernberg I’m now an official maintainer for this gem! I’ve started contributing to PR’s, investigating issues and tidied up the wiki a little. I hope you enjoy this lolcommits release as much as I enjoyed developing it. Happy coding!

One command to rule them all

no comments yet, post one now

LOTR rings DRY isn’t only a good practice in software development. Developers should always be looking for better ways to not repeat themselves. Everything we do more than once could (and probably should) be a candidate for optimisation. From simple repetitive tasks like, connecting to remote servers to setting up a fresh development environment (and everything in between).

At HouseTrip we have amassed a collection of shortcuts, scripts, and rake tasks to make our lives that little bit easier. In an effort share them throughout the team and make some of them more well-known, I decided to create an all new ht command.

37signals Sub

The ht command uses sub, a command line framework from 37signals. It’s a great starting point for building commands like this; with autocompletion, help, bash/zsh support and aliases all built in.

Helpers and configuration

Sub has some great conventions, but to allow us to direct commands at any one of our (many) staging and production servers, I forked it and added some helpers and configuration options of our own. Our staging servers can be temporary things, so its important we can easily change where we want to direct and auto-complete commands to.

Our server connection information lies in a simple yml file for each Rails environment we have. So we can issue a single command like this;

ht console staging100 # or ht c staging100

and have a Rails console (running on the remote staging100 machine) in no time. In the future I’ll try to create a pull-request with these helpers (to sub).

These helpers also include methods for handling and logging command output (even speaking output with the built-in OSXsay’ command). So far the ht command has been a hit, a trusty friend (with a voice) to call on for saving time.

Command Ideas

To give you some idea of what I’ve implemented so far (or plan to soon);

  • `ht-ssh (env)` – connect to any server
  • `ht-console (env)` – ssh and ans start a Rails console on a remote server
  • `ht-dump (db-name)` – grab a fresh (anonymised) database dump and import it to your development env
  • `ht-cache-clear (env)` – clear the Rails.cache on a remote server
  • `ht-be-admin (env) (username)` – convert an existing remote user to an admin
  • `ht-jobs (env)` – get some basic stats on job queues
  • `ht-bump-job (env) (id)` – bump the priority of a remote job
  • `ht-booking (env)` – show stats on the last few live bookings
  • `ht-gif (keyword)` – fetch an animated gif into your paste buffer (with giphy.com)
  • `ht-git-visual (time-ago) (repo)` – visualise git repository activity (with Gource)
  • `ht-add-server (config)` – add new server details to your local ht config file
  • `ht-mugshot (keyword)` – search and grab a photo from our team intranet page
  • `ht-laptop` – kick off a setup script that installs and configures your laptop for HouseTrip development
  • `ht-hammer` – replay a realistic log file of traffic requests to a staging server (with httperf)

Our ht command isn’t open source just yet, but it will be soon!

Animated Lolcommits

no comments yet, post one now

A few months ago, I was lucky enough to attend the very excellent La Conf with a few colleagues from HouseTrip. More on the actual conference in another post…

At some point during our trip I cooked up an idea for the popular lolcommits gem.

lolcommits takes a snapshot with your webcam every time you git commit code, and archives a lolcat style image with it. Git blame has never been so much fun.

My crazy plan was to try and make this even more fun, capturing video and converting that to an animated lolcommit gif!

Scope it!

So my change would add one new command line option. Calling lolcommits with `—animate=3`, a 3 second video would be captured, converted and annotated with your commit message. Without `—animate`, or `—animate=0` lolcommits would operate as normal, snapping a photo.

I could have added more options; video width, height, frame rate, optimisation params etc. But I wanted to keep the implementation (and my pull-request) simple, so I chose sensible defaults for these values and moved on.

Lolcommits has a framework for plugins where additional actions can be made on captured images (share with twitter, upload to a server etc.) I wanted this feature to stay compatible with plugins, and since I was dealing with a change to the initial capturing process, it didn’t make sense to to code this as a “post capture” plugin.

Also since my time was limited, I would only be targeting Mac/OSX for now (I could always add support for Linux or Windows in a future PR).

Code it!

After forking and cloning the repository, I found `bin/lolcommits` the best place to start (the main executable). Written in Ruby this code is well structured and easy to follow. Choice is used to handle command line arguments, so its pretty clear how each option executes.

After option handling in `bin/lolcommits` execution moves to the runner where a `Capture` class is instantiated and `capture` is called.

The gem organises capturing into `Capture` classes for each platform it supports. Taking the same approach I chose to implement my own, `CaptureMacAnimated`.

I followed the gem’s convention for running and logging system calls (with Methadone::CLILogging) and then (on the command line) I started experimenting with different ways to create an animated gif. The approach I eventually settled on was this;

  • Created a new command line app to capture video, VideoSnap (uses the QTKit framework)
  • Capture video in 320×240 AVI format with no audio
  • Use ffmpeg (already an optional requirement for this gem) to build a folder full of png frames from this video
  • Use ImageMagick to build an animated gif from these frames (choosing every 3rd frame) and optimising the gif, to keep file size low
  • Pass the generated animated gif onto lolcommits in the `snapshot_location` to be annotated with the commit message

Developing this feature I had hoped to use wacaw a command line video capturing tool for the Mac. Unfortunately its an old SourceForge project (that doesn’t seem to be maintained) and it fails to work with some of the latest Mac hardware. After a little effort I released VideoSnap an alternative using the latest QTKit framework. It accepts a variety of options and is built in the Unix style of doing one thing well. It even has a `-d` option that maps nicely to an existing lolcommits option `—device`.

You can see this all implemented here, lib/lolcommits/capture_mac_animated.rb

Some more notes

  • I use the seq command to pass every 3rd frame to convert
  • Different browsers (and OSX preview) will play animated gifs at different rates. Choosing every 3rd frame and a delay of `9` gave a reasonable playback speed on the browsers I chose to test with
  • Lolcommits already had a precedent for bundling executables; which meant including the wacaw binary was trivial

Ship it!

After updating the README, adding some plumbing for option handling, I ran the tests and finally submitted the PR just a few days ago! Fingers crossed it will be reviewed or accepted soon!

Me, wishing on a PR

Get your animated lolcommits right now!

mroth has been reviewing the plugin and with a few tweaks from me, it could be merged for a new gem release soon. Lolcommits v0.5.0 has now been released! Simply follow the README to enable and configure animated gif capturing.

Finally, use the fork!

Depending on your machine, generating animated gifs is much slower than snapping a photo. And since this gem hooks into all post-commits, this change could quickly result in some angry impatient developers. To help with this I’ve submitted another PR that adds a `—fork` option.

With `—fork` you get your prompt back immediately after git commit, and image capturing/processing happens in a background thread.

If you made it this far, I hope you enjoyed this walk through. If you start using animated lolcommits in your repositories let me know!

Configuring Pow with NGINX and SSL on OSX

7 comments

pow.png This is a step by step guide on how to setup your local development environment to serve a Rails (or any Rack) app with Pow and NGINX over HTTPS.

To begin i’m going to assume you’re using OSX (probably Mountain Lion), HomeBrew and rbenv. For other setups ymmv.

What is Pow?

Pow (a 37signals project) runs as your user on an unprivileged port, and includes both an HTTP and a DNS server. The installation process sets up a firewall rule to forward incoming requests on port 80 to Pow. It also sets up a system hook so that all DNS queries for a special top-level domain (.dev) resolve to your local machine.

For more information on Pow, read the intro or browse the manual.

Why use Pow?

  • Easily host multiple Rack apps on your local machine under different domains e.g http://your-app.dev
  • Configure local apps to run under SSL (explained below)
  • Use the xip.io domain to visit your app from other devices on your local network
  • Serve requests with multiple Pow workers
  • Easy to configure, customise and works with multiple Rubies (via rbenv or RVM) and Bundler

Installing Pow

Install Pow with this command;

curl get.pow.cx | sh

Next create a symlink in ~/.pow to your app’s base directory like so;

ln -s /full/path/to/your-app ~/.pow/your-app

Zsh Rbenv users

If you are running zsh with rbenv you may need to follow these instructions and add a PATH export to your ~/.powconfig file like so;

export PATH=`brew --prefix rbenv`/shims:`brew --prefix rbenv`/bin:$PATH

Then restart the pow process with;

touch ~/.pow/restart.txt

This should be enough for you to see your app at http://your-app.dev. The next steps assume you have this working.

Installing & configuring NGINX

Install NGINX via brew;

brew install nginx

By default brew will install and configure NGINX to listen on port 8080. We need to run it on port 443 (decrypting SSL and proxy-ing all requests through to our Pow server).

Using this config file we can set up NGINX with some good defaults, and tell it to look for sites in `/usr/local/etc/nginx/sites-enabled`.

mkdir -p /usr/local/etc/nginx/sites-enabled
mkdir -p /usr/local/etc/nginx/sites-available

curl -0 https://gist.github.com/matthutchinson/5815393/raw/9845b99433a0e1ebd2763b264643fe308ea74b4f/nginx.conf > /usr/local/etc/nginx/nginx.conf

Next we create our site configuration in `/usr/local/etc/nginx/sites-available`

curl -0 https://gist.github.com/matthutchinson/5822750/raw/4790d7030d55a955b3c3a90fe2669b81235b95d2/your-app.dev > /usr/local/etc/nginx/sites-available/your-app.dev

Edit this file, setting the root (public) directory and replacing `your-app.dev` throughout. Finally symlink it into sites-enabled;

ln -s /usr/local/etc/nginx/sites-available/your-app.dev /usr/local/etc/nginx/sites-enabled/your-app.dev

Generating an SSL Cert

You might have noticed that the config file you just edited referenced an SSL cert that we have not yet created.

In a tmp directory, let’s use this handy gist to generate it and move the cert files into place;

curl https://gist.github.com/matthutchinson/5815498/raw/9da28acd6bf0ce1666f39cc0351dd5eee764be8b/nginx_gen_cert.rb > /tmp/nginx_gen_cert.rb
ruby /tmp/nginx_gen_cert.rb your-app.dev
rm /tmp/nginx_gen_cert.rb

You should now have SSL cert files for your app properly configured and contained in `/usr/local/etc/nginx/ssl`.

Trying it out

Thats it! To start NGINX (since we are listing on port 443) you need to run it with sudo;

sudo nginx

Visit https://your-app.dev/ now to see your app served via HTTPS.

Controlling things

The web app can be restarted by running `touch tmp/restart.txt` in the base directory. And you can control NGINX from the command line with flags like this;

sudo nginx -s stop
sudo nginx -s reload

Debugging with pry-remote

Since your app is now running in Pow’s own worker processes, to operate a live debugger you will need to use something like pry-remote.

First add the pry and pry-remote gems to your Gemfile (and `bundle install`). Then to introduce a breakpoint use this in your code;

binding.remote_pry

Fire off a request and when it stalls, run this command from your app’s base directory;

bundle exec pry-remote

A connection to the running worker process is established and you should be presented with a regular pry prompt. You can read more about pry-remote and pry here.

Further steps

Your browser may complain about not trusting your new SSL cert — we can fix that!

Restart or open Safari and visit https://your-app.dev. Click ‘Show Certificate’ from the SSL warning dialog. Choose the ‘Trust’ drop-down and select ‘Always Trust’. This adds your newly generated cert to the OSX keychain.

Setting up more sites is easy, just add them with a similar NGINX site config, generate an SSL cert (using the helper script again) and symlink things into place.

You can play with Pow’s configuration (e.g timeouts, workers) by defining ENV variables in ~/.powconfig, for example;

export POW_DOMAINS=dev,test
export POW_DST_PORT=80
export POW_TIMEOUT=300
export POW_WORKERS=3

Any change to ~/.powconfig needs a Pow restart;

touch ~/.pow/restart.txt

I hope this guide has been useful. Comments or questions are always welcome. (Pow artwork by Jamie Dihiansan)

June 20, 2013 17:02 by

Web APIs with Rack, Sinatra and MongoDB by Oisin Hurley

1 comment

Oisin Hurley gave a great talk on ‘Constructing Web APIs with Rack, Sinatra and MongoDB’ at Ruby Ireland last Tuesday. I took the opportunity to try recording the video/audio of his presentation and map it together with his slides on slideshare. While quality isn’t anything near HD, using a combination of iMovie and Screenflow I got there in the end.

Recurring billing talk at Ruby Ireland

no comments yet, post one now

Thanks to everyone who showed up last night for the monthly Ruby Ireland user group meet-up at the Twisted Pepper. Unfortunately we completely forgot to try recording the presentation, but for those interested the slides are available here.

There did seem to be a genuine interest in this topic so I’ve offered to write a blog post (or perhaps a series of posts) explaining the implementation in more detail and with actual code snippets. To summarize what I spoke about;

I gave an overview on the basics of taking payments online, specifically;

  • Banks
  • Payment Gateways
  • Libraries/code/web services available for your Rails app.

I explained each core method implemented in an ActiveMerchant billing gateway class, and I covered the important design decisions I took before building in recurring billing;

  • High level decisions (which bank/gateway etc.)
  • Pricing decisions (price plans, upgrade/downgrade paths/logic etc.)
  • Actual implementation decisions (how its coded etc.)

Finally I talked about some pain points you’ll likely to come across when working with recurring billing and online payments in general. I hope everyone enjoyed the talk and I’d be happy to answer any further questions if you have them.

Ps. Apologies for the VGA cable problems, thanks to Declan’s trusty netbook we got there in the end!

Yet another Rails security checklist

3 comments

I recently spent some time looking at application security on an old and new (Rails 3) app. Below is short (rather long) summary of my findings.

Quick caveat, this is by no means meant to be an exhaustive post on Rails security (for that I’d recommend you read the official Ruby On Rails Security Guide)

I’ve basically highlighted what I think are the bare minimum of checks you should be doing before throwing an application into the wild. And i’m focusing more on the actual preventative measures. Also I should point out that (unless your app is very small) security is not something you can retro-fit at the end of a project. To avoid holes you really need to be thinking about these things as you go.

Do not trust logged in users! (Authentication != Authorisation)

‘Change-affecting’ actions create/update/delete (and their corresponding views new/edit) need to be protected. So always check the current logged user is allowed to perform the requested action.

One way to do this, is to have the concept of a user owning a model and scoping find’s on these actions by the user. Or consider adding a before filter to the controller, that validates the logged in user against the action being performed on the object. For instance, create methods like can_edit?(by_user) or can_destroy?(by_user)

Remember too, that the most popular Rails plugins, Clearance, Devise and Restful Authentication only authenticate users, and provide no helpers or methods for authorisation. Also, I’d advise against building your own authentication system from scratch!

Mass assignment

A params hash can contain anything, so protect all sensitive attributes from re-assignment. The best way to do this is through disabling mass assignment by using attr_accessible (or attr_protected)in your models. Attributes like counter caches, account_id’s and user_id’s are prime examples for protection.

Un-editable attributes

Disable updates on protected attributes. Using attr_readonly in ActiveRecord allows the attribute to be set on create, but never edited on update. E.g. In many cases an account_id should be set once and never change.

SQL Injection

Don’t include user submitted strings in database queries! Check all model scopes and find conditions that include params or interpolated strings. E.g. LIKE searches should have the keyword passed in this form;

posts.where('posts.title LIKE ?', "%#{keyword}%")

Prevent executable files from being uploaded

Consider validating the content type on all attachments, and place uploaded files in protected directories or on another server/service e.g. S3/Cloudfront. Content-types can easily be faked, so check file extensions and be sure to disable your web server from executing scripts in the upload directory.

Also, beware of plugins creating or writing in tmp directories during file upload creation. They may create files or directories from user submitted params without checking the file path. In general be familiar with your gems and plugin code.

Filter sensitive params from logs

Use filter_parameter_logging or config.filter_parameters (Rails 3) to remove sensitive data from your logs (:password, :password_confirmation, :credit_card_number etc.)

CSRF

Enable protect_from_forgery and use form helpers to include the Rails authenticity token in all form submissions. In Rails 3 ensure the HTML head tag includes the csrf_meta_tag helper.

XSS

Prevent XSS and javascript injection by escaping all rendered user content (use the ‘h’ helper). In Rails 3 this happens by default so be careful what you whitelist for rendering (with raw, or html_safe!) If you are using something like Textile or Markdown, make use of the whitelist filtering methods their libraries provide.

Session Hijacks

Sessions are open to a number of vulnerabilities. Hijacking, replaying cookies, session fixation etc. Check the Ruby On Rails Security Guide for all the details. Here are some general tips;

If storing login information e.g. a remember_me cookie for authentication use CookieStore with a digest and reasonably complex server-side secret.

Be careful what data you decide to store and use in your sessions. Don’t store ActiveRecord objects, future code changes or migrations may change an objects behaviour/attributes and clients with old session data will be out of sync. Better to store only id’s for records in the session and load model objects server-side.

Other things to NOT store in your session include; money balances, user access privileges or sensitive state based information etc. These are open to session replay attacks.

Finally, to avoid session fixation attacks, call reset_session after a successful login.

Redirecting

Avoid using redirect_to(params[:some_param]). When the arguments for a redirect come from params, you are open to redirects to unintended URLs.

Downloading files, raw data

Avoid using params or user content in the send_file method. Consider having these as absolute filenames stored in the database, or validate the path to the file before calling send_file.

Make non-action controller methods private.

Remember that any methods you declare in a controller could be accessible to the public depending on your routes.

Gems and plugins

Remember to check your dependencies for security updates and patches. If possible subscribe to the GitHub issues list or (any mailing list) for the gems or plugins your using.

Passwords

Don’t store passwords in the database as clear text! Maybe that one goes without saying :) Encourage strong alphanumeric passwords and if necessary follow other strong password practices (throttle multiple failed logins, password expiry/reset etc.)

Also, you really should use BCrypt to discourage rainbow table attacks. Read this to find out more.

Further reading

Most of this checklist has been compiled from the following links.

October 21, 2010 22:10 by

Rails 3 ambiguous column names and multiple order scopes

3 comments

After updating a few apps to Rails 3 I came across some issues with the new scoping syntax.

Ambiguous column names

In one case I use a module to define some common scopes that are used across more than one model. I quickly came up against the ambiguous column name error (e.g. Column ‘created_at’ in order clause is ambiguous —from MySQL). To fix this I used the table_name prefix right in my scoping string. In general you should always do this anywhere you use strings in your scopes. For example;

# remember to add the table_name to avoid ambiguous columns
scope :recent, order("#{table_name}.created_at DESC")
# or directly
scope :recent, order("questions.created_at DESC")

# you don't have to add the table_name when using a hash, Rails takes care of that!
scope :by_user, lambda{|user| where(:user_id => user.id).recent}

Multiple order scopes

Another issue I found was combining multiple order scopes. In Rails 2 with named scopes this used to work as you’d expect. But combining multiple order scopes in Rails 3 seems to fail, only applying the last order scope to the query.

# given the recent scope defined above, in this case we'd expect;
scope :flagged, order(" #{table_name}.flag_count DESC").recent
# to be 
'ORDER BY questions.flag_count DESC, questions.created_at DESC'
# BUT its not! instead its simply
'ORDER BY questions.created_at DESC'
# to order on multiple columns inside a scope, I have to duplicate my recent order scope like so
scope :flagged, order(" #{table_name}.flag_count DESC, #{table_name}.created_at DESC").recent

All in all, i’m very much in favour of the new syntax and AREL in general. It simplifies compounding queries together with additional scopes and delays their execution until it is needed.

Some Rails 3 ActiveRecord links

September 22, 2010 13:46 by

FAQtory update

no comments yet, post one now

Its been a while since I mentioned anything about the FAQtory app. Rest assured that, along with Bugle it’s been keeping me busy in my spare time!

To re-cap quickly, the FAQtory (pronounced Factory) is a simplified stackoverflow-like question and answer app. It allows you to Ask and Answer questions on any topic you like. The app is entirely content-focused and deliberately simple. A simple user voting mechanism ensures the most popular questions and most correct answers filter to the top.

Looking at the logs, it was last August when I kicked off the very first commit. After almost a year of on-again, off-again work and long periods of inactivity, last week finally saw it launch! In its current form the FAQtory is deployed for a single account, (pmfaqtory.com) – you can take a look here. Over time PMfaqtory intend to use FAQtory to build a resource of project management questions and answers.

This is a big milestone for me and with all features locked down for launch, I can concentrate on preparing for a general release. This will involve adding an account model, pricing options and payment gateway integration. I’ll also be moving the app to a new server stack and cloud based host.

Over the coming weeks I hope to talk more about some the design and technology decisions I have made so far. For a bit of fun, here is a replay of the Git log using the latest Gource visualisation engine (now with added ‘Bloom’ effect!)

music is 'Penguin' by the Books

Gource is now available on Mac Ports which is a much easier install than it used to be (again, compiling from source stills give errors on FTGL)

← (k) prev | next (j) →