Oisin Hurley gave a great talk on ‘Constructing Web APIs with Rack, Sinatra and MongoDB’ at Ruby Ireland last Tuesday. I took the opportunity to try recording the video/audio of his presentation and map it together with his slides on slideshare. While quality isn’t anything near HD, using a combination of iMovie and Screenflow I got there in the end.
Thanks to everyone who showed up last night for the monthly Ruby Ireland user group meet-up at the Twisted Pepper. Unfortunately we completely forgot to try recording the presentation, but for those interested the slides are available here.
There did seem to be a genuine interest in this topic so I’ve offered to write a blog post (or perhaps a series of posts) explaining the implementation in more detail and with actual code snippets. To summarize what I spoke about;
I gave an overview on the basics of taking payments online, specifically;
I explained each core method implemented in an ActiveMerchant billing gateway class, and I covered the important design decisions I took before building in recurring billing;
Finally I talked about some pain points you’ll likely to come across when working with recurring billing and online payments in general. I hope everyone enjoyed the talk and I’d be happy to answer any further questions if you have them.
Ps. Apologies for the VGA cable problems, thanks to Declan’s trusty netbook we got there in the end!
I recently spent some time looking at application security on an old and new (Rails 3) app. Below is short (rather long) summary of my findings.
Quick caveat, this is by no means meant to be an exhaustive post on Rails security (for that I’d recommend you read the official Ruby On Rails Security Guide)
I’ve basically highlighted what I think are the bare minimum of checks you should be doing before throwing an application into the wild. And i’m focusing more on the actual preventative measures. Also I should point out that (unless your app is very small) security is not something you can retro-fit at the end of a project. To avoid holes you really need to be thinking about these things as you go.
‘Change-affecting’ actions create/update/delete (and their corresponding views new/edit) need to be protected. So always check the current logged user is allowed to perform the requested action.
One way to do this, is to have the concept of a user owning a model and scoping find’s on these actions by the user. Or consider adding a before filter to the controller, that validates the logged in user against the action being performed on the object. For instance, create methods like can_edit?(by_user) or can_destroy?(by_user)
Remember too, that the most popular Rails plugins, Clearance, Devise and Restful Authentication only authenticate users, and provide no helpers or methods for authorisation. Also, I’d advise against building your own authentication system from scratch!
A params hash can contain anything, so protect all sensitive attributes from re-assignment. The best way to do this is through disabling mass assignment by using attr_accessible (or attr_protected)in your models. Attributes like counter caches, account_id’s and user_id’s are prime examples for protection.
Disable updates on protected attributes. Using attr_readonly in ActiveRecord allows the attribute to be set on create, but never edited on update. E.g. In many cases an account_id should be set once and never change.
Don’t include user submitted strings in database queries! Check all model scopes and find conditions that include params or interpolated strings. E.g. LIKE searches should have the keyword passed in this form;
posts.where('posts.title LIKE ?', "%#{keyword}%")Consider validating the content type on all attachments, and place uploaded files in protected directories or on another server/service e.g. S3/Cloudfront. Content-types can easily be faked, so check file extensions and be sure to disable your web server from executing scripts in the upload directory.
Also, beware of plugins creating or writing in tmp directories during file upload creation. They may create files or directories from user submitted params without checking the file path. In general be familiar with your gems and plugin code.
Use filter_parameter_logging or config.filter_parameters (Rails 3) to remove sensitive data from your logs (:password, :password_confirmation, :credit_card_number etc.)
Enable protect_from_forgery and use form helpers to include the Rails authenticity token in all form submissions. In Rails 3 ensure the HTML head tag includes the csrf_meta_tag helper.
Prevent XSS and javascript injection by escaping all rendered user content (use the ‘h’ helper). In Rails 3 this happens by default so be careful what you whitelist for rendering (with raw, or html_safe!) If you are using something like Textile or Markdown, make use of the whitelist filtering methods their libraries provide.
Sessions are open to a number of vulnerabilities. Hijacking, replaying cookies, session fixation etc. Check the Ruby On Rails Security Guide for all the details. Here are some general tips;
If storing login information e.g. a remember_me cookie for authentication use CookieStore with a digest and reasonably complex server-side secret.
Be careful what data you decide to store and use in your sessions. Don’t store ActiveRecord objects, future code changes or migrations may change an objects behaviour/attributes and clients with old session data will be out of sync. Better to store only id’s for records in the session and load model objects server-side.
Other things to NOT store in your session include; money balances, user access privileges or sensitive state based information etc. These are open to session replay attacks.
Finally, to avoid session fixation attacks, call reset_session after a successful login.
Avoid using redirect_to(params[:some_param]). When the arguments for a redirect come from params, you are open to redirects to unintended URLs.
Avoid using params or user content in the send_file method. Consider having these as absolute filenames stored in the database, or validate the path to the file before calling send_file.
Remember that any methods you declare in a controller could be accessible to the public depending on your routes.
Remember to check your dependencies for security updates and patches. If possible subscribe to the GitHub issues list or (any mailing list) for the gems or plugins your using.
Don’t store passwords in the database as clear text! Maybe that one goes without saying :) Encourage strong alphanumeric passwords and if necessary follow other strong password practices (throttle multiple failed logins, password expiry/reset etc.)
Also, you really should use BCrypt to discourage rainbow table attacks. Read this to find out more.
Most of this checklist has been compiled from the following links.
After updating a few apps to Rails 3 I came across some issues with the new scoping syntax.
In one case I use a module to define some common scopes that are used across more than one model. I quickly came up against the ambiguous column name error (e.g. Column ‘created_at’ in order clause is ambiguous —from MySQL). To fix this I used the table_name prefix right in my scoping string. In general you should always do this anywhere you use strings in your scopes. For example;
# remember to add the table_name to avoid ambiguous columns
scope :recent, order("#{table_name}.created_at DESC")
# or directly
scope :recent, order("questions.created_at DESC")
# you don't have to add the table_name when using a hash, Rails takes care of that!
scope :by_user, lambda{|user| where(:user_id => user.id).recent}
Another issue I found was combining multiple order scopes. In Rails 2 with named scopes this used to work as you’d expect. But combining multiple order scopes in Rails 3 seems to fail, only applying the last order scope to the query.
# given the recent scope defined above, in this case we'd expect;
scope :flagged, order(" #{table_name}.flag_count DESC").recent
# to be
'ORDER BY questions.flag_count DESC, questions.created_at DESC'
# BUT its not! instead its simply
'ORDER BY questions.created_at DESC'
# to order on multiple columns inside a scope, I have to duplicate my recent order scope like so
scope :flagged, order(" #{table_name}.flag_count DESC, #{table_name}.created_at DESC").recent
All in all, i’m very much in favour of the new syntax and AREL in general. It simplifies compounding queries together with additional scopes and delays their execution until it is needed.
Its been a while since I mentioned anything about the FAQtory app. Rest assured that, along with Bugle it’s been keeping me busy in my spare time!
To re-cap quickly, the FAQtory (pronounced Factory) is a simplified stackoverflow-like question and answer app. It allows you to Ask and Answer questions on any topic you like. The app is entirely content-focused and deliberately simple. A simple user voting mechanism ensures the most popular questions and most correct answers filter to the top.
Looking at the logs, it was last August when I kicked off the very first commit. After almost a year of on-again, off-again work and long periods of inactivity, last week finally saw it launch! In its current form the FAQtory is deployed for a single account, (pmfaqtory.com) – you can take a look here. Over time PMfaqtory intend to use FAQtory to build a resource of project management questions and answers.
This is a big milestone for me and with all features locked down for launch, I can concentrate on preparing for a general release. This will involve adding an account model, pricing options and payment gateway integration. I’ll also be moving the app to a new server stack and cloud based host.
Over the coming weeks I hope to talk more about some the design and technology decisions I have made so far. For a bit of fun, here is a replay of the Git log using the latest Gource visualisation engine (now with added ‘Bloom’ effect!)
music is 'Penguin' by the Books
Gource is now available on Mac Ports which is a much easier install than it used to be (again, compiling from source stills give errors on FTGL)
I have jumped on board the RVM bandwagon for managing my development environments. I recently sold my Mac Pro on eBay and have a fresh new 27" iMac on the way. So I thought I’d document this part of the install process and a couple of gotcha’s.
First, grab the Postgres one-click installer and run it. Or you can build & compile it from source. Then for RVM and the latest REE follow these instructions;
# install readline for OSX if needed - check if it already exists in /usr/local/lib wget ftp://ftp.cwru.edu/pub/bash/readline-6.1.tar.gz tar -xvzf readline-6.1.tar.gz cd readline-6.1 ./configure && make && sudo make install # install RVM from github bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head ) # edit your .bash_login (or profile) and add this at the bottom [[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # check RVM has installed OK, look for 'is a function' . ~/.bash_login type rvm | head -n1 rvm notes # install REE, or whatever Ruby you'd like rvm install ree -C --enable-shared,--with-readline-dir=/usr/local # show what rubies are installed rvm list # set REE as the default and current ruby rvm --default ree # check your working directories which ruby && which gem # In general I use Bundler for apps, but for .irbrc and some scripts I need these too gem install wirble awesome_print hirb bundler mysql heroku # and finally for the postgres (pg) gem sudo env ARCHFLAGS='-arch i386' gem install pg
I’m surprised by how few steps it takes these days (to get Ruby & Rails up and running with a database), four years ago this was a different story altogether. With tools like RVM and Bundler, it couldn’t be simpler.
Here’s a few links on Rails 3 I’ve collected in preparation for its release. I haven’t read these all through yet, or started using Rails 3 in any way, but I have a couple of production apps primed and waiting to be upgraded as soon as it launches.
Start with this one from RubyInside, 36 links to get you going
And finally some official release notes;
I’d like to share another handy little rake task that helps you fill up your Rails application with realistic ‘fake’ data. Here it is
Using the very excellent Faker gem by Benjamin Curtis this task can be configured to populate your models with random amounts of fake information. This can a be a real time-saver for load testing, preparing demos/screencasts, or just filling up your pages with realistic data so you can get to work on your views.
The classic 'comb-over', a traditional fake.
The task includes the following features;
Simply configure it (see code comments for help on this), drop it into your Rails lib/tasks folder and run like so;
sudo gem install faker # if you haven't got the gem already OR
sudo gem install ffaker # use the faster faker gem from Emmanuel Oga
rake fakeout:small # or tiny / medium / large
Cleaning all ...
Faking it ... (small)
* Users: 53
* Questions: 53
* Answers: 38
* Tags: 38
* Taggings: 160
Done, I Faked it!
Along with this subdomain script from last month, I have pushed this code to github as a gist, so you can track it and grab updates to it whenever. The example there shows the task configured for a small Question & Answer app I’ve been working on.
And yes, I have started drawing a bit again, maybe it’ll make these posts more interesting to look at until I properly build this blog out.
Update – I have amended the task definitions to take a single argument; no_prompt. This turns off the confirm prompt and is useful for non-interactive running, e.g. with heroku rake commands for instance.
Update – This rake task is also fully compatible with the faster ffaker gem by Emmanuel Oga
A time saving rake task for adding or updating your local /etc/hosts file. I created this for Bugle, allowing me to quickly configure my development machine hosts file with subdomains used in the app. See the inline comments for an explanation.
Has the added feature of an array of default hosts to always add when you run it. Work is done on a tmp file then, then sudo copied on top of /etc/hosts. It exists as a gist on github so i’ll be sure to post any updates to it there.
# subdomains.rake (in /lib/tasks)
namespace :subdomains do
desc "adds the necessary hosts to your /etc/hosts file from current subdomains in your application"
task :setup => :environment do
# NOTE: default_hosts is used as a locator for the line to update in /etc/hosts
tmp_file, changed = '/tmp/etc_hosts_copy', false
default_hosts, hosts = %w(blog.local emptyblog.blog.local), []
# find all the subdomains used in your app (push to hosts array) - modify this to suit your app
Blog.find(:all).each { |blog| hosts << "#{blog.subdomain}.blog.local" unless blog.subdomain.blank? }
# build hosts line to add/edit
host_line = "127.0.0.1 " + hosts.sort.unshift(default_hosts).join(' ')
# work with a copied hosts file in tmp
%x[cp /etc/hosts #{tmp_file}]
file = File.new(tmp_file)
lines = file.readlines
lines.each do |line|
changed = true if line.gsub!(/^127.0.0.1 #{Regexp.escape(default_hosts.join(' '))}.+$/, host_line)
end
# add line, if no line found for update
lines += ["\n", host_line, "\n"] unless changed
file = File.new(tmp_file,'w')
lines.each { |line| file.write(line) }
file.close
# copy hosts file from tmp - may ask for sudo password
%x[sudo -p "Password:" cp #{tmp_file} /etc/hosts]
# explain what happened
puts "\nAdded the following domains:"
hosts.each { |host| puts "* http://#{host}" }
puts "\nAlso added defaults:"
default_hosts.each { |default| puts "* http://#{default}" }
puts "\n"
end
endTo run simply type
rake subdomains:setupWorking on the rails day in and out now, I’ve found the following aliases to come in handy;
# General Commands
alias ls='ls -al'
# TextMate, mate all of current dir and crucial rails folders only
alias et='mate . &'
alias ett='mate app config lib db public test vendor/plugins &'
# RAILS, (run these from your rails folder)
# rails scripts
alias ss='./script/server'
alias sc='./script/console'
alias sg='./script/generate'
alias sp='./script/plugin'
alias mr='mongrel_rails start'
# rails testing
alias att='autotest'
alias tu='rake test:units'
alias tf='rake test:functionals'
# tail logs
alias tl='tail -f ./log/development.log'
alias tt='tail -f ./log/test.log'
# clean the logs
alias ctl='cp /dev/null ./log/test.log'
alias cdl='cp /dev/null ./log/development.log'
I should credit Peep Code for the idea. To use, (e.g. in OSX) place the above in a ~/.bash_aliases file and in ~/.bash_profile, load it in with this command;
if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases ; fiAlso, (and before I forget it myself) – here’s a quick session cleaner command to put in your cron, (for day old, mysql session clearage action); rather than build a rake task, or extra controller to clean them out.
#!/bin/bash
cd /u/apps/matthewhutchinson.net/current
echo "<== CRON TASK ==> clear day old sessions data on matthewhutchinson.net"
ruby script/runner -e production "ActiveRecord::Base.connection.delete(\"DELETE FROM sessions WHERE updated_at < NOW() - INTERVAL 1 DAY\")"
I'm a professional web developer and the founder of Hiddenloop a small web shop working on Bugle. You can follow me on twitter @matthutchin or find out more about me here.
This is a building site at the moment, so bear with me until I carefully dismantle the rest of the scaffolding (like this awfully messy sidebar).
powered by Bugle