articles by matt

Following on with subdomains, say you want one field to auto-suggest into another. For example, when a user enters their company name (or username, last name etc.) this same value could be used to suggest a valid subdomain. Here’s the javascript and html (the function uses prototype for the element lookup).

The suggestion is based on the facts that valid subdomains should be alphanumeric, with dashes, less than 63 chars and not start or end with a dash.

<label for="username">Username</label>
<input id="username" onkeyup="suggestSubdomain(value, 'subdomain');" type="text" name="username" /><br/>
<label for="subdomain">Subdomain</label>
<input id="subdomain" type="text" name="subdomain" />

<script type="text/javascript">
  function suggestSubdomain(value, element) {
    var subdomain = value.gsub(/\s+/, '-');
    subdomain = subdomain.gsub(/[^a-z0-9\-]/i, '');
    if(subdomain[0] == '-')
      subdomain = subdomain.substr(1, subdomain.length)
    if(subdomain[subdomain.length-1] == '-')
      subdomain = subdomain.substr(0, subdomain.length-1)

    $(element).value = subdomain.substr(0, 63).toLowerCase();
  }
</script>

# subdomain_validator.rb (place in your lib/ or extra/ load path)
class SubdomainValidator < ActiveModel::EachValidator
  
  def validate_each(object, attribute, value)
    return unless value.present?
    reserved_names = %w(www ftp mail pop smtp admin ssl sftp)
    reserved_names = options[:reserved] if options[:reserved]
    if reserved_names.include?(value)
      object.errors[attribute] << 'cannot be a reserved name'
    end                                              
    
    object.errors[attribute] << 'must have between 3 and 63 letters' unless (3..63) === value.length
    object.errors[attribute] << 'cannot start or end with a hyphen' unless value =~ /^[^-].*[^-]$/i                                                                                                    
    object.errors[attribute] << 'must be alphanumeric; A-Z, 0-9 or hyphen' unless value =~ /^[a-z0-9\-]*$/i
  end
end

# And in your model
validates  :subdomain, :presence   => true,
                       :uniqueness => true,
                       :subdomain  => true

# Or with your own reserved names
validates  :subdomain, :presence   => true,
                       :uniqueness => true,
                       :subdomain  => { :reserved => %w(foo bar) }

For more on creating custom Rails 3 validators, check our Ryan Bates’s screencast on the topic.

NOTE: Apparently there are issues with this code and the latest Paperclip gem (currently 2.3.4) – its down to the use of reprocess and this known issue i’m currently looking at a work around here’s a patch fix

I use Paperclip for pretty much all upload processing. Its flexible, fast and easily extendable. One particular feature that has cropped up (a couple of times now) – has been the ability to edit and update the contents of uploaded files. For example; editing css, html or javascript in a CMS. Something I’ve needed for Bugle

In the past I struggled getting this to work with Paperclip. You’ll find me rambling to myself in the mailing list almost a year ago. I figured the Shopify guys we’re doing this in their app, so it had to be possible.

One solution, was to simply read the file contents (on create) from the uploaded file into a database column. Then on future requests for the file, serve it virtually from the database; through a Rails controller/action responding with the appropriate content type and content data.

But, this meant the Rails app would be handling all the css/js requests in the CMS. I really wanted to serve these uploaded files from S3/Cloudfront making full use of Amazon’s CDN. So I set about building a Paperclip::Processor to store the file contents (in the database) on create then on update, update contents and re-upload the file again. To work with cache expiry in the CDN I could use the updated_on timestamp in the URL to the file.

Here’s most of the code below, i’ve also created a git repository with a working simple app. I’m using a RESTful UploadsController with an Upload model. The model has an Paperclip attachment (asset) and the file contents (for editable files) are stored in a TEXT column (‘asset_contents’ in the database).

Controller

Nothing crazy going on here, just straight forward RESTful controller logic (without a show action)

class UploadsController < ApplicationController                       
  
  def index
    @uploads = Upload.scoped
  end
  
  def new
    @upload = Upload.new
  end

  def edit
    @upload = Upload.find(params[:id])
  end

  def create
    @upload = Upload.new(params[:upload])    
    if @upload.save
      flash[:notice] = 'Upload was successfully created'
      redirect_to uploads_url
    else 
      render 'new'
    end
  end

  def update
    @upload = Upload.find(params[:id])    
    if @upload.editable? && @upload.update_attributes(params[:upload])
      flash[:notice] = 'Upload was successfully updated'
      redirect_to uploads_url
    else 
      render 'edit'
    end
  end 
  
  def destroy
    @upload = Upload.find(params[:id])    
    if @upload.destroy
      flash[:notice] = 'Upload was successfully deleted'
    end
    redirect_to uploads_url                          
  end
end

Model

Two things to notice here. I’m using lambda’s on the style and processor attributes. In both cases they check the content-type to see if the file is either editable or thumbnailable. If it is a thumbnailable image, I give it a thumbnail style and the thumbnail (default) Paperclip::Processor. For editable files, I give it a style for the original file only, and use the new FileContents Processor (see below). The style hash sets which database column will be used for storing the file contents, in this case it’s the ‘asset_contents’ attribute.

Second is the after_update hook. When thew Upload model gets saved, I want Paperclip to reprocess the asset again. This ensures that when the asset is saved on update the FileContents processor executes. The thumbnailable? and editable? methods let you decide what file types should be considered for processing.

class Upload < ActiveRecord::Base
                      
  after_update :reprocess

  has_attached_file :asset, :styles         => lambda { |a|
                                                 if a.instance.thumbnailable?
                                                   {:thumb => ["64x64#", :jpg]}
                                                 elsif a.instance.editable?
                                                   {:original => {:contents => 'asset_contents'}}
                                                 end
                                               },
                            :path           => "/:id/:style/:basename.:extension",
                            :storage        => :s3,     
                            :s3_credentials => "#{Rails.root}/config/s3.yml",
                            :bucket         => "paperclip-example-bucket-#{Rails.env}",
                            :processors     => lambda { |a|
                                                 if a.editable?
                                                   [:file_contents]
                                                 elsif a.thumbnailable?
                                                   [:thumbnail]
                                                 end
                                               }
  
  attr_protected :asset_file_name, :asset_content_type, :asset_size          
  
  validates_attachment_size     :asset, :less_than => 6.megabytes
  validates_attachment_presence :asset

  def editable?
    return false unless asset.content_type
    ['text/css', 'application/js', 'text/plain', 'text/x-json', 'application/json', 'application/javascript',
     'application/x-javascript', 'text/javascript', 'text/x-javascript', 'text/x-json',
     'text/html', 'application/xhtml', 'application/xml', 'text/xml', 'text/js'].join('').include?(asset.content_type)
  end
  
  def thumbnailable?
    return false unless asset.content_type
    ['image/jpeg', 'image/pjpeg', 'image/gif', 'image/png', 'image/x-png', 'image/jpg'].join('').include?(asset.content_type)
  end 
  
  private
  def reprocess
    asset.reprocess! if editable?
  end
end

FileContents Paperclip::Processor

This processor basically reads the uploaded file contents on create and sets the asset_contents attribute. On update, it creates a new Tempfile with its content from the asset_contents attribute and then returns this Tempfile for Paperclip uploading. Comments in the code below explain further, (place this file in lib/paperclip/file_contents.rb).

module Paperclip
  class FileContents < Processor
    
    def initialize file, options = {}, attachment = nil
      @file           = file
      @options        = options
      @instance       = attachment.instance
      @current_format = File.extname(attachment.instance.asset_file_name)
      @basename       = File.basename(@file.path, @current_format)
      @whiny          = options[:whiny].nil? ? true : options[:whiny]
    end

    def make
      begin
        # new record, set contents attribute by reading the attachment file
        if(@instance.new_record?)
          @file.rewind # move pointer back to start of file in case handled by other processors
          file_content = File.read(@file.path)
          @instance.send("#{@options[:contents]}=", file_content)
        else                                                     
          # existing record, set contents by reading contents attribute
          file_content = @instance.send(@options[:contents])
          # create new file with contents from model
          tmp = Tempfile.new([@basename, @current_format].compact.join("."))
          tmp << file_content
          tmp.flush 
          @file = tmp
        end         
                 
        @file
      rescue StandardError => e
        raise PaperclipError, "There was an error processing the file contents for #{@basename} - #{e}" if @whiny
      end
    end
  end
end

Views

The view code is simple, a new and edit form with a textarea for contents editing.

# uploads/new.html.erb
<%= form_for(:upload, :url => uploads_path,
                      :html => { :method => :post, :multipart => true }) do |f| %>               
  <input type="file" name="upload[asset]"> <%= f.submit 'upload', :disable_with => 'uploading ...' %>  
<% end %>

# uploads/edit.html.erb
<%= form_for @upload do |f| %>
  <%= f.text_area :asset_contents, :rows => 20, :cols => 100, :id => 'file_asset_contents' %>
  <p><%= f.submit 'Save changes', :disable_with => 'saving ...' %></p>
<% end -%>

# reference upload URL always with timestamp
<%= @upload.asset.url(:original, true) %>

Some gotchas

If you are using an Amazon S3 bucket, make sure you set it to be ‘world’ readable, so your uploaded files are publicly accessible. Also, the file_contents.rb processor should live in lib/paperclip/file_contents.rb. And for a Rails 3 add this to your load path, in config/application.rb

config.autoload_paths += %W(#{Rails.root}/lib)

I’ve been running this code with no issues in production for some time now. I should point out that I limit these editable uploads to ~3Mb-6Mb and you may have performance issues with larger files. Some solutions could be to use delayed_job (or something similar) to background process the task, and/or change the processor code to read/write one line at a time.

Further reading

• Application Example (on gitHub)
• Other posts of mine on Paperclip
• Paperclip and docs
• Amazon Cloudfront

I recently spent some time looking at application security on an old and new (Rails 3) app. Below is short (rather long) summary of my findings.

Quick caveat, this is by no means meant to be an exhaustive post on Rails security (for that I’d recommend you read the official Ruby On Rails Security Guide)

I’ve basically highlighted what I think are the bare minimum of checks you should be doing before throwing an application into the wild. And i’m focusing more on the actual preventative measures. Also I should point out that (unless your app is very small) security is not something you can retro-fit at the end of a project. To avoid holes you really need to be thinking about these things as you go.

Do not trust logged in users! (Authentication != Authorisation)

‘Change-affecting’ actions create/update/delete (and their corresponding views new/edit) need to be protected. So always check the current logged user is allowed to perform the requested action.

One way to do this, is to have the concept of a user owning a model and scoping find’s on these actions by the user. Or consider adding a before filter to the controller, that validates the logged in user against the action being performed on the object. For instance, create methods like can_edit?(by_user) or can_destroy?(by_user)

Remember too, that the most popular Rails plugins, Clearance, Devise and Restful Authentication only authenticate users, and provide no helpers or methods for authorisation. Also, I’d advise against building your own authentication system from scratch!

Mass assignment

A params hash can contain anything, so protect all sensitive attributes from re-assignment. The best way to do this is through disabling mass assignment by using attr_accessible (or attr_protected)in your models. Attributes like counter caches, account_id’s and user_id’s are prime examples for protection.

Un-editable attributes

Disable updates on protected attributes. Using attr_readonly in ActiveRecord allows the attribute to be set on create, but never edited on update. E.g. In many cases an account_id should be set once and never change.

SQL Injection

Don’t include user submitted strings in database queries! Check all model scopes and find conditions that include params or interpolated strings. E.g. LIKE searches should have the keyword passed in this form;

posts.where('posts.title LIKE ?', "%#{keyword}%")

Prevent executable files from being uploaded

Consider validating the content type on all attachments, and place uploaded files in protected directories or on another server/service e.g. S3/Cloudfront. Content-types can easily be faked, so check file extensions and be sure to disable your web server from executing scripts in the upload directory.

Also, beware of plugins creating or writing in tmp directories during file upload creation. They may create files or directories from user submitted params without checking the file path. In general be familiar with your gems and plugin code.

Filter sensitive params from logs

Use filter_parameter_logging or config.filter_parameters (Rails 3) to remove sensitive data from your logs (:password, :password_confirmation, :credit_card_number etc.)

CSRF

Enable protect_from_forgery and use form helpers to include the Rails authenticity token in all form submissions. In Rails 3 ensure the HTML head tag includes the csrf_meta_tag helper.

XSS

Prevent XSS and javascript injection by escaping all rendered user content (use the ‘h’ helper). In Rails 3 this happens by default so be careful what you whitelist for rendering (with raw, or html_safe!) If you are using something like Textile or Markdown, make use of the whitelist filtering methods their libraries provide.

Session Hijacks

Sessions are open to a number of vulnerabilities. Hijacking, replaying cookies, session fixation etc. Check the Ruby On Rails Security Guide for all the details. Here are some general tips;

If storing login information e.g. a remember_me cookie for authentication use CookieStore with a digest and reasonably complex server-side secret.

Be careful what data you decide to store and use in your sessions. Don’t store ActiveRecord objects, future code changes or migrations may change an objects behaviour/attributes and clients with old session data will be out of sync. Better to store only id’s for records in the session and load model objects server-side.

Other things to NOT store in your session include; money balances, user access privileges or sensitive state based information etc. These are open to session replay attacks.

Finally, to avoid session fixation attacks, call reset_session after a successful login.

Redirecting

Avoid using redirect_to(params[:some_param]). When the arguments for a redirect come from params, you are open to redirects to unintended URLs.

Downloading files, raw data

Avoid using params or user content in the send_file method. Consider having these as absolute filenames stored in the database, or validate the path to the file before calling send_file.

Make non-action controller methods private.

Remember that any methods you declare in a controller could be accessible to the public depending on your routes.

Gems and plugins

Remember to check your dependencies for security updates and patches. If possible subscribe to the GitHub issues list or (any mailing list) for the gems or plugins your using.

Passwords

Don’t store passwords in the database as clear text! Maybe that one goes without saying :) Encourage strong alphanumeric passwords and if necessary follow other strong password practices (throttle multiple failed logins, password expiry/reset etc.)

Also, you really should use BCrypt to discourage rainbow table attacks. Read this to find out more.

Further reading

Most of this checklist has been compiled from the following links.

• Ruby On Rails Security Guide (start with this!)
• Security lessons from the Diaspora launch
• Enough with rainbow tables, what you need to know about secure passwords
• 7 Security Tips (screencast)

I really enjoyed this, as a programmer myself, it was great to finally see a film where hacking was portrayed like hacking actually is in the real world! Without the bells, whistles, tin-foil hats, 3D visualizations or CGI we’d normally expect from Hollywood.

By comparison, here are some fine prime examples of ridiculous movie hacking sequences. If I took Hollywood as the basis for what hacking or coding actually looked like, I would be under the impression I’ve been doing it wrong all these years.

Oh! And another reason to see this movie, vi has finally made it to the big screen!! For a comprehensive, detailed review of The Social Network, check out thefilmtalk.com

So after months of on again, off again training, here’s the proof that I finally did finish the 2010 Berlin Marathon last weekend. Thanks to everyone who donated money to the British Heart Foundation (hint, you can still add you’re donation now, if you haven’t already)

As further proof, and thanks to multiple cameras at different points along the route, you can actually watch me run. It might be hard to spot me in the crowd, I’m wearing that same red vest as in the photo above, lumbering around in the center or off to the side.

It was a good run, I was hoping for around 4 hours, but happy with 4:30 and no injuries. A lot faster than my first marathon in New York City three years ago.

While I was in Berlin with other friends who were running, I ran this one pretty much on my own. With an ipod nano and the same playlist I’d been training to. It did help and the time passed very quickly. I was 2km from the finish when the 4:30 pace runners came up behind me, that spurred me on a bit and I managed to keep up with them until the final stretch.

The last 1km of a marathon is a funny sight, if you take the time to look around you, you’ll see plenty of worn out people just stopping right beside you. Either they’ve seen the finish line in the distance and realised its further than they thought, or they’re just plain beat. On the other hand you’ve got people sprinting towards the finish, or couples joining to run together over the line.

I should mention the rain. It never stopped the entire time; from arriving at the airport, until leaving 3 days later, it rained constantly. This probably affected the turnout of supporters along the route. It was great to see people out cheering, but the atmosphere couldn’t really compare to NYC.

Finish & split times

These stats from the official race clock were sent to my phone and email as I ran.

Nettozeit / Chip Total	4:34:57
Geschwindigkeit / Speed	9.2 km/h  6:31 min/km
  
Halbmarathon 1 / Half Time 1	2:06:37
Halbmarathon 2 / Half Time 2	2:28:21
  
km 5     	0:28:20
km 10     	0:57:40
km 15     	1:27:22
km 20     	1:59:03
km 25     	2:32:31
km 30     	3:08:03
km 35     	3:41:57
km 40     	4:19:45
Finish          4:34:57

After updating a few apps to Rails 3 I came across some issues with the new scoping syntax.

Ambiguous column names

In one case I use a module to define some common scopes that are used across more than one model. I quickly came up against the ambiguous column name error (e.g. Column ‘created_at’ in order clause is ambiguous —from MySQL). To fix this I used the table_name prefix right in my scoping string. In general you should always do this anywhere you use strings in your scopes. For example;

# remember to add the table_name to avoid ambiguous columns
scope :recent, order("#{table_name}.created_at DESC")
# or directly
scope :recent, order("questions.created_at DESC")

# you don't have to add the table_name when using a hash, Rails takes care of that!
scope :by_user, lambda{|user| where(:user_id => user.id).recent}

Multiple order scopes

Another issue I found was combining multiple order scopes. In Rails 2 with named scopes this used to work as you’d expect. But combining multiple order scopes in Rails 3 seems to fail, only applying the last order scope to the query.

# given the recent scope defined above, in this case we'd expect;
scope :flagged, order(" #{table_name}.flag_count DESC").recent
# to be 
'ORDER BY questions.flag_count DESC, questions.created_at DESC'
# BUT its not! instead its simply
'ORDER BY questions.created_at DESC'
# to order on multiple columns inside a scope, I have to duplicate my recent order scope like so
scope :flagged, order(" #{table_name}.flag_count DESC, #{table_name}.created_at DESC").recent

All in all, i’m very much in favour of the new syntax and AREL in general. It simplifies compounding queries together with additional scopes and delays their execution until it is needed.

Some Rails 3 ActiveRecord links

• ActiveRecord querying
• ActiveRecord Query Interface
• ActiveRecord Queries in Rails 3 (railscast)
• Advanced Queries in Rails 3 (railscast)
• The skinny on scopes

I’ve been upgrading some apps and gems to the latest and greatest Rails 3. I’ve taken the time to update my ~/.bash_aliases and ~/.irbrc files to be both Rails 3 and Rails 2.x compatible. See below for the code.

~/.bash_aliases

# rails 3 shortcut 'r'
alias r='rails'

# launching console/server
sc () {
  if [ -f ./script/rails ]; then 
    rails c $@
  else
    ./script/console $@
  fi
}

sg () {
  if [ -f ./script/rails ]; then
    rails g $@
  else
    ./script/generate $@
  fi
}

ss () {
  if [ -f ./script/rails ]; then 
    rails s $@
  else
    ./script/server $@
  fi
}

sspe () {
  if [ -f ./script/rails ]; then 
    sudo rails s -p80 $@
  else
    sudo ./script/server -p80 $@
  fi
}

# database migrate
alias rdbm='rake db:migrate'

# tests
alias rspec='rake spec'

# rails logs, tailing and cleaning
alias tdl='tail -f ./log/development.log'
alias ttl='tail -f ./log/test.log'
alias ctl='> ./log/test.log'
alias cdl='> ./log/development.log'

~/.irbrc

require 'rubygems' rescue nil
require 'wirble'
require 'hirb'
require 'ap'

# load wirble
Wirble.init
Wirble.colorize

# load hirb
Hirb::View.enable

IRB.conf[:AUTO_INDENT] = true

if ENV.include?('RAILS_ENV')
  if !Object.const_defined?('RAILS_DEFAULT_LOGGER')
    require 'logger'
    Object.const_set('RAILS_DEFAULT_LOGGER', Logger.new(STDOUT))
  end

  def sql(query)
    ActiveRecord::Base.connection.select_all(query)
  end
  
  if ENV['RAILS_ENV'] == 'test'
    require 'test/test_helper'
  end

# for rails 3
elsif defined?(Rails) && !Rails.env.nil?
  if Rails.logger
    Rails.logger =Logger.new(STDOUT)
    ActiveRecord::Base.logger = Rails.logger
  end
  if Rails.env == 'test'
    require 'test/test_helper'
  end
else
  # nothing to do
end

# annotate column names of an AR model
def show(obj)
  y(obj.send("column_names"))
end

puts "> all systems are go wirble/hirb/ap/show <"

Note that if your’e using Bundler in your Rails app, AND use gems in your ~/.irbrc file AND attempt to start the Rails console; you’ll get errors/warnings on requiring them UNLESS you define them in your Gemfile. I use a ‘development’ group in my Gemfile for these, like so.

Gemfile

group :development do
  gem "wirble"
  gem "hirb"
  gem "awesome_print"
end

platforms :ruby_18 do
  gem 'ruby-debug'
end 

platforms :ruby_19 do
  gem 'ruby-debug19'
end