articles tagged with rails

Recurring billing talk at Ruby Ireland

no comments yet, post one now

Thanks to everyone who showed up last night for the monthly Ruby Ireland user group meet-up at the Twisted Pepper. Unfortunately we completely forgot to try recording the presentation, but for those interested the slides are available here.

There did seem to be a genuine interest in this topic so I’ve offered to write a blog post (or perhaps a series of posts) explaining the implementation in more detail and with actual code snippets. To summarize what I spoke about;

I gave an overview on the basics of taking payments online, specifically;

  • Banks
  • Payment Gateways
  • Libraries/code/web services available for your Rails app.

I explained each core method implemented in an ActiveMerchant billing gateway class, and I covered the important design decisions I took before building in recurring billing;

  • High level decisions (which bank/gateway etc.)
  • Pricing decisions (price plans, upgrade/downgrade paths/logic etc.)
  • Actual implementation decisions (how its coded etc.)

Finally I talked about some pain points you’ll likely to come across when working with recurring billing and online payments in general. I hope everyone enjoyed the talk and I’d be happy to answer any further questions if you have them.

Ps. Apologies for the VGA cable problems, thanks to Declan’s trusty netbook we got there in the end!

Rails 3 subdomain validation (ActiveModel::EachValidator)

4 comments

# subdomain_validator.rb (place in your lib/ or extra/ load path)
class SubdomainValidator < ActiveModel::EachValidator
  
  def validate_each(object, attribute, value)
    return unless value.present?
    reserved_names = %w(www ftp mail pop smtp admin ssl sftp)
    reserved_names = options[:reserved] if options[:reserved]
    if reserved_names.include?(value)
      object.errors[attribute] << 'cannot be a reserved name'
    end                                              
    
    object.errors[attribute] << 'must have between 3 and 63 letters' unless (3..63) === value.length
    object.errors[attribute] << 'cannot start or end with a hyphen' unless value =~ /^[^-].*[^-]$/i                                                                                                    
    object.errors[attribute] << 'must be alphanumeric; A-Z, 0-9 or hyphen' unless value =~ /^[a-z0-9\-]*$/i
  end
end

# And in your model
validates  :subdomain, :presence   => true,
                       :uniqueness => true,
                       :subdomain  => true

# Or with your own reserved names
validates  :subdomain, :presence   => true,
                       :uniqueness => true,
                       :subdomain  => { :reserved => %w(foo bar) }

For more on creating custom Rails 3 validators, check our Ryan Bates’s screencast on the topic.

Editing file uploads with a Paperclip processor

8 comments

NOTE: Apparently there are issues with this code and the latest Paperclip gem (currently 2.3.4) – its down to the use of reprocess and this known issue i’m currently looking at a work around here’s a patch fix

I use Paperclip for pretty much all upload processing. Its flexible, fast and easily extendable. One particular feature that has cropped up (a couple of times now) – has been the ability to edit and update the contents of uploaded files. For example; editing css, html or javascript in a CMS. Something I’ve needed for Bugle

In the past I struggled getting this to work with Paperclip. You’ll find me rambling to myself in the mailing list almost a year ago. I figured the Shopify guys we’re doing this in their app, so it had to be possible.

One solution, was to simply read the file contents (on create) from the uploaded file into a database column. Then on future requests for the file, serve it virtually from the database; through a Rails controller/action responding with the appropriate content type and content data.

But, this meant the Rails app would be handling all the css/js requests in the CMS. I really wanted to serve these uploaded files from S3/Cloudfront making full use of Amazon’s CDN. So I set about building a Paperclip::Processor to store the file contents (in the database) on create then on update, update contents and re-upload the file again. To work with cache expiry in the CDN I could use the updated_on timestamp in the URL to the file.

Here’s most of the code below, i’ve also created a git repository with a working simple app. I’m using a RESTful UploadsController with an Upload model. The model has an Paperclip attachment (asset) and the file contents (for editable files) are stored in a TEXT column (‘asset_contents’ in the database).

Controller

Nothing crazy going on here, just straight forward RESTful controller logic (without a show action)

class UploadsController < ApplicationController                       
  
  def index
    @uploads = Upload.scoped
  end
  
  def new
    @upload = Upload.new
  end

  def edit
    @upload = Upload.find(params[:id])
  end

  def create
    @upload = Upload.new(params[:upload])    
    if @upload.save
      flash[:notice] = 'Upload was successfully created'
      redirect_to uploads_url
    else 
      render 'new'
    end
  end

  def update
    @upload = Upload.find(params[:id])    
    if @upload.editable? && @upload.update_attributes(params[:upload])
      flash[:notice] = 'Upload was successfully updated'
      redirect_to uploads_url
    else 
      render 'edit'
    end
  end 
  
  def destroy
    @upload = Upload.find(params[:id])    
    if @upload.destroy
      flash[:notice] = 'Upload was successfully deleted'
    end
    redirect_to uploads_url                          
  end
end

Model

Two things to notice here. I’m using lambda’s on the style and processor attributes. In both cases they check the content-type to see if the file is either editable or thumbnailable. If it is a thumbnailable image, I give it a thumbnail style and the thumbnail (default) Paperclip::Processor. For editable files, I give it a style for the original file only, and use the new FileContents Processor (see below). The style hash sets which database column will be used for storing the file contents, in this case it’s the ‘asset_contents’ attribute.

Second is the after_update hook. When thew Upload model gets saved, I want Paperclip to reprocess the asset again. This ensures that when the asset is saved on update the FileContents processor executes. The thumbnailable? and editable? methods let you decide what file types should be considered for processing.

class Upload < ActiveRecord::Base
                      
  after_update :reprocess

  has_attached_file :asset, :styles         => lambda { |a|
                                                 if a.instance.thumbnailable?
                                                   {:thumb => ["64x64#", :jpg]}
                                                 elsif a.instance.editable?
                                                   {:original => {:contents => 'asset_contents'}}
                                                 end
                                               },
                            :path           => "/:id/:style/:basename.:extension",
                            :storage        => :s3,     
                            :s3_credentials => "#{Rails.root}/config/s3.yml",
                            :bucket         => "paperclip-example-bucket-#{Rails.env}",
                            :processors     => lambda { |a|
                                                 if a.editable?
                                                   [:file_contents]
                                                 elsif a.thumbnailable?
                                                   [:thumbnail]
                                                 end
                                               }
  
  attr_protected :asset_file_name, :asset_content_type, :asset_size          
  
  validates_attachment_size     :asset, :less_than => 6.megabytes
  validates_attachment_presence :asset

  def editable?
    return false unless asset.content_type
    ['text/css', 'application/js', 'text/plain', 'text/x-json', 'application/json', 'application/javascript',
     'application/x-javascript', 'text/javascript', 'text/x-javascript', 'text/x-json',
     'text/html', 'application/xhtml', 'application/xml', 'text/xml', 'text/js'].join('').include?(asset.content_type)
  end
  
  def thumbnailable?
    return false unless asset.content_type
    ['image/jpeg', 'image/pjpeg', 'image/gif', 'image/png', 'image/x-png', 'image/jpg'].join('').include?(asset.content_type)
  end 
  
  private
  def reprocess
    asset.reprocess! if editable?
  end
end

FileContents Paperclip::Processor

This processor basically reads the uploaded file contents on create and sets the asset_contents attribute. On update, it creates a new Tempfile with its content from the asset_contents attribute and then returns this Tempfile for Paperclip uploading. Comments in the code below explain further, (place this file in lib/paperclip/file_contents.rb).

module Paperclip
  class FileContents < Processor
    
    def initialize file, options = {}, attachment = nil
      @file           = file
      @options        = options
      @instance       = attachment.instance
      @current_format = File.extname(attachment.instance.asset_file_name)
      @basename       = File.basename(@file.path, @current_format)
      @whiny          = options[:whiny].nil? ? true : options[:whiny]
    end

    def make
      begin
        # new record, set contents attribute by reading the attachment file
        if(@instance.new_record?)
          @file.rewind # move pointer back to start of file in case handled by other processors
          file_content = File.read(@file.path)
          @instance.send("#{@options[:contents]}=", file_content)
        else                                                     
          # existing record, set contents by reading contents attribute
          file_content = @instance.send(@options[:contents])
          # create new file with contents from model
          tmp = Tempfile.new([@basename, @current_format].compact.join("."))
          tmp << file_content
          tmp.flush 
          @file = tmp
        end         
                 
        @file
      rescue StandardError => e
        raise PaperclipError, "There was an error processing the file contents for #{@basename} - #{e}" if @whiny
      end
    end
  end
end

Views

The view code is simple, a new and edit form with a textarea for contents editing.

# uploads/new.html.erb
<%= form_for(:upload, :url => uploads_path,
                      :html => { :method => :post, :multipart => true }) do |f| %>               
  <input type="file" name="upload[asset]"> <%= f.submit 'upload', :disable_with => 'uploading ...' %>  
<% end %>

# uploads/edit.html.erb
<%= form_for @upload do |f| %>
  <%= f.text_area :asset_contents, :rows => 20, :cols => 100, :id => 'file_asset_contents' %>
  <p><%= f.submit 'Save changes', :disable_with => 'saving ...' %></p>
<% end -%>

# reference upload URL always with timestamp
<%= @upload.asset.url(:original, true) %>

Some gotchas

If you are using an Amazon S3 bucket, make sure you set it to be ‘world’ readable, so your uploaded files are publicly accessible. Also, the file_contents.rb processor should live in lib/paperclip/file_contents.rb. And for a Rails 3 add this to your load path, in config/application.rb

config.autoload_paths += %W(#{Rails.root}/lib)

I’ve been running this code with no issues in production for some time now. I should point out that I limit these editable uploads to ~3Mb-6Mb and you may have performance issues with larger files. Some solutions could be to use delayed_job (or something similar) to background process the task, and/or change the processor code to read/write one line at a time.

Further reading

October 25, 2010 19:23

Yet another Rails security checklist

3 comments

I recently spent some time looking at application security on an old and new (Rails 3) app. Below is short (rather long) summary of my findings.

Quick caveat, this is by no means meant to be an exhaustive post on Rails security (for that I’d recommend you read the official Ruby On Rails Security Guide)

I’ve basically highlighted what I think are the bare minimum of checks you should be doing before throwing an application into the wild. And i’m focusing more on the actual preventative measures. Also I should point out that (unless your app is very small) security is not something you can retro-fit at the end of a project. To avoid holes you really need to be thinking about these things as you go.

Do not trust logged in users! (Authentication != Authorisation)

‘Change-affecting’ actions create/update/delete (and their corresponding views new/edit) need to be protected. So always check the current logged user is allowed to perform the requested action.

One way to do this, is to have the concept of a user owning a model and scoping find’s on these actions by the user. Or consider adding a before filter to the controller, that validates the logged in user against the action being performed on the object. For instance, create methods like can_edit?(by_user) or can_destroy?(by_user)

Remember too, that the most popular Rails plugins, Clearance, Devise and Restful Authentication only authenticate users, and provide no helpers or methods for authorisation. Also, I’d advise against building your own authentication system from scratch!

Mass assignment

A params hash can contain anything, so protect all sensitive attributes from re-assignment. The best way to do this is through disabling mass assignment by using attr_accessible (or attr_protected)in your models. Attributes like counter caches, account_id’s and user_id’s are prime examples for protection.

Un-editable attributes

Disable updates on protected attributes. Using attr_readonly in ActiveRecord allows the attribute to be set on create, but never edited on update. E.g. In many cases an account_id should be set once and never change.

SQL Injection

Don’t include user submitted strings in database queries! Check all model scopes and find conditions that include params or interpolated strings. E.g. LIKE searches should have the keyword passed in this form;

posts.where('posts.title LIKE ?', "%#{keyword}%")

Prevent executable files from being uploaded

Consider validating the content type on all attachments, and place uploaded files in protected directories or on another server/service e.g. S3/Cloudfront. Content-types can easily be faked, so check file extensions and be sure to disable your web server from executing scripts in the upload directory.

Also, beware of plugins creating or writing in tmp directories during file upload creation. They may create files or directories from user submitted params without checking the file path. In general be familiar with your gems and plugin code.

Filter sensitive params from logs

Use filter_parameter_logging or config.filter_parameters (Rails 3) to remove sensitive data from your logs (:password, :password_confirmation, :credit_card_number etc.)

CSRF

Enable protect_from_forgery and use form helpers to include the Rails authenticity token in all form submissions. In Rails 3 ensure the HTML head tag includes the csrf_meta_tag helper.

XSS

Prevent XSS and javascript injection by escaping all rendered user content (use the ‘h’ helper). In Rails 3 this happens by default so be careful what you whitelist for rendering (with raw, or html_safe!) If you are using something like Textile or Markdown, make use of the whitelist filtering methods their libraries provide.

Session Hijacks

Sessions are open to a number of vulnerabilities. Hijacking, replaying cookies, session fixation etc. Check the Ruby On Rails Security Guide for all the details. Here are some general tips;

If storing login information e.g. a remember_me cookie for authentication use CookieStore with a digest and reasonably complex server-side secret.

Be careful what data you decide to store and use in your sessions. Don’t store ActiveRecord objects, future code changes or migrations may change an objects behaviour/attributes and clients with old session data will be out of sync. Better to store only id’s for records in the session and load model objects server-side.

Other things to NOT store in your session include; money balances, user access privileges or sensitive state based information etc. These are open to session replay attacks.

Finally, to avoid session fixation attacks, call reset_session after a successful login.

Redirecting

Avoid using redirect_to(params[:some_param]). When the arguments for a redirect come from params, you are open to redirects to unintended URLs.

Downloading files, raw data

Avoid using params or user content in the send_file method. Consider having these as absolute filenames stored in the database, or validate the path to the file before calling send_file.

Make non-action controller methods private.

Remember that any methods you declare in a controller could be accessible to the public depending on your routes.

Gems and plugins

Remember to check your dependencies for security updates and patches. If possible subscribe to the GitHub issues list or (any mailing list) for the gems or plugins your using.

Passwords

Don’t store passwords in the database as clear text! Maybe that one goes without saying :) Encourage strong alphanumeric passwords and if necessary follow other strong password practices (throttle multiple failed logins, password expiry/reset etc.)

Also, you really should use BCrypt to discourage rainbow table attacks. Read this to find out more.

Further reading

Most of this checklist has been compiled from the following links.

October 21, 2010 22:10

Rails 3 ambiguous column names and multiple order scopes

3 comments

After updating a few apps to Rails 3 I came across some issues with the new scoping syntax.

Ambiguous column names

In one case I use a module to define some common scopes that are used across more than one model. I quickly came up against the ambiguous column name error (e.g. Column ‘created_at’ in order clause is ambiguous —from MySQL). To fix this I used the table_name prefix right in my scoping string. In general you should always do this anywhere you use strings in your scopes. For example;

# remember to add the table_name to avoid ambiguous columns
scope :recent, order("#{table_name}.created_at DESC")
# or directly
scope :recent, order("questions.created_at DESC")

# you don't have to add the table_name when using a hash, Rails takes care of that!
scope :by_user, lambda{|user| where(:user_id => user.id).recent}

Multiple order scopes

Another issue I found was combining multiple order scopes. In Rails 2 with named scopes this used to work as you’d expect. But combining multiple order scopes in Rails 3 seems to fail, only applying the last order scope to the query.

# given the recent scope defined above, in this case we'd expect;
scope :flagged, order(" #{table_name}.flag_count DESC").recent
# to be 
'ORDER BY questions.flag_count DESC, questions.created_at DESC'
# BUT its not! instead its simply
'ORDER BY questions.created_at DESC'
# to order on multiple columns inside a scope, I have to duplicate my recent order scope like so
scope :flagged, order(" #{table_name}.flag_count DESC, #{table_name}.created_at DESC").recent

All in all, i’m very much in favour of the new syntax and AREL in general. It simplifies compounding queries together with additional scopes and delays their execution until it is needed.

Some Rails 3 ActiveRecord links

September 22, 2010 13:46

Rails 3 bash aliases and .irbrc configs

13 comments

I’ve been upgrading some apps and gems to the latest and greatest Rails 3. I’ve taken the time to update my ~/.bash_aliases and ~/.irbrc files to be both Rails 3 and Rails 2.x compatible. See below for the code.

~/.bash_aliases

# rails 3 shortcut 'r'
alias r='rails'

# launching console/server
sc () {
  if [ -f ./script/rails ]; then 
    rails c $@
  else
    ./script/console $@
  fi
}

sg () {
  if [ -f ./script/rails ]; then
    rails g $@
  else
    ./script/generate $@
  fi
}

ss () {
  if [ -f ./script/rails ]; then 
    rails s $@
  else
    ./script/server $@
  fi
}

sspe () {
  if [ -f ./script/rails ]; then 
    sudo rails s -p80 $@
  else
    sudo ./script/server -p80 $@
  fi
}

# database migrate
alias rdbm='rake db:migrate'

# tests
alias rspec='rake spec'

# rails logs, tailing and cleaning
alias tdl='tail -f ./log/development.log'
alias ttl='tail -f ./log/test.log'
alias ctl='> ./log/test.log'
alias cdl='> ./log/development.log'

~/.irbrc

require 'rubygems' rescue nil
require 'wirble'
require 'hirb'
require 'ap'

# load wirble
Wirble.init
Wirble.colorize

# load hirb
Hirb::View.enable

IRB.conf[:AUTO_INDENT] = true

if ENV.include?('RAILS_ENV')
  if !Object.const_defined?('RAILS_DEFAULT_LOGGER')
    require 'logger'
    Object.const_set('RAILS_DEFAULT_LOGGER', Logger.new(STDOUT))
  end

  def sql(query)
    ActiveRecord::Base.connection.select_all(query)
  end
  
  if ENV['RAILS_ENV'] == 'test'
    require 'test/test_helper'
  end

# for rails 3
elsif defined?(Rails) && !Rails.env.nil?
  if Rails.logger
    Rails.logger =Logger.new(STDOUT)
    ActiveRecord::Base.logger = Rails.logger
  end
  if Rails.env == 'test'
    require 'test/test_helper'
  end
else
  # nothing to do
end

# annotate column names of an AR model
def show(obj)
  y(obj.send("column_names"))
end

puts "> all systems are go wirble/hirb/ap/show <"

Note that if your’e using Bundler in your Rails app, AND use gems in your ~/.irbrc file AND attempt to start the Rails console; you’ll get errors/warnings on requiring them UNLESS you define them in your Gemfile. I use a ‘development’ group in my Gemfile for these, like so.

Gemfile

group :development do
  gem "wirble"
  gem "hirb"
  gem "awesome_print"
end

platforms :ruby_18 do
  gem 'ruby-debug'
end 

platforms :ruby_19 do
  gem 'ruby-debug19'
end
September 19, 2010 18:33

FAQtory update

no comments yet, post one now

Its been a while since I mentioned anything about the FAQtory app. Rest assured that, along with Bugle it’s been keeping me busy in my spare time!

To re-cap quickly, the FAQtory (pronounced Factory) is a simplified stackoverflow-like question and answer app. It allows you to Ask and Answer questions on any topic you like. The app is entirely content-focused and deliberately simple. A simple user voting mechanism ensures the most popular questions and most correct answers filter to the top.

Looking at the logs, it was last August when I kicked off the very first commit. After almost a year of on-again, off-again work and long periods of inactivity, last week finally saw it launch! In its current form the FAQtory is deployed for a single account, (pmfaqtory.com) – you can take a look here. Over time PMfaqtory intend to use FAQtory to build a resource of project management questions and answers.

This is a big milestone for me and with all features locked down for launch, I can concentrate on preparing for a general release. This will involve adding an account model, pricing options and payment gateway integration. I’ll also be moving the app to a new server stack and cloud based host.

Over the coming weeks I hope to talk more about some the design and technology decisions I have made so far. For a bit of fun, here is a replay of the Git log using the latest Gource visualisation engine (now with added ‘Bloom’ effect!)

music is 'Penguin' by the Books

Gource is now available on Mac Ports which is a much easier install than it used to be (again, compiling from source stills give errors on FTGL)

July 24, 2010 19:08

ActsAsTextcaptcha

1 comment

Announcing ActsAsTextcaptcha! It’s the first gem to be extracted from Bugle.

ActAsTextcaptcha

ActsAsTextcaptcha - pretending to be human just got tougher!

ActsAsTextcaptcha provides spam protection for your Rails models using logic questions from the excellent Text CAPTCHA web service (by Rob Tuley of Openknot).

To get started, grab an API key for your website and follow along with the instructions in the README.

You can also play around with a working demo on heroku.

The gem can be configured with your very own logic questions (to fall back on if the textcaptcha service is down) or as a replacement for the service. It also makes use of bcrypt encryption when storing the answers in your session (recommended if you’re using the default Rails CookieStore)

About the code

The gem contains two parts, a module for your ActiveRecord models, and a tiny helper method (spamify).

A call to spamify(@model) in your controller will query the Text CAPTCHA web service. A restful GET request is made with Net::HTTP and parsed using the standard XML::Parser. A spam_question is assigned to the model, and an array of possible answers are encrypted in the session.

validate_spam_answer() is called on @model.validate() and checks that the @model.spam_answer matches one of those possible answers in the session. This validation is only carried out on new records, i.e. never on edit, only on create. User’s attempted spam answers are not case-sensitive and have trailing/leading white-space removed.

BCrypt encryption is used to securely store the possible answers in your session. You must specify a valid bcrypt-salt and (computational) cost in your options. Without these options possible answers will be MD5-hashed only.

allowed?() and perform_spam_check?() are utility methods (that can be overridden in your model) They basically act as flags allowing you to control creation of new records, or whether the spam check should be carried out at all.

If an error occurs in loading or parsing the web service XML, ActsAsTextcaptcha will fall back to choose a random logic question defined in your options. Additionally, if you’d prefer not to use the service at all, you can omit the api_key from your options entirely.

If the web service fails or no-api key is specified AND no alternate questions are configured, the @model will not require spam checking and will pass as valid.

For more details on the code please check the documentation.

About the logic questions

Text CAPTCHA’s logic questions are aimed at a child’s age of 7, so they can be easily solved by all but the most cognitively impaired users. As they involve human logic, such questions cannot be solved by a robot. There are both advantages and disadvantages for using logic questions rather than image based captchas, find out more at Text CAPTCHA.Rob Tuley of Openknot

Finally, since things have changed so much over the years, i’ll be doing a refresher post on the state of play for creating/testing and releasing a Rails gem/plugin from scratch.

Rake Tasks

If you choose to install as a plugin, or are interested in the code, the following rake tasks are available;

  • rake spec (run the tests)
  • rake rcov (run tests showing coverage)
  • rake rdoc (generate docs)

Links

Who’s who?

Rails 3 preparation

no comments yet, post one now

Here’s a few links on Rails 3 I’ve collected in preparation for its release. I haven’t read these all through yet, or started using Rails 3 in any way, but I have a couple of production apps primed and waiting to be upgraded as soon as it launches.

Start with this one from RubyInside, 36 links to get you going

And finally some official release notes;

April 15, 2010 14:27

Bugle, FAQtory and other things

no comments yet, post one now

Just a quick update to explain what I’ve been working on lately.

Some years ago (lets say 3) I was approached by a long-time business pal. He had the idea of creating a very simple FAQ gathering web application. It would offer no other features, only questions and answers, and would provide a rich knowledge base for those working in the field of project management. Sound familiar ?

As the years passed and other projects, distractions got in the way, the FAQtory (as it was named) was put on hold. Then in September 2008, stackoverflow launched and soon after their 3rd party offering, stackexchange and following that an entire barnyard of clones; stacked, cnprog, soclone to name a few.

This was all a bit dis-heartening, given we had the idea some years ago and never jumped on it. Around Christmas last year, I picked up the FAQtory again, dusted it off and got to work in my spare time building it. Right now I am at the final stages of styling the UI and launching the first version as a sub site at pmfaqtory.com.

some notebook sketches made while designing The FAQtory

some notebook sketches made while designing the FAQtory

So with my attention on the FAQtory, work on Bugle has been put on hold. After the FAQtory launches however, i’ll be back developing on it again. Being a one-man show its hard to maintain a focus on more than one app at a time. Im encouraged by Rework and DHH’s concept of a little italian restaurant on the web

I’ve learnt a great deal from building these two apps and in many cases I’ve applied learning gained from experiences in one, to the other. I also have some valuable by-products I hope to share. I’ve already mentioned some on this blog and on github. So stay tuned, in future posts I’ll try to explain more of the decisions I’ve made while building each app, and hopefully get around to open sourcing more code.

March 18, 2010 18:09
← (k) prev | next (j) →