articles tagged with bugle

Editing file uploads with a Paperclip processor

8 comments

NOTE: Apparently there are issues with this code and the latest Paperclip gem (currently 2.3.4) – its down to the use of reprocess and this known issue i’m currently looking at a work around here’s a patch fix

I use Paperclip for pretty much all upload processing. Its flexible, fast and easily extendable. One particular feature that has cropped up (a couple of times now) – has been the ability to edit and update the contents of uploaded files. For example; editing css, html or javascript in a CMS. Something I’ve needed for Bugle

In the past I struggled getting this to work with Paperclip. You’ll find me rambling to myself in the mailing list almost a year ago. I figured the Shopify guys we’re doing this in their app, so it had to be possible.

One solution, was to simply read the file contents (on create) from the uploaded file into a database column. Then on future requests for the file, serve it virtually from the database; through a Rails controller/action responding with the appropriate content type and content data.

But, this meant the Rails app would be handling all the css/js requests in the CMS. I really wanted to serve these uploaded files from S3/Cloudfront making full use of Amazon’s CDN. So I set about building a Paperclip::Processor to store the file contents (in the database) on create then on update, update contents and re-upload the file again. To work with cache expiry in the CDN I could use the updated_on timestamp in the URL to the file.

Here’s most of the code below, i’ve also created a git repository with a working simple app. I’m using a RESTful UploadsController with an Upload model. The model has an Paperclip attachment (asset) and the file contents (for editable files) are stored in a TEXT column (‘asset_contents’ in the database).

Controller

Nothing crazy going on here, just straight forward RESTful controller logic (without a show action)

class UploadsController < ApplicationController                       
  
  def index
    @uploads = Upload.scoped
  end
  
  def new
    @upload = Upload.new
  end

  def edit
    @upload = Upload.find(params[:id])
  end

  def create
    @upload = Upload.new(params[:upload])    
    if @upload.save
      flash[:notice] = 'Upload was successfully created'
      redirect_to uploads_url
    else 
      render 'new'
    end
  end

  def update
    @upload = Upload.find(params[:id])    
    if @upload.editable? && @upload.update_attributes(params[:upload])
      flash[:notice] = 'Upload was successfully updated'
      redirect_to uploads_url
    else 
      render 'edit'
    end
  end 
  
  def destroy
    @upload = Upload.find(params[:id])    
    if @upload.destroy
      flash[:notice] = 'Upload was successfully deleted'
    end
    redirect_to uploads_url                          
  end
end

Model

Two things to notice here. I’m using lambda’s on the style and processor attributes. In both cases they check the content-type to see if the file is either editable or thumbnailable. If it is a thumbnailable image, I give it a thumbnail style and the thumbnail (default) Paperclip::Processor. For editable files, I give it a style for the original file only, and use the new FileContents Processor (see below). The style hash sets which database column will be used for storing the file contents, in this case it’s the ‘asset_contents’ attribute.

Second is the after_update hook. When thew Upload model gets saved, I want Paperclip to reprocess the asset again. This ensures that when the asset is saved on update the FileContents processor executes. The thumbnailable? and editable? methods let you decide what file types should be considered for processing.

class Upload < ActiveRecord::Base
                      
  after_update :reprocess

  has_attached_file :asset, :styles         => lambda { |a|
                                                 if a.instance.thumbnailable?
                                                   {:thumb => ["64x64#", :jpg]}
                                                 elsif a.instance.editable?
                                                   {:original => {:contents => 'asset_contents'}}
                                                 end
                                               },
                            :path           => "/:id/:style/:basename.:extension",
                            :storage        => :s3,     
                            :s3_credentials => "#{Rails.root}/config/s3.yml",
                            :bucket         => "paperclip-example-bucket-#{Rails.env}",
                            :processors     => lambda { |a|
                                                 if a.editable?
                                                   [:file_contents]
                                                 elsif a.thumbnailable?
                                                   [:thumbnail]
                                                 end
                                               }
  
  attr_protected :asset_file_name, :asset_content_type, :asset_size          
  
  validates_attachment_size     :asset, :less_than => 6.megabytes
  validates_attachment_presence :asset

  def editable?
    return false unless asset.content_type
    ['text/css', 'application/js', 'text/plain', 'text/x-json', 'application/json', 'application/javascript',
     'application/x-javascript', 'text/javascript', 'text/x-javascript', 'text/x-json',
     'text/html', 'application/xhtml', 'application/xml', 'text/xml', 'text/js'].join('').include?(asset.content_type)
  end
  
  def thumbnailable?
    return false unless asset.content_type
    ['image/jpeg', 'image/pjpeg', 'image/gif', 'image/png', 'image/x-png', 'image/jpg'].join('').include?(asset.content_type)
  end 
  
  private
  def reprocess
    asset.reprocess! if editable?
  end
end

FileContents Paperclip::Processor

This processor basically reads the uploaded file contents on create and sets the asset_contents attribute. On update, it creates a new Tempfile with its content from the asset_contents attribute and then returns this Tempfile for Paperclip uploading. Comments in the code below explain further, (place this file in lib/paperclip/file_contents.rb).

module Paperclip
  class FileContents < Processor
    
    def initialize file, options = {}, attachment = nil
      @file           = file
      @options        = options
      @instance       = attachment.instance
      @current_format = File.extname(attachment.instance.asset_file_name)
      @basename       = File.basename(@file.path, @current_format)
      @whiny          = options[:whiny].nil? ? true : options[:whiny]
    end

    def make
      begin
        # new record, set contents attribute by reading the attachment file
        if(@instance.new_record?)
          @file.rewind # move pointer back to start of file in case handled by other processors
          file_content = File.read(@file.path)
          @instance.send("#{@options[:contents]}=", file_content)
        else                                                     
          # existing record, set contents by reading contents attribute
          file_content = @instance.send(@options[:contents])
          # create new file with contents from model
          tmp = Tempfile.new([@basename, @current_format].compact.join("."))
          tmp << file_content
          tmp.flush 
          @file = tmp
        end         
                 
        @file
      rescue StandardError => e
        raise PaperclipError, "There was an error processing the file contents for #{@basename} - #{e}" if @whiny
      end
    end
  end
end

Views

The view code is simple, a new and edit form with a textarea for contents editing.

# uploads/new.html.erb
<%= form_for(:upload, :url => uploads_path,
                      :html => { :method => :post, :multipart => true }) do |f| %>               
  <input type="file" name="upload[asset]"> <%= f.submit 'upload', :disable_with => 'uploading ...' %>  
<% end %>

# uploads/edit.html.erb
<%= form_for @upload do |f| %>
  <%= f.text_area :asset_contents, :rows => 20, :cols => 100, :id => 'file_asset_contents' %>
  <p><%= f.submit 'Save changes', :disable_with => 'saving ...' %></p>
<% end -%>

# reference upload URL always with timestamp
<%= @upload.asset.url(:original, true) %>

Some gotchas

If you are using an Amazon S3 bucket, make sure you set it to be ‘world’ readable, so your uploaded files are publicly accessible. Also, the file_contents.rb processor should live in lib/paperclip/file_contents.rb. And for a Rails 3 add this to your load path, in config/application.rb

config.autoload_paths += %W(#{Rails.root}/lib)

I’ve been running this code with no issues in production for some time now. I should point out that I limit these editable uploads to ~3Mb-6Mb and you may have performance issues with larger files. Some solutions could be to use delayed_job (or something similar) to background process the task, and/or change the processor code to read/write one line at a time.

Further reading

October 25, 2010 19:23 by

ActsAsTextcaptcha

1 comment

Announcing ActsAsTextcaptcha! It’s the first gem to be extracted from Bugle.

ActAsTextcaptcha

ActsAsTextcaptcha - pretending to be human just got tougher!

ActsAsTextcaptcha provides spam protection for your Rails models using logic questions from the excellent Text CAPTCHA web service (by Rob Tuley of Openknot).

To get started, grab an API key for your website and follow along with the instructions in the README.

You can also play around with a working demo on heroku.

The gem can be configured with your very own logic questions (to fall back on if the textcaptcha service is down) or as a replacement for the service. It also makes use of bcrypt encryption when storing the answers in your session (recommended if you’re using the default Rails CookieStore)

About the code

The gem contains two parts, a module for your ActiveRecord models, and a tiny helper method (spamify).

A call to spamify(@model) in your controller will query the Text CAPTCHA web service. A restful GET request is made with Net::HTTP and parsed using the standard XML::Parser. A spam_question is assigned to the model, and an array of possible answers are encrypted in the session.

validate_spam_answer() is called on @model.validate() and checks that the @model.spam_answer matches one of those possible answers in the session. This validation is only carried out on new records, i.e. never on edit, only on create. User’s attempted spam answers are not case-sensitive and have trailing/leading white-space removed.

BCrypt encryption is used to securely store the possible answers in your session. You must specify a valid bcrypt-salt and (computational) cost in your options. Without these options possible answers will be MD5-hashed only.

allowed?() and perform_spam_check?() are utility methods (that can be overridden in your model) They basically act as flags allowing you to control creation of new records, or whether the spam check should be carried out at all.

If an error occurs in loading or parsing the web service XML, ActsAsTextcaptcha will fall back to choose a random logic question defined in your options. Additionally, if you’d prefer not to use the service at all, you can omit the api_key from your options entirely.

If the web service fails or no-api key is specified AND no alternate questions are configured, the @model will not require spam checking and will pass as valid.

For more details on the code please check the documentation.

About the logic questions

Text CAPTCHA’s logic questions are aimed at a child’s age of 7, so they can be easily solved by all but the most cognitively impaired users. As they involve human logic, such questions cannot be solved by a robot. There are both advantages and disadvantages for using logic questions rather than image based captchas, find out more at Text CAPTCHA.Rob Tuley of Openknot

Finally, since things have changed so much over the years, i’ll be doing a refresher post on the state of play for creating/testing and releasing a Rails gem/plugin from scratch.

Rake Tasks

If you choose to install as a plugin, or are interested in the code, the following rake tasks are available;

  • rake spec (run the tests)
  • rake rcov (run tests showing coverage)
  • rake rdoc (generate docs)

Links

Who’s who?

Opinions matter

no comments yet, post one now

Be opinionated. In software its important. I’d go as far as saying it can completely make or break your app. It’s something that 37signals have been advocating for a long time, and something they practice daily in saying ‘No’. In fact in both Rework and Getting Real it is clearly stated that saying ‘No’ should be the default answer when considering any new feature (including their own!) I couldn’t agree more.

What happens when you don’t say no? It leads directly to feature bloat or in the long-term a big ball of mud. And if your business is beholden to clients who can throw financial weight around, then your’e really in trouble. You can end up second guessing their needs or prospecting new features that will attract them. This can be more true in b2b products where I think it can be much harder for people to say ‘No’ by default.

Apple take sides in a very obvious way. Rather than be all things to all people, they continually make game changing decisions that their competitors probably haven’t even considered. No DVD-drive in the Apple Air, loosing the conventional right-mouse button, more often than not they are constantly removing and simplifying their products. Look at everything thats not built into the iPad, you’ll find a long list of features if you compare against traditional touchpad devices.

All this creates division (and a buzz) amongst fans and critics. But have you ever heard an outcry over the latest updates to Microsoft Office? People just don’t care when an already bloated app tries to be all things to all people (and yes, I’m writing this post in Writeroom)

In building Bugle I have made lots of decisions along the way in code, the interface and even the server stack. Drawing on my own experiences and motivations has helped to keep things simple. Im not doing competitor comparisons or even looking at what people consider normal steps in the processes. I’m just defining the problems a user faces and trying to offer the simplest solution that will meet their needs.

Of course opinions matter in blog posts too, so what do you think, agree or disagree?

April 01, 2010 21:04 by

Bugle, FAQtory and other things

no comments yet, post one now

Just a quick update to explain what I’ve been working on lately.

Some years ago (lets say 3) I was approached by a long-time business pal. He had the idea of creating a very simple FAQ gathering web application. It would offer no other features, only questions and answers, and would provide a rich knowledge base for those working in the field of project management. Sound familiar ?

As the years passed and other projects, distractions got in the way, the FAQtory (as it was named) was put on hold. Then in September 2008, stackoverflow launched and soon after their 3rd party offering, stackexchange and following that an entire barnyard of clones; stacked, cnprog, soclone to name a few.

This was all a bit dis-heartening, given we had the idea some years ago and never jumped on it. Around Christmas last year, I picked up the FAQtory again, dusted it off and got to work in my spare time building it. Right now I am at the final stages of styling the UI and launching the first version as a sub site at pmfaqtory.com.

some notebook sketches made while designing The FAQtory

some notebook sketches made while designing the FAQtory

So with my attention on the FAQtory, work on Bugle has been put on hold. After the FAQtory launches however, i’ll be back developing on it again. Being a one-man show its hard to maintain a focus on more than one app at a time. Im encouraged by Rework and DHH’s concept of a little italian restaurant on the web

I’ve learnt a great deal from building these two apps and in many cases I’ve applied learning gained from experiences in one, to the other. I also have some valuable by-products I hope to share. I’ve already mentioned some on this blog and on github. So stay tuned, in future posts I’ll try to explain more of the decisions I’ve made while building each app, and hopefully get around to open sourcing more code.

March 18, 2010 18:09 by

Fake it till you make it!

2 comments

I’d like to share another handy little rake task that helps you fill up your Rails application with realistic ‘fake’ data. Here it is

Using the very excellent Faker gem by Benjamin Curtis this task can be configured to populate your models with random amounts of fake information. This can a be a real time-saver for load testing, preparing demos/screencasts, or just filling up your pages with realistic data so you can get to work on your views.

The classic 'comb-over', a traditional fake.

The classic 'comb-over', a traditional fake.

The task includes the following features;

  • Assign relationships between models with a random object picker
  • A random tag picker, for populating tag lists
  • Fake out your app at different different sizes; tiny, small, medium and large
  • Clean task, to clear stuff out before you start your faking
  • Disables mail sending before faking out, since before/after and other code hooks in your app might trigger sending mails
  • Helper to fake out a randomly picked time object within the last year
  • Prompts to ask before deleting any data on cleaning task.
  • Summary report after faking out, explaining exactly how many objects were created

Simply configure it (see code comments for help on this), drop it into your Rails lib/tasks folder and run like so;

sudo gem install faker # if you haven't got the gem already OR 
sudo gem install ffaker # use the faster faker gem from Emmanuel Oga 
rake fakeout:small # or tiny / medium / large

Cleaning all ...
Faking it ... (small)
  * Users: 53
  * Questions: 53
  * Answers: 38
  * Tags: 38
  * Taggings: 160
Done, I Faked it!

Along with this subdomain script from last month, I have pushed this code to github as a gist, so you can track it and grab updates to it whenever. The example there shows the task configured for a small Question & Answer app I’ve been working on.

And yes, I have started drawing a bit again, maybe it’ll make these posts more interesting to look at until I properly build this blog out.

Update – I have amended the task definitions to take a single argument; no_prompt. This turns off the confirm prompt and is useful for non-interactive running, e.g. with heroku rake commands for instance.

Update – This rake task is also fully compatible with the faster ffaker gem by Emmanuel Oga

February 07, 2010 15:31 by

Cyclemeter & Other Stats

no comments yet, post one now

If you follow my tweets, you might have noticed I occasionally use the excellent Cyclemeter (iPhone app) for collecting stats. What you might not have noticed is that you can actually shout support (or more likely abuse) at me while I’m cycling in and out of work!

The app uses text-to-speech to convert your @hiddenloop replies (pushed to the iphone through the app) and plays them over whatever music I’m listening to. Clever stuff! So next time you see a tweet like this holla back!

If you’re big into tracking exercise stats, I would recommend Cyclemeter over the other apps out there right now. I’ve tried MapMyRide, MapMyRun (and some others) and none work as well or are as simple to use. CycleMeter can also be used for running, but I prefer to use a Nike+ kit with an iPod nano (rather than lugging the iPhone around with me)

I’m working on a new feed & push importer service (with web-hooks) for Bugle that will allow any valid feed data to be displayed/updated within your site or blog. First examples of this will probably be put to use here, with my tweets, run data, cycle stats etc.

Cleaning your Rails routes

no comments yet, post one now

After watching the excellent Rails Best Practices presentation by Wen-Tien Chang, I took the opportunity to perform this optimization on my apps. Basically removing the default Rails route mappings from routes.rb. If your Rails app is RESTful then you should have all your resource endpoints defined in routes; and be using the Rails url/path view helpers in your templates. If this is the case you can remove these default mappings at the base of your routes file;

# these can go!
map.connect ':controller/:action/:id'
map.connect ':controller/:action/:id.:format'

Doing this on my smaller Rails apps caused no problems, but in Bugle (a bigger app) I came across the following gotchas;

Look for urls or paths in your code that are defined with { :controller => :foo, :action => :bar } options, these should be replaced for the equivalent Rails helper for the controller – rake routes is your friend in figuring out what to use; eg.

url_for(:controller => 'blogs', :action => 'show' :id => @blog.id}
# should simply change to
blog_url(@blog) 
# or
blog_path(@blog)

Calls to link_to should again use the same urls/path helpers as above – but in some instances this became a “Route not found problem”;

link_to 'comment website', @comment.website
# where @comment.website is a String e.g. http://hallo.com
# instead ditch link_to and use simple HTML tags

Happily I was able to delete a large amount of unnecessary routing specs, that were essentially testing the behavior of the 7 standard RESTFul Rails routes. Rails 3 has a completely revamped routing system (see here for details), so making optimizations and cleaning up routes.rb prior to it’s release is advisable.

Related Links

January 21, 2010 12:18 by

A timesaving subdomain rake task

1 comment

A time saving rake task for adding or updating your local /etc/hosts file. I created this for Bugle, allowing me to quickly configure my development machine hosts file with subdomains used in the app. See the inline comments for an explanation.

Has the added feature of an array of default hosts to always add when you run it. Work is done on a tmp file then, then sudo copied on top of /etc/hosts. It exists as a gist on github so i’ll be sure to post any updates to it there.

# subdomains.rake (in /lib/tasks)
namespace :subdomains do

  desc "adds the necessary hosts to your /etc/hosts file from current subdomains in your application"
  task :setup => :environment do
    # NOTE: default_hosts is used as a locator for the line to update in /etc/hosts
    tmp_file, changed = '/tmp/etc_hosts_copy', false
    default_hosts, hosts    = %w(blog.local emptyblog.blog.local), []

    # find all the subdomains used in your app (push to hosts array) - modify this to suit your app
    Blog.find(:all).each { |blog| hosts << "#{blog.subdomain}.blog.local" unless blog.subdomain.blank? }
    
    # build hosts line to add/edit
    host_line = "127.0.0.1 " + hosts.sort.unshift(default_hosts).join(' ')

    # work with a copied hosts file in tmp
    %x[cp /etc/hosts #{tmp_file}]

    file = File.new(tmp_file)
    lines = file.readlines
    lines.each do |line|
      changed = true if line.gsub!(/^127.0.0.1 #{Regexp.escape(default_hosts.join(' '))}.+$/, host_line)
    end

    # add line, if no line found for update
    lines += ["\n", host_line, "\n"] unless changed
    file = File.new(tmp_file,'w')
    lines.each { |line| file.write(line) }
    file.close

    # copy hosts file from tmp - may ask for sudo password
    %x[sudo -p "Password:" cp #{tmp_file} /etc/hosts]
    
    # explain what happened
    puts "\nAdded the following domains:"
    hosts.each { |host| puts "* http://#{host}" }
    puts "\nAlso added defaults:"
    default_hosts.each { |default| puts "* http://#{default}" }
    puts "\n"
  end
end

To run simply type

rake subdomains:setup
January 14, 2010 16:53 by

NGINX upload module with Paperclip (on Rails)

19 comments

Over the Christmas holidays I started looking at integrating the nginx upload module into Bugle.

The nginx upload module has been around for a while, but I couldn’t find anything to explain exactly what went on with the params and the best way to integrate it with the Paperclip gem in Rails (which Bugle uses for all upload handling). As I worked with it I found a few caveats along the way.

Why bother?

Without the module, your Rails app will receive the raw uploaded data, parsing its entire contents before it can be used. For large uploads this can be quite slow, since Ruby is the work horse throughout.

With this module, parsing the file happens in C through nginx and before your Ruby application gets it. The module puts the parsed file into a tmp directory and strips all the multipart params out of the POST body, replacing it with params you can use (in Rails) to get the name and location of the file on disk. So by the time the request hits your application, all the expensive parsing has been done and the file is ready to be used by your app. Basically hard work is moved from Ruby to C.

Compiling Nginx to include the module

To install the module you need to build nginx from source and pass it the upload module source directory as an argument. Since I run Bugle on a live production machine I wanted to work with things locally first. I began by setting up my local (OSX) box to match the production stack. I currently use nginx with Passenger and Ruby Enterprise Edition.

First download and untar both the nginx and upload module sources. Then build using the following commands (these worked on both OSX and my Ubuntu production server)

sudo /opt/ruby-enterprise-1.8.7-20090928/bin/passenger-install-nginx-module --nginx-source-dir=<path to nginx sources> --extra-configure-flags=--add-module='<path to upload module sources>'

Or if you’re just building nginx from source (without using the handy passenger installer) go with this;

cd <path to nginx sources>
./configure --add-module=<path to upload module sources>
make
make install

Don’t worry about any existing nginx.conf files you have or nginx vhosts etc. They will be unaffected after recompiling.

Configuring Nginx

The next step is to configure nginx to use the module. In Bugle uploads are normally sent via a POST request to the uploads controller using a restful URL that can be one of;

POST /admin/blogs/:blog_id/uploads
or
POST /admin/themes/:theme_id/uploads

These hit the ‘uploads’ controller, ‘create’ action. I wanted to keep the same restful URLs so I tried the following regex in the nginx config.

location ~* admin\/(themes|blogs)\/([0-9]+)\/uploads { }

While this did work in recognising the URL, the upload module wouldn’t work with it. In the end I opted to use a new defined url for faster uploads, here is the route for it in Rails. You may have a simpler upload URL controller action making this unnecessary.

map.connect 'admin/uploads/fast_upload', :controller => 'admin/uploads', 
                                        :action     => 'create', 
                                        :conditions => { :method => :post }

So the modified nginx server config becomes;

server {
 listen   80;
 server_name bugleblogs.com *.bugleblogs.com;
 
 # ...
 # somewhere inside your server block
 # ...
 
 # Match this location for the upload module
 location /admin/uploads/fast_upload {
   # pass request body to here
   upload_pass @fast_upload_endpoint;

   # Store files to this directory
   # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist    
   # i.e. make sure to create /u/apps/bugle/shared/uploads_tmp/0 /u/apps/bugle/shared/uploads_tmp/1 etc.
   upload_store /u/apps/bugle/shared/uploads_tmp 1;

   # set permissions on the uploaded files
   upload_store_access user:rw group:rw all:r;

   # Set specified fields in request body
   # this puts the original filename, new path+filename and content type in the requests params
   upload_set_form_field upload[fast_asset][original_name] "$upload_file_name";
   upload_set_form_field upload[fast_asset][content_type] "$upload_content_type";
   upload_set_form_field upload[fast_asset][filepath] "$upload_tmp_path";

   upload_pass_form_field "^theme_id$|^blog_id$|^authenticity_token$|^format$";
   upload_cleanup 400 404 499 500-505;
 }
 
 location @fast_upload_endpoint {
   passenger_enabled on;  # or this could be your mongrel/thin backend
 } 
}
  • After processing, the upload module puts the file in one of 10 tmp directories, these should be already created and accessible by your Rails app, since I am using Capistrano, I’ve chosen the shared/ folder to house the tmp files (you’d risk loosing a file mid-deploy if you chose somewhere in the RAILS_APP current/ directory)
  • The upload_pass_form_field directive preserves any params that match the regex (we don’t want the module to strip the following params: format, blog_id, theme_id or authenticity_token)
  • The upload_pass directive sets what should handle the request after the upload module has finished with the file (we want to handle it using Rails, through passenger, so a @fast_upload_endpoint location is defined)
  • The upload_set_form_field directives are used to specify the params that Rails will now receive, this will give;
    • params[‘upload’][‘fast_asset’][‘original_name’]
    • params[‘upload’][‘fast_asset’][‘content_type’]
    • params[‘upload’][‘fast_asset’][‘filepath’]

At this point its worth testing the app. Perform an upload and check that nginx is sending these params to your controller action. For more info here is a complete guide to all the module directives.

Working with Paperclip and Rails

Finally I needed to modify Rails to make use of these new params. In Bugle the upload module has_attached_file :asset using Paperclip. One problem is that new file in the tmp/ directory exists with a hashed meaningless filename, so simply passing this file to self.asset will not work for Paperclip processing. It needs to have the original filename and content_type. Fortunately we have those in the new params too. So the new fast_asset= method shifts and renames the file into a sub tmp directory (which gets cleaned on the after_create filter). All this seems a little convoluted, but I couldn’t see any other way to do this, without perhaps modifying the Paperclip internals. If anyone has any suggestions around this let me know in the comments.

class Upload < ActiveRecord::Base
  has_attached_file :asset, :styles => {:thumb => ["64x64#", :jpg]},
                            :url    => ":base_url/:resource/:styles_folder:basename:style_filename.:extension",
                            :path   => "public/u/:resource/:styles_folder:basename:style_filename.:extension" 
        
  attr_accessor :tmp_upload_dir
  after_create  :clean_tmp_upload_dir
  
  # handle new param
  def fast_asset=(file)
    if file && file.respond_to?('[]')
      self.tmp_upload_dir = "#{file['filepath']}_1"
      tmp_file_path = "#{self.tmp_upload_dir}/#{file['original_name']}"
      FileUtils.mkdir_p(self.tmp_upload_dir)
      FileUtils.mv(file['filepath'], tmp_file_path)
      self.asset = File.new(tmp_file_path)
    end
  end    
  
  private
  # clean tmp directory used in handling new param
  def clean_tmp_upload_dir
    FileUtils.rm_r(tmp_upload_dir) if self.tmp_upload_dir && File.directory?(self.tmp_upload_dir)
  end                     
end

For completeness here is the regular controller action;

def create
  # ...
  @upload = @resource.uploads.build(params[:upload])
  if @upload.save
  # ...
end

Go Go Uploads!

Thats it, you should now have much faster uploads through nginx! To see the improvement try uploading a 50Mb+ file with/without the module. In a future series of posts I will be conducting a complete walkthrough of the uploader I have built for Bugle. End to end from the browser, to Rails and the actual server configuration.

Related Links

Gource on OSX (Snow Leopard)

9 comments

Some months ago I played around with code_swarm by Michael Ogawa – partly for fun and partly to see what all the fuss was about with the Processing framework (something I have yet to really investigate). Last week I came across Gource another source code visualisation tool this time using 3D rendering.

Software projects are displayed by Gource as an animated tree with the root directory of the project at its centre. Directories appear as branches with files as leaves. Developers can be seen working on the tree at the times they contributed to the project.

Some recent commits to the project fixed build issues on OSX. Despite these fixes I still had trouble compiling. So in summary here’s what I did to get it working. Note that I did resort to installing mac ports (something I’d rather NOT do) – after many attempts to download and manually compile the prerequisites (FTGL 2.1.3~rc5-2 kept giving me problems)

# get mac ports
http://www.macports.org/install.php

# get all the ports you need for gource
sudo port install pcre libsdl libsdl_image ftgl

# get and build Gource from github and follow the instructions in INSTALL
git clone git://github.com/acaudwell/Gource.git
cd Gource
autoreconf -f -i
./configure && make && sudo make install

# navigate to your project directory and type;
gource

# have a look at all the options
man gource

# here are the settings I used (to get a large user icon and change the speed/size
gource ./ -s 0.5 -b 000000 --user-image-dir ~/images/avatars/ --user-scale 2.0 -800x600

# video it in h264 using ffmpeg, first install ffmpeg via mac ports (with codecs)
sudo port install ffmpeg +gpl +lame +x264 +xvid

# pipe PPM images to ffmpeg to generate a h264 encoded movie
gource ./ -s 0.5 -b 000000 --user-image-dir ~/images/avatars/ --user-scale 2.0 -800x600 --output-ppm-stream - | ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -vpre hq -crf 28 -threads 0 bugle.mp4

# finally, if you want to add some audio to the video (from an mp3)
ffmpeg -i audio.mp3 -i bugle.mp4 -vcodec libx264 -vpre hq -crf 28 -threads 0 bugle-with-audio.mp4
(this output will be the length of the video/audio track, whichever is longer)

# phew! - now go here to see how to remove macports when you're done :)
http://trac.macports.org/wiki/FAQ#uninstall

Bugle did exist (for a time) as an open source project on github before I moved it to be privately hosted. So here is the result; Bugle’s git log from April ’09 to the present parsed through Gource, with just one committer (me).

In summary you can see real bursts of activity at the start followed by some long periods of inactivity, and coming toward the present date renewed development and work going on. Gource is more impressive when visualising big projects with multiple committers over long periods, like the history of Git itself for example.

← (k) prev | next (j) →