Editing file uploads with a Paperclip processor

8 comments

NOTE: Apparently there are issues with this code and the latest Paperclip gem (currently 2.3.4) – its down to the use of reprocess and this known issue i’m currently looking at a work around here’s a patch fix

I use Paperclip for pretty much all upload processing. Its flexible, fast and easily extendable. One particular feature that has cropped up (a couple of times now) – has been the ability to edit and update the contents of uploaded files. For example; editing css, html or javascript in a CMS. Something I’ve needed for Bugle

In the past I struggled getting this to work with Paperclip. You’ll find me rambling to myself in the mailing list almost a year ago. I figured the Shopify guys we’re doing this in their app, so it had to be possible.

One solution, was to simply read the file contents (on create) from the uploaded file into a database column. Then on future requests for the file, serve it virtually from the database; through a Rails controller/action responding with the appropriate content type and content data.

But, this meant the Rails app would be handling all the css/js requests in the CMS. I really wanted to serve these uploaded files from S3/Cloudfront making full use of Amazon’s CDN. So I set about building a Paperclip::Processor to store the file contents (in the database) on create then on update, update contents and re-upload the file again. To work with cache expiry in the CDN I could use the updated_on timestamp in the URL to the file.

Here’s most of the code below, i’ve also created a git repository with a working simple app. I’m using a RESTful UploadsController with an Upload model. The model has an Paperclip attachment (asset) and the file contents (for editable files) are stored in a TEXT column (‘asset_contents’ in the database).

Controller

Nothing crazy going on here, just straight forward RESTful controller logic (without a show action)

class UploadsController < ApplicationController                       
  
  def index
    @uploads = Upload.scoped
  end
  
  def new
    @upload = Upload.new
  end

  def edit
    @upload = Upload.find(params[:id])
  end

  def create
    @upload = Upload.new(params[:upload])    
    if @upload.save
      flash[:notice] = 'Upload was successfully created'
      redirect_to uploads_url
    else 
      render 'new'
    end
  end

  def update
    @upload = Upload.find(params[:id])    
    if @upload.editable? && @upload.update_attributes(params[:upload])
      flash[:notice] = 'Upload was successfully updated'
      redirect_to uploads_url
    else 
      render 'edit'
    end
  end 
  
  def destroy
    @upload = Upload.find(params[:id])    
    if @upload.destroy
      flash[:notice] = 'Upload was successfully deleted'
    end
    redirect_to uploads_url                          
  end
end

Model

Two things to notice here. I’m using lambda’s on the style and processor attributes. In both cases they check the content-type to see if the file is either editable or thumbnailable. If it is a thumbnailable image, I give it a thumbnail style and the thumbnail (default) Paperclip::Processor. For editable files, I give it a style for the original file only, and use the new FileContents Processor (see below). The style hash sets which database column will be used for storing the file contents, in this case it’s the ‘asset_contents’ attribute.

Second is the after_update hook. When thew Upload model gets saved, I want Paperclip to reprocess the asset again. This ensures that when the asset is saved on update the FileContents processor executes. The thumbnailable? and editable? methods let you decide what file types should be considered for processing.

class Upload < ActiveRecord::Base
                      
  after_update :reprocess

  has_attached_file :asset, :styles         => lambda { |a|
                                                 if a.instance.thumbnailable?
                                                   {:thumb => ["64x64#", :jpg]}
                                                 elsif a.instance.editable?
                                                   {:original => {:contents => 'asset_contents'}}
                                                 end
                                               },
                            :path           => "/:id/:style/:basename.:extension",
                            :storage        => :s3,     
                            :s3_credentials => "#{Rails.root}/config/s3.yml",
                            :bucket         => "paperclip-example-bucket-#{Rails.env}",
                            :processors     => lambda { |a|
                                                 if a.editable?
                                                   [:file_contents]
                                                 elsif a.thumbnailable?
                                                   [:thumbnail]
                                                 end
                                               }
  
  attr_protected :asset_file_name, :asset_content_type, :asset_size          
  
  validates_attachment_size     :asset, :less_than => 6.megabytes
  validates_attachment_presence :asset

  def editable?
    return false unless asset.content_type
    ['text/css', 'application/js', 'text/plain', 'text/x-json', 'application/json', 'application/javascript',
     'application/x-javascript', 'text/javascript', 'text/x-javascript', 'text/x-json',
     'text/html', 'application/xhtml', 'application/xml', 'text/xml', 'text/js'].join('').include?(asset.content_type)
  end
  
  def thumbnailable?
    return false unless asset.content_type
    ['image/jpeg', 'image/pjpeg', 'image/gif', 'image/png', 'image/x-png', 'image/jpg'].join('').include?(asset.content_type)
  end 
  
  private
  def reprocess
    asset.reprocess! if editable?
  end
end

FileContents Paperclip::Processor

This processor basically reads the uploaded file contents on create and sets the asset_contents attribute. On update, it creates a new Tempfile with its content from the asset_contents attribute and then returns this Tempfile for Paperclip uploading. Comments in the code below explain further, (place this file in lib/paperclip/file_contents.rb).

module Paperclip
  class FileContents < Processor
    
    def initialize file, options = {}, attachment = nil
      @file           = file
      @options        = options
      @instance       = attachment.instance
      @current_format = File.extname(attachment.instance.asset_file_name)
      @basename       = File.basename(@file.path, @current_format)
      @whiny          = options[:whiny].nil? ? true : options[:whiny]
    end

    def make
      begin
        # new record, set contents attribute by reading the attachment file
        if(@instance.new_record?)
          @file.rewind # move pointer back to start of file in case handled by other processors
          file_content = File.read(@file.path)
          @instance.send("#{@options[:contents]}=", file_content)
        else                                                     
          # existing record, set contents by reading contents attribute
          file_content = @instance.send(@options[:contents])
          # create new file with contents from model
          tmp = Tempfile.new([@basename, @current_format].compact.join("."))
          tmp << file_content
          tmp.flush 
          @file = tmp
        end         
                 
        @file
      rescue StandardError => e
        raise PaperclipError, "There was an error processing the file contents for #{@basename} - #{e}" if @whiny
      end
    end
  end
end

Views

The view code is simple, a new and edit form with a textarea for contents editing.

# uploads/new.html.erb
<%= form_for(:upload, :url => uploads_path,
                      :html => { :method => :post, :multipart => true }) do |f| %>               
  <input type="file" name="upload[asset]"> <%= f.submit 'upload', :disable_with => 'uploading ...' %>  
<% end %>

# uploads/edit.html.erb
<%= form_for @upload do |f| %>
  <%= f.text_area :asset_contents, :rows => 20, :cols => 100, :id => 'file_asset_contents' %>
  <p><%= f.submit 'Save changes', :disable_with => 'saving ...' %></p>
<% end -%>

# reference upload URL always with timestamp
<%= @upload.asset.url(:original, true) %>

Some gotchas

If you are using an Amazon S3 bucket, make sure you set it to be ‘world’ readable, so your uploaded files are publicly accessible. Also, the file_contents.rb processor should live in lib/paperclip/file_contents.rb. And for a Rails 3 add this to your load path, in config/application.rb

config.autoload_paths += %W(#{Rails.root}/lib)

I’ve been running this code with no issues in production for some time now. I should point out that I limit these editable uploads to ~3Mb-6Mb and you may have performance issues with larger files. Some solutions could be to use delayed_job (or something similar) to background process the task, and/or change the processor code to read/write one line at a time.

Further reading

October 25, 2010 19:23 by

8 comments so far

  • photo of Mario Chavez Mario Chavez Oct 30, 2010

    Hi;

    Do you have this sample running against Rails3? I’m asking because I’m trying to run Paperclip with a custom processor but I have no luck on that. By the way I’m using paperclip 2.3.5.

    Do you have by any change a suggestion on where to look?

  • photo of Matt Matt Oct 30, 2010

    Paperclip 2.3.5 just came out recently. I had the above code working with Rails 3, and Paperclip 2.3.4 (through this patch)

    The example app here is Rails 3, Paperclip 2.3.4 – If I get a chance soon, i’ll test things work with Paperclip 2.3.5 – I can’t think why not.

  • photo of Tom Tom Nov 16, 2011

    Thanks for this outstanding post! I’m a rails newbie and I’m building a site which needs to sense content type and process through various processors – paperclip or ffmpeg – then output various file types. I seriously struggled for days trying to debug my code and your post made my mistakes clear as day. Thanks for the help!

  • photo of Stephanie Stephanie Apr 29, 2012

    Hi Matthew

    Thanks for this very nice blog post – it really helped me get my head around custom processors for Paperclip.

    I was searching for your blog again to review some details, and I fell over this guy, who seems to have copied this post to the letter, with no attribution or traceback to your blog:

    http://railspro.blogspot.com/2011/02/editing-file-uploads-with-paperclip_4896.html

    Thought that you might want to know.

  • photo of Matt Matt Apr 29, 2012

    Ahhh, the internet, land of all that is sacred – Thanks for the heads up anyways

  • photo of Jonathan Jonathan Mar 29, 2013

    Thank you! Hopefully this is still current.

  • photo of Jonathan Jonathan Mar 29, 2013

    Okay, almost everything worked well. The only update is that you can no longer use the after_update callback, since this creates an infinite loop now with Paperclip (see https://github.com/thoughtbot/paperclip/issues/866 for example).

    Also, “File.extname(attachment.instance.asset_file_name)” could possibly be abstracted more for cases where the paperclip attribute is not named “asset”.

    Still, this was a huge help. Saved me tons of time.

    Thanks again!

  • photo of Jonathan Jonathan Mar 29, 2013

    Sorry for the numerous comments – just thought I would add some updates. I ran into a problem where my CSS file content-type was getting set to “text/x-c” when updated. I believe it’s related to this change: https://github.com/thoughtbot/paperclip/commit/48736ce523b0dbc596dd4bbe0b9d07f264493e5b .

    To resolve this, I swapped out:

    Tempfile.new([@basename, @current_format].compact.join(“.”))

    with

    Tempfile.new([@basename, @current_format])

    Thanks!

    Jonathan

Leave a comment