articles tagged with guide

Part4. Having a Mint

8 comments

Following on from part 3 you should now have your Rails application nicely hosted and deployed using Capistrano from your SVN repository. All good – but what happens when you want to serve another public web folder into the mix e.g. in public/myfolder, but you DON’T want it under version control ? Why you ask ?

Well lets take this example further, what if this folder contained a copy of the Mint stats package in public/mint – i.e. PHP code, that you wanted to execute and run. Since new versions of Mint are released fequently, and i’m often adding and removing ‘Peppers’ (read Mint Plugins), I see no need to put Mint under version control. Purists might argue differently, but I dont collect stats on my local dev box either so it makes no sense to have it there after a checkout.

Assuming your convinced I’ll start off getting Mint up and running on the server, with PHP through Lighttpd and stats being recorded. Then i’ll add some extra functions to the Capistrano deployment, allowing us to deploy ‘around’ this folder on the public server.

Configuring Lighttpd with PHP and FastCGI

First ensure your server already has PHP installed and is configured with the FastCGI module;

$> php -v
PHP 5.1.4 (cgi-fcgi) (built: Aug  2 2006 23:53:20)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies

If you dont see the (cgi-fcgi) part, then you’ll need to re-configure PHP with the FastCGI module. There are instructions for installing PHP and configuring modules at PHP.net – and elsewhere on the web – Google is your friend.

Now configure your config/lighttpd.conf file (do this locally on your dev machine, checkin to SVN and redeploy – or just edit on the server to get things working then apply the same changes locally, checkin & deploy), in config/lighttpd.conf;

server.modules = ( "mod_rewrite", "mod_accesslog", "mod_fastcgi", "mod_compress", "mod_expire", "mod_proxy" )

fastcgi.server = (".php" => ("127.0.0.1" =>
  ("socket" => CWD + "/tmp/sockets/php.socket",
    "bin-path" => "/usr/local/bin/php",
    "bin-environment" => ("PHP_FCGI_CHILDREN" => "1", "PHP_FCGI_MAX_REQUESTS" => "5000")
  ))
)

# lighttpd proxy server (pound) - no proxy for mint please
$HTTP["url"] !~ "^/mint/" {
  proxy.balance = "fair"
  proxy.server  = ( "/" => ( ( "host" => "127.0.0.1", "port" => 6000 ) ) )
}

Things to note here are;

  • Add mod_fastcgi to your server.modules if its not there already.
  • fastcgi.server is configured with your server’s PHP bin path, and a tmp socket (which can be anywhere).
  • We DON’T want to proxy requests for Mint through Pound to Mongrel – since this is a PHP app, we just want Lighttpd to deal with it using FastCGI – hence the need for the if statement on the pound proxy configuration.

Setting up Mint (e.g. in Typo)

First grab your licensed copy of Mint for your domain – and drop the mint/ folder into current/public/mint to the main Rails directory on your server. Next add the following url.rewrite to your lighttpd configuration;

  # configure for mint access
  url.rewrite = ("^/mint/$" => "/mint/index.php", "^/mint/\?(.*)" => "/mint/index.php?$1")

This url.rewrite is nessecary to ensure that Lighttpd doesnt treat files or folders under /mint/ as Rails specific. Again this should also be added in your local copy, and checked in to SVN. DON’T deploy with Capistrano just yet – because doing so will check out a new release and archive the existing current/ folder (and hence remove our /current/public/mint) – You’ll also want to add the following javascript in the header of any page where stats should be reported;

<script src="/mint/?js" type="text/javascript"></script>

Setting up Capistrano

In order to keep using Capistrano to deploy to your server with mint in the current/public/mint folder – we need to deploy around it. There are probably better ways to do this – in the Capistrano deployment recipe file (config/deploy.rb) – I added two action functions; one occurs before deployment starts, the other after. The functions basically move the mint/ folder out of the way (to the top level shared folder) while Capistrano does its stuff. So in config/deploy.rb;

  # executed before deployment
  task :before_deploy, :roles => [:web, :app] do
    # copy the mint/ and files/ folders to holding area in shared/
    puts "before deploy ---> copy mint and files to shared from current"
    run "sudo mv #{deploy_to}/#{current_dir}/public/mint  #{shared_dir}/mint"
    run "sudo mv #{deploy_to}/#{current_dir}/public/files  #{shared_dir}/files"
  end

  # executed after deployment
  task :after_deploy, :roles => [:web, :app] do
    # copy the mint/ and files/ folders back from holding area in shared/
    puts "after deploy ---> copy mint and files from shared to current"
    run "sudo mv #{shared_dir}/mint #{deploy_to}/#{current_dir}/public/mint"
    run "sudo mv #{shared_dir}/files #{deploy_to}/#{current_dir}/public/files"
  end

You’ll also see i’m doing the same thing for a current/public/files folder – This folder is used by Typo for uploaded files for blog entries. Without these actions in place, each Capistrano deploy would clear out the files/ folder on your server.

Trying it out

Check the changes in, make sure the mint folder is on your server (and correctly configured) and run a new Capistrano deploy. During this you should see the before and after tasks running (you may will be asked for a password to sudo). You should then see your copy of mint up and running, like so; /mint/

Thus ends the mini-guide; Any suggestions, comments or questions are appreciated. Normal useless entries will resume here as of today.

References

Part3. Lighttpd, Pound and Mongrel with Rails

no comments yet, post one now

This (king-kong long) post is a continuation of a mini-guide that has already covered putting your Rails app under SVN version control, and deploying it with Capistrano. Now to explain how to serve it all up on a layered and scalable web server stack using Lighttpd, Pound and Mongrel. But first some obvious questions;

  • What is Lighttpd ?
    (aka. Lighty) – is an open-source all purpose webserver. With a small memory footprint, effective management of cpu-load, and an advanced feature set (FastCGI, CGI, Auth, Output-Compression, URL-Rewriting and more) LightTPD is the perfect solution for every server that is suffering load problems. Oh – and it can serve up PHP faster than a shiny red truck.
  • What is Pound ?
    Pound is a reverse proxy, load balancer and HTTPS front-end for Web servers. Pound was developed to enable distributing the load among several Web-servers and to allow for a convenient SSL wrapper for those Web servers that do not offer it natively. It comes with different ‘balancing’ settings to easily configure load distribution.
  • What is Mongrel ?
    It is a hybrid Ruby/C HTTP server designed to be small, fast and very secure. Written by Zed Shaw it is intended for hosting Ruby web applications using plain old HTTP rather than FastCGI or SCGI. A ‘Mongrel Cluster’ simply means a collection of Mongrel server proccess (one or more). Mongrel works well with the load-balancing offered by Pound (above), meaning as traffic increases to your Ruby based website, you can easily distribute the load across a number of Mongrel processes in your backend servers. Capistrano also has built in function to support management of your Mongrel Cluster – all good.
  • What is the point ?
    The main reason I switched to this stack was because Dreamhost’s shared hosting environment serving Rails with Apache2/FCGI just wasn’t cutting it. Especially with a memory hungry Typo build and a bunch of sidebars. FCGI processes would crash, create CORE dumps, need to be reaped every hour and generally cause a lot of trouble. Thats not to say that you cant get a Rails site up and running under Apache2/FCGI – many people do it very well. However – when I finally got a new VPS server at Rimuhost (with root access) – I took the opportunity to try Lighttpd and Mongrel out.

So, Lighttpd is small, versatile and can serve up static pages (and PHP) super fast and Mongrel (see above) is better with the Ruby stuff. What we want to do ideally, is to proxy requests to our Rails app to Mongrel → from Lighttpd. We could use Lighttpd’s mod_proxy for this – BUT – as it stands right now, its a little bit, ‘not working right’. So, to get around that (or until it is fixed in Lighttpd 1.5) – we can use the ‘Pound’ proxy to sit between Lighttpd and Mongrel. If I could be bothered drawing a diagram it would probably look something like this – (gv) – clear as mud.

Starting Point

  • You have got your Rails app under version control with SVN as described in the first tutorial
  • You have got Capistrano working and deploying to your server as described in the second tutorial
  • You have root access on your server for installing/configuring Lighttpd, Pound and Mongrel

Install Mongrel on your Server

  • Get it’s dependencies, daemons and the gem_plugin. Also install mongrel_cluster, which contains a few tools which make managing a cluster of Mongrel servers easier.
    sudo gem install daemons gem_plugin mongrel mongrel_cluster sendfile --include-dependencies
  • Note: this also includes support for ‘sendfile’ – an important part of the this layered stack – With ‘sendfile’, requests can be sent via a proxy (Pound) to your Mongrel servers directly via the local filesystem – Which is much faster than over a local HTTP pipe. Install sendfile on your server (debian) with;
    sudo apt-get install sendfile
  • I’ll leave configuring Mongrel for now – First just test that mongrel is installed ok, try a ‘whereis’ on it, and then try running it to check its there. If not you may need to add its install directory to your $PATH variable;
    whereis mongrel_rails
    mongrel_rails

Install Pound

  • Follow the instructions here, make sure zLib and openSSL are installed first, then install Pound. If (during build/compiling) you experience errors (as I did) relating to libssl and/or libcrypto libraries not being present – then you will also need to install libssl-dev. Then try making and installing zLib/openSSL again. On Debian this would be;
    apt-get install libssl-dev
  • If problems still persist after install (as they did for me!), you can try ‘faking’ the older library file verison with the new like so.
    sudo ln -s libssl.so.0.9.7 /usr/lib/libssl.so.0.9.8
  • Now configure the Pound config file (in /usr/local/etc/pound.cfg, or maybe in /etc/pound.cfg) – Set up Pound up on your server listening on port 6000, and directing requests onto lets say, 3 Mongrel Servers running on ports 6001, 6002 and 6003 (which I’ll setup next).
    ListenHTTP
      Address 0.0.0.0
      Port    6000
      Service
        BackEnd
          Address 127.0.0.1
          Port    6001
        End
        BackEnd
          Address 127.0.0.1
          Port    6002
        End
        BackEnd
          Address 127.0.0.1
          Port    6003
        End
        Session
          Type BASIC
          TTL  300
        End
      End
    End
  • You can run a check on your Pound configuration with;
    sudo pound -v -c
  • You should now start the Pound proxy with;
    sudo pound -v
  • Note: while Pound is running, it wont output errors or even messages to its own log files, instead you should monitor the system log file to see whats going on;
    tail -f /var/log/sys.log  (for debian, or system.log in OSX)

Configure Lighttpd

Lighttpd will be the main port of call for all HTTP requests to port 80. It is our main front-facing web server to the general browsing public, and it will pass off requests (for our Rails app) to the Pound proxy on port 6000; Your config/lighttpd.conf (normally in your main Rails directory) should look a little like this;

server.bind = "0.0.0.0"
server.port = 80

server.modules           = ( "mod_rewrite", "mod_accesslog", "mod_fastcgi", "mod_compress", "mod_expire", "mod_proxy" )

server.error-handler-404 = "/dispatch.fcgi"
server.document-root     = CWD + "/public/"
server.errorlog          = CWD + "/log/lighttpd.error.log"
accesslog.filename       = CWD + "/log/lighttpd.access.log"

#url.rewrite              = ( "^/$" => "index.html", "^([^.]+)$" => "$1.html" )

compress.filetype        = ( "text/plain", "text/html", "text/css", "text/javascript" )
compress.cache-dir       = CWD + "/tmp/cache"

expire.url               = ( "/favicon.ico"  => "access 3 days",
                             "/images/"      => "access 3 days",
                             "/stylesheets/" => "access 3 days",
                             "/javascripts/" => "access 3 days" )

# lighttpd configured with mod_proxy to only one backend server (pound)
proxy.balance = "fair"
proxy.server  = ( "/" => ( ( "host" => "127.0.0.1", "port" => 6000 ) ) )

.
.
... more config stuff ... etc ...
.
  • Note: Whatever your lighttpd.conf looks like, the important things to change here are to add mod_proxy to the modules and to add the proxy.balance and proxy.server variables. (You might have noticed I commented out the url.rewrite used in the Typo 4.0 build, for some reason this was causing errors and I havent figured out why yet.)

Configure Mongrel

  • Finally we need to setup those 3 Mongrels on ports 6001, 6002 and 6003 – to do so, in your main Rails folder create a /config/mongrel_cluster.yml file like so;
    cwd: /u/apps/yourdomainname.com/current
    port: "6001"
    environment: production
    address: 127.0.0.1
    pid_file: log/mongrel.pid
    servers: 3
  • Where cwd, is the full path to your Rails ‘current’ working directory – Remember, since we are using Capistrano, this will be the current/ symlink that points to the latest ‘release’ folder. Note the servers: 3, which creates 3 Mongrel processes, just change this number to specify how many Mongrel processes you want running.
  • Note: It might be a good time to checkin all these changes to SVN at this point.

Fire it up!

  • With Pound already running, we just need to start Lighttpd and our 3 Mongrels. I dont believe it matters in what order you do this, but i’ll start with Lighttpd, from your main Rails directory;
    sudo ruby script/server -d
  • Now start the 3 Mongrels barking, from within the main Rails directory;
    sudo mongrel_rails cluster::start
  • Thats it! open up your browser and check your site. You can tail -f to monitor for errors/messages in Mongrel, Pound and Lighttpd like so;
    tail -f /var/log/sys.log (will show pound errors and other sys info)
    tail -f rails_dir/log/mongrel.log (mongrel info)
    tail -f rails_dir/log/lighttpd.access.log (lighttpd access log)
    tail -f rails_dir/log/lighttpd.error.log (lighttpd error log)

Help! its not working!

  • Use the following command to list running processes that are using open ports on your system;
    sudo lsof -i -P
  • Use the following command to port scan your server;
    sudo nmap -sT -O localhost
  • To start debugging problems, check your logs and try those port mapping commands (above) first !
  • Feel free to drop me an email (matt [at] hiddenloop [dot] com)about any part of this tutorial, or post a comment and i’ll try to get back to you.

To Do

You’ll probably want to setup /etc/init.d/ scripts for Lighttpd, Mongrel and Pound to start automatically when your server reboots. Maybe i’ll post about this later. Also some tidying up could be done to extract the Lighttpd and Mongrel configuration files out from inside the Rails app directory. AND ideally, its better practice to create a specific user (rather than root) for running the processes for this stack.

Coming soon

In the next (and last) part of this guide, I’ll go through what it takes to adding a copy of the Mint webstats package into your (now nicely setup) Rails app. Showing you how to;

  • make sure it gets ignored by version control
  • is skipped from any Capistrano deployments
  • is served up directly by Lighttpd and PHP

The procedure for doing this with Mint, applies to any non-versioned/non-rails folder you want to place in your Rails app directory structure. And also explains how to mingle serving PHP and Rails from within this stack.

References

Part 2. Setup Capistrano on the Rails app and run a test deploy to your server

1 comment

Starting Point

  • You have got your Rails app under version control with SVN as described in the first tutorial
  • First make sure you have Capistrano installed, you can get it as a free Ruby gem with;
    gem install capistrano
  • ‘Apply’ capistrano to your Rails app on the dev box, this creates some config files locally for Capistrano deployment to work.
    cap --apply-to /u/apps/sitename.com sitename.com
  • Edit the Capistrano’s deploy configuration in /u/apps/sitename.com/config/deploy.rb to set the correct roles for your server;
    set :repository, "svn+ssh://matt@dagobah/svn/#{application}"
    role :web, "server"
    role :app, "server"
    role :db,  "server"
    set :deploy_to, "/u/apps/#{application}"
    
  • From within the Rails directory (/u/apps/sitename.com) run the rake ‘setup’ task to remotely setup Capistrano’s directories on your server;
    rake remote:exec ACTION=setup
  • Try a test deploy to the server (may ask for password during sudo) this will;
    • Checkout the latest revision of your application from your remote repository to the /u/apps/sitename.com/releases directory on your server
    • Update (or create) the /u/apps/sitename.com/current symlink so it points to this new revision
      rake deploy
  • If all goes well, your Rails app should be checked out under /u/apps/sitename.com/current on your remote server. Run ‘rake deploy’ again if you like and see what happens. A new ‘release’ will be checked out to /u/apps/sitename.com/releases/timestamp/ and the symlink at /u/apps/sitename.com/current will be automatically setup to point to it.

Capistrano Tips

  • Obviously with lots of deploys, you’ll start getting loads of release directories on your server, all containing your entire Rails app. You can run the ‘cleanup’ command to remove older releases. In config/deploy.rb – set the following to say how many releases you’d like to keep after a cleanup
    set :keep_releases, 3
  • Then in the Rails directory run;
    cap cleanup

So there you have it, you can now deploy using Capistrano to your server from your dev box and your Rails app is safely under SVN version control. There is loads more you can do with Capistrano (see the links below)

In the next part of this guide, I’ll go through building the hosting stack (lighttpd/pound/mongrel) on your server. And making use of Capistrano to help with deploying your versioned Rails app around an unversioned folder containing the mints/ stats package (which we’ll also get running under PHP with lighttpd)

References

Part 1. Setting up SVN on a working Typo build (or any Rails app)

no comments yet, post one now

So in this post its all about getting your existing Typo (or rails app) under version control with SVN.

Starting Point

  • Have a rails site working and using lighttpd on your local dev box under /u/apps/sitename.com
  • Your remote server has SVN installed (try svn -v) and Rails/Lighttpd installed
  • Make sure you have an ssh key setup for accessing your server from your dev box
  • If you will have multiple users checking in and out and using Capistrano/SVN – set them all up a with ssh keys and create a new ‘developer’ group on the server (that has permissions on svn repository folders and /u/apps/)
  • Note: if you make a mistake, you can use this command to clean away all svn folders from the CWD downwards
    find . -type d -name '.svn' -print0 | xargs -0 rm -rdf 

Create SVN repository on svn server, checkin all rails code, then check out a working copy again

  • Login to the server and create a top level directory svn to hold your repositories;
    mkdir /svn
  • Create a new repository in it with;
    svnadmin create /svn/sitename.com
  • On your dev box, import the entire rails app to this new repository on ‘server’ with a simple comment;
    svn import -m "initial import" /u/apps/sitename.com svn+ssh://matt@server/svn/sitename.com
  • On dev box, rename /u/apps/sitename.com to sitename.com.stepaside – you could delete it before checking out your working copy, but its safer to do this in case anything goes wrong.
    mv /u/apps/sitename.com /u/apps/sitename.com.stepaside
  • Now checkout from the new svn repository to your dev box with;
    svn co svn+ssh://matt@server/svn/sitename.com /u/apps/sitename.com/
  • If all is ok, you can now delete sitename.com.stepaside
    rm -r /u/apps/sitename.com.stepaside

SVN Tips

  • You can run svn update, to update your current working copy with the latest from the server repository;
    svn update /u/apps/sitename.com
  • You can run svn commit -m “comment” to commit all changes from your working copy to the repository on the server
    svn commit -m "your comment" /u/apps/sitename.com
  • You can run svn status to see what files have changed or are not under version control;
    svn status /u/apps/sitename.com
  • You can add files/folders to svn’s ignore list for a particular file under version control using propedit; e.g. to set whats to be ignored in the config/ folder;
    svn propedit svn:ignore config/
  • This will popup a text editor (usually vim) – allowing you to specify files/folders inside config/ to ignore – e.g. .rb or deploy etc. In vim, press I to start editing (inserting), edit the file and press Esc and ‘ZZ’ to save and exit. (easy!) – Of course for these ignores to take place you’ll have to do an svn commit.
  • IMPORTANT for these ignores to work on files/folders that are already under SVN version control, you’ll first have to un-version and delete them; So back up the file/folder first then SVN remove it with;
    svn remove /path/to/file
  • Then place the backed up file/folder into your working copy again (if its a folder, delete any .svn folders in it) (Its a real pain, but ‘svn status’ is your friend during this process)
  • You can run svn status —no-ignore to see what files you’ve added to svn’s ignore list (these don’t get version controlled
    svn status --no-ignore /u/apps/sitename.com

References

Intro. All kinds of 'newness'

1 comment

So here we are, your looking at a fresh new install of Typo 4.0, with my custom theme, migrated blog content and mint stats running on a new VPS host, all served up on a platter with a new hosting stack (Lighttpd/Pound/Mongrel) – So basically, all kinds of ‘newness’.

As I mentioned – in the next couple of posts, ill try to walk through what it took to set this up. There are lots of guides already out there that take you through parts of this process, but I am attempting to gather my entire experience together and apply it to an existing Rails application (this Typo blog) – Working with a real Rails app like this, you come across ‘gotchas’ that you wouldnt normally see with a fresh vanilla Rails.

I have taken to using a blank page on my wiki scratchpad to log what I do, when I do it. What you get below is basically a cut & paste from there. Im working with a Mac/OSX as my local development machine and a Debian Rimuhost VPS as my server. Im assuming you have root access on both boxes to do this. And while you could do all this with Windows – I choose not to (I’ve been down that road before)

Terms Used

  • server (production/svn server) – the remote box hosting the svn repository and live website, and its alias on your local dev box
  • dev box – local client machine contains working copy of site
  • sitename.com – domain name of site
  • CWD – abbreviation for current working directory
  • Rails directory – the top level Rails dir, i.e. the one containing app/ config/ public/ etc.
  • start reading PART 1
← (k) prev | next (j) →