How we use Chef

Chef is a (young) very powerful system integration framework. When we started developing the Fotopedia server apps and infrastructure, we were already sold on automation. Initially, we had a hairy collection of script shells to install our server components. The goal was - and it did not change - to have a standardized way to configure a remote server and to set up a local virtualized linux instance for development purposes. A bit later, we switched to Puppet, when we set up fully separated testing and production environments.

In early 2009, Opscode unveiled Chef, a new system configuration framework. There was a lot to love in Chef, even in the early releases: client-side execution, Ruby as a DSL, the search index and how easy it is to express complex dependencies between distinct services and servers. After a few experiments and prototypes, we slowly switched our infrastructure to Chef in may. Oh by the way, we have a little secret. We did not upgrade to recent Chef versions yet - we’re still in 0.6 and not proud of it :) - but considering all the good things coming in 0.8...

Our Chef workflow

So, what’s our day-to-day workflow with Chef ? We code on OS X, but everybody runs a Linux virtual machine. This local instance (we call it the Infrabox) is configured with the same Chef setup powering our servers. The same setup, but not the exact same configuration: our recipes detect we’re running in “Infrabox” mode and we adapt the installed recipes and tweak the attributes. For instance, we can run it without checking out all application code, we can just use HGFS to point to a directory on the Mac side. We also change the defaults for several components in our stack (i.e. there is no need for a 1 GB cache when developing locally).

We store our chef configuration in a git repository, and our workflow is not very exotic: tweaking the cookbooks, testing locally in infrabox, and finally committing into the testing branch - which is then deployed on the test environment. So it’s quite easy to keep our infraboxes in sync between us: updating the repository, pushing the new cookbooks to the local chef-server, running chef-client. When we're happy with the change, we merge on the master branch.

Chef in the cloud

Our Chef server setup isn’t very exotic. Our architecture was broadly described in a post on the Amazon Web Services blog, so I will just do a quick recap here. We always have a “master” server in a grid (production, testing, ..) where we first bootstrap the Chef-server with chef-solo.

We then have a small ruby script to fire a new EC2 instance and configure a boot script. The instance automatically registers itself in our DNS, installs a base ruby and the Chef gems and announces itself to the Chef server. All is left to do is to wait a few minutes and associate the recipes for the new node in the Chef server UI. We don’t use custom AMI, we just use the Alestic Ubuntu images, we prefer to lose a few minutes of instance setup but keep everything completely modular.

Cookbooks

Some words about our cookbooks: we have about 50 cookbooks in our configuration. Some are straightforward adaptations of the example cookbooks, some are heavily tweaked, and about 1/3 are very specific fotopedia recipes. We’re still using Chef 0.6, so some of these are a bit hackish, meta-recipes - with Chef 0.7, we would have used roles.

In the “heavily tweaked” category, kali coded awesome recipes for varnish and nginx, where we download a source version, patch it, recompile everything and install, while keeping idempotence properties.

Chef really shines when you have to keep a global view of your infrastructure with the global configuration index and search, like updating a load balancer configuration when a new backend is up. Another area where we take advantage of the Chef search index is monitoring. We set a "monitorable" attribute in our recipes, and in the Nagios cookbook, we search for all nodes with this setting, and update the configuration file to add a new probe for the component kind.

Unified deployment

There’s one thing that bothered me with our previous setup: we were using two distinct deployment tools: capistrano and puppet. Both are great tools, but it means using two pieces of software for a single task: having the whole thing deployed, controled and up to date. Capistrano is great, but there’s more than Rails in a Rails app: a version might have a new dependency on an external service, a new memcached hostname might be introduced in the environment.rb, etc. Managing Rails and the underlying software stack through the same tool did remove some friction in this area. To replace the capistrano part, we’re now using chef-deploy by ezmobius.

Another interesting aspect in our conf is that we have several small & large ruby apps sharing common setups. It can be configuration files (e.g database.yml) where we just link to the right file at deploy time in the Chef recipe. But we also have many fine-grain configuration points. Things like well-known URL templates (what is the base URL for assets in this deploy configuration ?), account keys, S3 bucket targets, etc... For these, Aymerick Jéhanne built a small library called Swissr which offers a simple set/get interface for path-like parameters. e.g. Swissr.get(“assets_store/bucket”). We can then directly use this in raw ruby code or by injecting the values into the app configuration in Rails and Merb environment setup. Many of these configuration values are already defined as attributes in the Chef recipe, which we dump into files read a bit later by Swissr. w00t, goodbye configuration duplication !

Deploying Rails apps

When deploying the main applications, our cookbooks checkout from git on the various Passenger backend servers. The process is actually a little bit more complex: we might have to execute database migrations, assets must be compacted and uploaded to the CDN with a version string including the sha1, and this version must be known by the Rails app to correctly forge the assets URL.

The trick is to only execute the assets and migration part once, so we silently deploy the main app on a “passive” server configured with a specialized recipe, where we execute these administrative tasks. The assets version is then pushed back to Chef as an attribute, and “active” Rails backends will pick it up through their regular recipe.

We actually have a small definition to handle our Rails & Merb deployment, which have a lot in common: creating the shared directories (pid, log, system, data upload, ...), changing user and groups, etc. A cool side-effect of using Ruby in Chef cookbooks: we can use our own gems in the recipes code, so we hooked into our own message bus. The chef nodes notify us on our internal IRC channel when an application is upgraded.

Final thoughts

Ok, this post is already too long, but I have to finish with another nice trick by kali: a script which fetches information from AWS, Chef and the DNS to output a graphviz file giving a complete picture of our infrastructure: hostname, instance types, Elastic IP, Elastic Load Balancers, node cookbooks, etc.

Also note that there's a lot of Chef goodness in the latest versions (0.7.14 and the upcoming 0.8) we're not using right now: light-weight resource providers (a great alternative to definitions), roles, data bags, deploy resources, Knife and Shef...

Introducing Casserole

We love Chef. It's a brand new systems integration framework, a configuration management tool, a cloud computing automation toolbox, etc, by the Opscode team. If you want to learn more about it, you should read the tutorial slides by Edd Dumbill at RailsConf 2009.

We're progressively deploying Chef in our infrastructure. And, I must admit, like many of us at Fotonauts, I'm still a desktop applications geek. So when I saw this amazing tool, with a REST API, ... I knew I had to write a Cocoa client for it. It has been a side project, something to hack on while a server is deploying.

We're releasing Casserole today as Open Source on GitHub, under an Apache license. You can also download a build there.

This first version is a bit of a preview, but it should be quite useable for Chef users. What can you do with Casserole ? Access to a chef server, explore nodes, registrations and cookbooks (no template content for now due to a temporary Chef REST API limitation), use the search index, live search your attribute tree or index results.

What does not work ? Write access ! The main goal for this version was to exercise most of the REST API and have a polished UI, but write support is not a big task and should happen soon (yes, I too really want to edit this "recipes" field).

Please remember that Casserole is nothing but a side project - although we're using it internally - so no warranty, no support, no schedules, etc. And don't get angry if Casserole shuts down your entire cloud. But hey, it's open-source, you can fix it !

Feel free to send feedback on casserole@fotonauts.com, and to use GitHub issue tracker.

For the Cocoa inclined crowd: Casserole is written in Objective-C in "modern" style, so you might want to take a peek at the source for Cocoa Bindings, NSOperation and Garbage Collection (don't blame me for some patterns: it was my excuse to play with new toys, my first 10.5 only app, yay!).

Fotonauts @ Paris Rails Camp 2

Fotonauts sponsored the Paris Railscamp v2 event, which took place at the Sun Briefing Customer Center last saturday. We're big fans of the events organized by the vibrant Ruby France association, it's always good opportunities to meet other Ruby coders and share our experience (and we had our logo printed on some cool T-Shirts).

We also presented some slides about our log infrastructure, two-fold: first a refinement on ErrorMailer / ExceptionLogger which takes advantage of our asynchronous architecture, and how we deal with long-term logging and storage (with data-mining in mind). CouchDB, Hadoop and AMQP in the mix, yay.

Sorry, the presentation was in french (we might do a english version in the future).

Oh, and Guillaume Desrat created a nice Fotonauts album. If you have pictures from the event, feel free to request an invite and contribute to his album !