MongoDB skew monitoring

We use MongoDB at Fotopedia for a variety of things. Last week, one of our mongo got stuck in its master-slave replication process and it took some time for us to detect the issue. Fortunately, mongod servers do expose their internal state regarding the replication and we quickly were able to find what was going on.

As a result, we have improved our Nagios probe to detect large skew (define large how you wish to) in the replication process. At the same time, we have been working on a Munin probe to graph the evolution of the skew across time and see if some pattern emerge.

The probe I'm introducing today is a probe that does both the Nagios monitoring and the Munin reporting:

• You can use it as a standalone nagios probe

./check_mongo_replica_member myservername.fqdn.com 27100

It will then behave as regular Nagios probe, reporting wrong status in the replica set or reporting WARNING or CRITICAL state if the slave skew is above the WARNING and CRITICAL threshold.

• You can also link this script to a file name mongolag-{your mongo port number} and use this as a Munin probe by creating the link in the /etc/munin/plugins/ folder:

./mongolag-27100 config
# returns the configuration
./mongolag-27100
# return the Munin data

The munin data can be used to collect the skew and graph that throughout time:

Here is the source code of the probe:
Feel free to fork and improve !

Nagios and Chef at Fotopedia

Introduction and Goals

We use Chef from Opscode at Fotopedia. Among many other things, Chef generates the Nagios configuration for all our services.

Hence, we have 220 tests running on the production grid and every test is declared inside the Chef cookbooks. Because the configuration of Nagios is quite complex, we want to have a high level of flexibility for a given test:

• Name of the probe
• active or passive test
• check command
• retry interval
• normal check interval
• dependency of the service
• maximum check attempts
• normal check interval
• notification period

All these settings are the usual Nagios settings and you probably know what they are about. If this is not the case, please refer to the Nagios documentation for more info.

These settings should have a reasonable default value so that we can quickly add a test that will most of the time do what it's supposed to do.

Of course, depending on the grid the test is running on, the default value should be different: No need to be notified of a warning disk space issue on the testing environment at night. On the production environment, we have to.

Apart from these settings, we also want to include the grids on which the test should run. the grid is a setting we have on our servers that indicates whether we are on the development, test, staging or production servers.

Finally, all tests should be declared in Chef recipes as close to the tested thing as possible so that it can be easy to maintain or at least cargo-cult reused for our fellow developpers.

Also, the machine running the Nagios server must have the ability to fetch a consistent list of services to monitor so as to be able to generate the configuration on the disk and have Nagios pick that up correctly. These lead us to several chef tricks and tips to implement this solution

Consistent per host registry

During a chef run, you have the ability to save the content of the node by calling node.save at any time. This is very useful but induce an additional reindex in the chef-solr. Before 0.10, solr was really CPU consuming and doing node.save outside the usual chef run saves (nodes are usually save at the end of the run) was not a very good idea.

Moreover, calling node.save during a chef run (at any moment) might result in closing the chef transaction of the current node which can have ugly side effects (your chef runs, triggers a restart at the end of chef, node is saved by your scripts, chef crashes. At next run the restart will not be triggered because the action leading to it has already been save to the registry).

When chef runs, we want to list all the currently active tests on the box and to save that at the end of the run in some part of the registry where the Nagios server will be able to search for it. Because we want something that reflects the actual content of the box, we want to enumerate all tests at every chef run, and hence start from an empty state and fill it up until the final state reflects all that's supposedly controlled by chef. If chef runs on the Nagios server at the same time, we don't want to expose a semi-empty list of services to monitor.

So until chef-client has finished running on the box, we want to keep the old list of services and switch it with the new list of service upon chef run completion and not before. And so we need an "atomic" commit in chef registry. The trick is a bit complicated to implement using chef because of the lack of hook to run pre and post run code.

There are hooks for post run reports but at the time of the writing of this code, this was not simple, so we implemented two recipes for that. These recipes HAVE to be run as the first and the last recipe of the whole chef run.

 The first recipe just creates an empty part of the registry with the "next_key" key that contains the place where all registered services will be stored until the end of the run. It also checks that it's running first and will crash if it's not the case. Because of the way we then declare monitored services, this could be optimized by being done at the first monitored service declaration (I leave that as an exercise for the reader) and not in the first recipe.

 Of course, you have to run this recipe as the first recipe on all nodes you want to monitor. Easy way to do that is to include the recipe inside a role that you run anyway on all your nodes.

The last recipes does a very simple job too: After having checked that it's really the last recipe to run, it will move the [:monitored][:next] content to [:monitored][:current]. This is done in a chef ruby_block because this as to run in the converge step of the chef-client run and not before.

As a bonus, the recipe also saves the list of monitored object on the disk. Useful for debugging and interfacing with CLI tools that don't speak Chef.

After the last recipe has run, the node is saved and [:monitored] will now contains the current list of services to monitor.

Chef Helpers to declare services

Default Values and Usual tests

In a the nagios/libraries folder of our nagios cookbook, the following file exists: This file first contains all the default values for the Nagios test (as described in the introduction). In particular, you can note that notification_period is an Hash that links the Grid to the notification period. The others functions are used as simple wrapper to declare what's used to run the tests in nagios (with commands name) and in the CLI (as I hinted you before we have a very cruft tool that allows us to run part of the tests from the CLI without Nagios). Nothing too fancy. It works nevertheless.

Monitored define

The helper we use a lot is a define that can be used anywhere in our recipes to declare a monitored service: This define bends a bit the define mechanism of chef, particularly regarding the parameters. Apart from that, note that the name must be unique on the node (as you'd expect, name is important also in Nagios so this limitations seems sane).

Monitored takes all the parameters you want and store them in the registry for later retrieval. The depends parameters is a magic one and will help you write dependency chain in nagios directly in chef. This must be an array. You can use the following syntax for each of its elements:

• fservice@localhost : currently declared service is dependent on fservice running on the same host
• fservice@server : service depends on fservice on server

There have been thought of extending this option to support fservice@* (so as to describe a service depending on all available fservice ) but this has not been implemented. Here are some examples of real-use:

monitored "apache2" do
  check_command check_http("-p #{node[:apache][:listen_ports][0]} -u /")
  restart_command "apache2ctl graceful"
end

Apache2 is monitored using check_http (helper defined in the usual tests, see previous gist), the port is fetched from the node registry. Oh yeah. The restart_command is used by our CLI tools to indicates our fellow developers what to do when apache is down. Of course, you could add has_internet true or any other parameter and retrieve it later.

[ :queues, :failures, :pending, :processed, :workers, :working, :failed ].each do |kind|
  passive_monitored "resque #{kind}"
end

Declare various Resque queue to be monitorable in Nagios.

{ 'Current_users' => 'check_users',
  'Total_Processes' => 'check_total_procs',
  'Current_Load' => 'check_load',
  'Disk_Usage' => 'check_disk'}.each do |name, command|
  
  monitored name do
    kind 'nrpe'
    check_command nrpe(command)
    infrabox_check false
  end
end

This one create nrpe monitored tests. Map the name to the nrpe command you'd better declare in the nrpe cookbook. Also indicates that these tests are not supposed to run on the infrabox. As you can see, it's very easy to add new tests to the node registry. All these monitored instructions can happen right next to the daemon configuration.

Generating Nagios Configuration

First, all the commands which are in check_commands must be declared in your Nagios commands.cfg.

By improving the OO structure of check_commands.rb it could be possible to store all that's necessary to generate the commands.cfg file of Nagios but the truth is that this file doesn't change a lot so it's usually very quick to add a new test to the file in the Nagios cookbook directly and then create a simple function in check_commands.rb

Then I will just describe how to generate the configuration file for all the nodes. The installation and regular setup of Nagios can be done as you like it.

We use the following recipe:

Nothing too fancy here, most of the magic has happened elsewhere. Three steps:

• Delete files corresponding to old nodes no longer monitored
• Get current monitored nodes (it is used to resolve the dependency graph). This means that you might have to run chef twice to have the correct tree available for your dependencies.
• Resolve dependencies and dump /etc/nagios3/conf.d/#{n['fqdn']}.cfg on the disk. Also ask Nagios to reload

And the missing part, the service template: Note that we reuse the defined Default before and dump the monitored registry of the node in the file. Note the node[:swissr][:grid] that's used as a selector for some of the default values. Also, the service dependency is generated at this point too.

How we use Chef

Chef is a (young) very powerful system integration framework. When we started developing the Fotopedia server apps and infrastructure, we were already sold on automation. Initially, we had a hairy collection of script shells to install our server components. The goal was - and it did not change - to have a standardized way to configure a remote server and to set up a local virtualized linux instance for development purposes. A bit later, we switched to Puppet, when we set up fully separated testing and production environments.

In early 2009, Opscode unveiled Chef, a new system configuration framework. There was a lot to love in Chef, even in the early releases: client-side execution, Ruby as a DSL, the search index and how easy it is to express complex dependencies between distinct services and servers. After a few experiments and prototypes, we slowly switched our infrastructure to Chef in may. Oh by the way, we have a little secret. We did not upgrade to recent Chef versions yet - we’re still in 0.6 and not proud of it :) - but considering all the good things coming in 0.8...

Our Chef workflow

So, what’s our day-to-day workflow with Chef ? We code on OS X, but everybody runs a Linux virtual machine. This local instance (we call it the Infrabox) is configured with the same Chef setup powering our servers. The same setup, but not the exact same configuration: our recipes detect we’re running in “Infrabox” mode and we adapt the installed recipes and tweak the attributes. For instance, we can run it without checking out all application code, we can just use HGFS to point to a directory on the Mac side. We also change the defaults for several components in our stack (i.e. there is no need for a 1 GB cache when developing locally).

We store our chef configuration in a git repository, and our workflow is not very exotic: tweaking the cookbooks, testing locally in infrabox, and finally committing into the testing branch - which is then deployed on the test environment. So it’s quite easy to keep our infraboxes in sync between us: updating the repository, pushing the new cookbooks to the local chef-server, running chef-client. When we're happy with the change, we merge on the master branch.

Chef in the cloud

Our Chef server setup isn’t very exotic. Our architecture was broadly described in a post on the Amazon Web Services blog, so I will just do a quick recap here. We always have a “master” server in a grid (production, testing, ..) where we first bootstrap the Chef-server with chef-solo.

We then have a small ruby script to fire a new EC2 instance and configure a boot script. The instance automatically registers itself in our DNS, installs a base ruby and the Chef gems and announces itself to the Chef server. All is left to do is to wait a few minutes and associate the recipes for the new node in the Chef server UI. We don’t use custom AMI, we just use the Alestic Ubuntu images, we prefer to lose a few minutes of instance setup but keep everything completely modular.

Cookbooks

Some words about our cookbooks: we have about 50 cookbooks in our configuration. Some are straightforward adaptations of the example cookbooks, some are heavily tweaked, and about 1/3 are very specific fotopedia recipes. We’re still using Chef 0.6, so some of these are a bit hackish, meta-recipes - with Chef 0.7, we would have used roles.

In the “heavily tweaked” category, kali coded awesome recipes for varnish and nginx, where we download a source version, patch it, recompile everything and install, while keeping idempotence properties.

Chef really shines when you have to keep a global view of your infrastructure with the global configuration index and search, like updating a load balancer configuration when a new backend is up. Another area where we take advantage of the Chef search index is monitoring. We set a "monitorable" attribute in our recipes, and in the Nagios cookbook, we search for all nodes with this setting, and update the configuration file to add a new probe for the component kind.

Unified deployment

There’s one thing that bothered me with our previous setup: we were using two distinct deployment tools: capistrano and puppet. Both are great tools, but it means using two pieces of software for a single task: having the whole thing deployed, controled and up to date. Capistrano is great, but there’s more than Rails in a Rails app: a version might have a new dependency on an external service, a new memcached hostname might be introduced in the environment.rb, etc. Managing Rails and the underlying software stack through the same tool did remove some friction in this area. To replace the capistrano part, we’re now using chef-deploy by ezmobius.

Another interesting aspect in our conf is that we have several small & large ruby apps sharing common setups. It can be configuration files (e.g database.yml) where we just link to the right file at deploy time in the Chef recipe. But we also have many fine-grain configuration points. Things like well-known URL templates (what is the base URL for assets in this deploy configuration ?), account keys, S3 bucket targets, etc... For these, Aymerick Jéhanne built a small library called Swissr which offers a simple set/get interface for path-like parameters. e.g. Swissr.get(“assets_store/bucket”). We can then directly use this in raw ruby code or by injecting the values into the app configuration in Rails and Merb environment setup. Many of these configuration values are already defined as attributes in the Chef recipe, which we dump into files read a bit later by Swissr. w00t, goodbye configuration duplication !

Deploying Rails apps

When deploying the main applications, our cookbooks checkout from git on the various Passenger backend servers. The process is actually a little bit more complex: we might have to execute database migrations, assets must be compacted and uploaded to the CDN with a version string including the sha1, and this version must be known by the Rails app to correctly forge the assets URL.

The trick is to only execute the assets and migration part once, so we silently deploy the main app on a “passive” server configured with a specialized recipe, where we execute these administrative tasks. The assets version is then pushed back to Chef as an attribute, and “active” Rails backends will pick it up through their regular recipe.

We actually have a small definition to handle our Rails & Merb deployment, which have a lot in common: creating the shared directories (pid, log, system, data upload, ...), changing user and groups, etc. A cool side-effect of using Ruby in Chef cookbooks: we can use our own gems in the recipes code, so we hooked into our own message bus. The chef nodes notify us on our internal IRC channel when an application is upgraded.

Final thoughts

Ok, this post is already too long, but I have to finish with another nice trick by kali: a script which fetches information from AWS, Chef and the DNS to output a graphviz file giving a complete picture of our infrastructure: hostname, instance types, Elastic IP, Elastic Load Balancers, node cookbooks, etc.

Also note that there's a lot of Chef goodness in the latest versions (0.7.14 and the upcoming 0.8) we're not using right now: light-weight resource providers (a great alternative to definitions), roles, data bags, deploy resources, Knife and Shef...

Introducing Casserole

We love Chef. It's a brand new systems integration framework, a configuration management tool, a cloud computing automation toolbox, etc, by the Opscode team. If you want to learn more about it, you should read the tutorial slides by Edd Dumbill at RailsConf 2009.

We're progressively deploying Chef in our infrastructure. And, I must admit, like many of us at Fotonauts, I'm still a desktop applications geek. So when I saw this amazing tool, with a REST API, ... I knew I had to write a Cocoa client for it. It has been a side project, something to hack on while a server is deploying.

We're releasing Casserole today as Open Source on GitHub, under an Apache license. You can also download a build there.

This first version is a bit of a preview, but it should be quite useable for Chef users. What can you do with Casserole ? Access to a chef server, explore nodes, registrations and cookbooks (no template content for now due to a temporary Chef REST API limitation), use the search index, live search your attribute tree or index results.

What does not work ? Write access ! The main goal for this version was to exercise most of the REST API and have a polished UI, but write support is not a big task and should happen soon (yes, I too really want to edit this "recipes" field).

Please remember that Casserole is nothing but a side project - although we're using it internally - so no warranty, no support, no schedules, etc. And don't get angry if Casserole shuts down your entire cloud. But hey, it's open-source, you can fix it !

Feel free to send feedback on casserole@fotonauts.com, and to use GitHub issue tracker.

For the Cocoa inclined crowd: Casserole is written in Objective-C in "modern" style, so you might want to take a peek at the source for Cocoa Bindings, NSOperation and Garbage Collection (don't blame me for some patterns: it was my excuse to play with new toys, my first 10.5 only app, yay!).

Fotonauts @ Paris Rails Camp 2

Fotonauts sponsored the Paris Railscamp v2 event, which took place at the Sun Briefing Customer Center last saturday. We're big fans of the events organized by the vibrant Ruby France association, it's always good opportunities to meet other Ruby coders and share our experience (and we had our logo printed on some cool T-Shirts).

We also presented some slides about our log infrastructure, two-fold: first a refinement on ErrorMailer / ExceptionLogger which takes advantage of our asynchronous architecture, and how we deal with long-term logging and storage (with data-mining in mind). CouchDB, Hadoop and AMQP in the mix, yay.

Sorry, the presentation was in french (we might do a english version in the future).

Oh, and Guillaume Desrat created a nice Fotonauts album. If you have pictures from the event, feel free to request an invite and contribute to his album !