Last week a colleague and I were talking about custom network protocols. As I haven’t done much networking programming in Linux, it piqued my curiosity about how hard it is to send data using IP but not one of the common transport protocols (TCP or UDP). Since I’m currently on a plane without wifi, I’ve written up some of my notes on this short bout of manpage surfing.

I quickly found that the raw(7) manual page has documentation on just what I was looking for:

Raw sockets allow new IPv4 protocols to be implemented in user space. A raw socket receives or sends the raw datagram not including link level headers.

The following is a minimal example that sends a single message, “Hello World!\0”, to localhost using the Internet Protocol and a raw AF_INET socket:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>

int main() {
  int sd;
  struct sockaddr_in dest;
  char msg[] = "Hello, World!";

  dest.sin_family = AF_INET;
  if (inet_pton(AF_INET, "127.0.0.1", &(dest.sin_addr)) != 1) {
      printf("Bad Address!\n");
      return(1);
  }

  if ((sd = socket(AF_INET, SOCK_RAW, 253)) < 0) {
      printf("socket() failed!\n");
      return(1);
  }

  if (sendto(sd, &msg, 14, 0, (struct sockaddr*) &dest, sizeof(struct sockaddr)) < 0)  {
    printf("sendto() failed!\n");
    return(1);
  }

  return(0);
}

inet_pton(), socket(), sendto()

This short program can be broken into three parts:

1. Specifying our destination,
2. Creating a socket, and
3. Sending a message over the socket.

The key system calls used to complete these tasks are inet_pton, socket() and sendto(), respectively.

inet_pton()

Our destination is stored in a sockaddr_in structure that has the following definition (from ip(7)):

struct sockaddr_in {
     sa_family_t    sin_family; /* address family: AF_INET */
     in_port_t      sin_port;   /* port in network byte order */
     struct in_addr sin_addr;   /* internet address */
};

In our example, sin_family will be AF_INET as we are using IPv4. Since port is a concept used by transport layer protocols such as UDP and TCP and not used by IP, we’ll ignore the port in this case. Finally, the sin_addr field is the destination address in network byte order.

Rather than figuring out how to turn a human-readable destination such as “127.0.0.1” into its network byte-order representation, we use inet_pton to do the dirty work for us:

inet_pton(AF_INET, "127.0.0.1", &(dest.sin_addr))

socket()

The socket system call has the following signature:

socket(int domain, int type, int protocol)

In this case, we want an AF_INET socket of type SOCK_RAW. In the context of an AF_INET socket, the protocol fields specifies which protocol will be used on top of IP. Since we don’t want to use a protocol on top of IP, I’ve used protocol 253 which RFC 3692 reserves for experimental and testing purposes.

This will return a file descriptor that we can pass to sendto().

sendto()

Finally, we use sendto() to send the desired message. sendto has the following signature:

sendto(int sockfd, const void *buf, size_t len, int flags,
       const struct sockaddr *dest_addr, socklen_t addrlen);

In the above example

sockfd is sd, the socket created by line 17, *buf is the message defined at line 9 and len is its length. Flags is set to 0 as we don’t need special flags this case.

*dest_addr is our destination specified as a sockadd_in structure. This contains the destination address we packed into it using inet_pton.

Verification with TCP Dump

If we compile and run this program, we can capture the incoming packet and verify that we didn’t send anything but our IP header and the payload “Hello, World!\0”.

$ gcc -Wall send_msg.c
$ bash -c 'sleep 10; sudo ./a.out'
$ sudo tcpdump -i lo -X
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes

20:46:26.470243 IP localhost > localhost:  ip-proto-253 14
        0x0000:  4500 0022 0000 4000 40fd 3bdd 7f00 0001  E.."..@.@.;.....
        0x0010:  7f00 0001 4865 6c6c 6f2c 2057 6f72 6c64  ....Hello,.World
        0x0020:  2100                                     !.

In hex our payload is:

$ printf 'Hello, World!\0' | hexdump
0000000 6548 6c6c 2c6f 5720 726f 646c 0021

which we can see at the end of the packet. Assuming that our packet starts with the IP header, RFC 791 specifies that the first 4 bits is the version field and the second four bits is the “Internet Header Length”. Here, we can see the version field is set to 4 and the IHL is 5. According to the RFC

Internet Header Length is the length of the internet header in 32 bit words, and thus points to the beginning of the data.

Five 32-bit words is 160 bits. 160 bits is 40 hexadecimal digits. That accounts for the rest of the packet we received, confirming that the packet we sent is nothing more than an IP header and our desired payload.

Verification with Socat

If we wanted to see our payload via something other than tcpdump we can use the swiss-army knife of socket communication, socat:

In one shell:

sudo socat IP4-RECVFROM:253 -

In another:

sudo ./a.out

Back in our original shell we should see:

Hello, World!

In fact, we can use socat to send the same payload using only IPv4:

printf 'Hello, World!\0' | sudo socat - IP4-SENDTO:localhost:253

Using TCP dump, we can see that socat sent the same packet we did:

21:16:35.729037 IP localhost > localhost:  ip-proto-253 14
    0x0000:  4500 0022 0000 4000 40fd 3bdd 7f00 0001  E.."..@.@.;.....
    0x0010:  7f00 0001 4865 6c6c 6f2c 2057 6f72 6c64  ....Hello,.World
    0x0020:  2100                                     !.

Further Exploration and Notes

Digging Deeper

If we wanted to dig deeper, Linux provides a couple of ways to dig even deeper into the network stack from userspace:

• Using setsockopt() we can set the IP_HDRINCL option. When this option is set, we can provide our own IP header information for each message we want to send.

• Using an AF_PACKET it is possible to send packets at the device driver level. See packet(7) for more details.

IPv6

Using the getaddrinfo() function we can support both IPv4 and IPv6 addresses. A short, un-elaborated example is below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
  int sd;
  struct addrinfo hints;
  struct addrinfo *res;

  if (argc != 3) {
    fprintf(stderr, "usage: send_msg IP MSG\n");
    return(1);
  }

  memset(&hints, 0, sizeof(struct addrinfo));
  hints.ai_family = AF_UNSPEC;
  hints.ai_socktype = SOCK_RAW;
  hints.ai_protocol = 253;
  if (getaddrinfo(argv[1], NULL, &hints, &res) != 0) {
    fprintf(stderr, "getaddrinfo() failed!\n");
  }

  if ((sd = socket(res->ai_family, res->ai_socktype, res->ai_protocol)) < 0) {
    fprintf(stderr, "socket() failed!\n");
    return(1);
  }

  if (sendto(sd, argv[2], strlen(argv[2]), 0, res->ai_addr, res->ai_addrlen) < 0) {
    fprintf(stderr, "sendto() failed!\n");
  }

  return(0);
}

The real Internet

If you try most of the small examples here with two hosts on a real network, you may find that you message doesn’t reach its destination. Much of the network equipment it has to traverse is likely doing transport layer inspection and dropping your message.

References

• raw(7) man page
• ip(7) man page
• socket(7) man page
• packet(7) man page
• sendto(2) man page
• inet_pton(3) man page
• getaddrinfo(3) man page
• RFC 791

Earlier this week I released version 0.9.0 of my xinetd cookbook. This cookbook provides:

• An LWRP for managing xinetd services,
• Attribute driven configuration of xinetd.conf, and
• a recipe for enabling the internal xinetd-provided services (chargen, daytime, discard, echo, time, and tcpmux-server).

This cookbook is not yet on the Opscode Community site as I am working with the maintainer of the currently uploaded xinetd cookbook to ensure we don’t break anyone using the existing cookbook.

For more details about this cookbook, please see the README. If you are interested in contributing to this cookbook, pull requests are accepted on Github.

passwd_min is an Ohai plugin that parses /etc/passwd and /etc/group and returns user and group information in a format that matches the passwd plugin shipped in Ohai.

This plugin may be useful to Chef users who have alternate password databases (such as LDAP or NIS) configured and have had to disable the default passwd plugin to avoid storing their organization’s entire LDAP directory in their node objects.

Background

The passwd plugin shipped in Ohai uses Ruby’s Etc module to popular the node['etc']['passwd'] and node['etc']['group'] attributes.

If one follows the code for the Etc module far enough, she will find that this information is obtained by calling the getpwent() system call:

1
2
3
4
5
6
7
8
9
10

passwd_iterate(void)
{
    struct passwd *pw;

    setpwent();
    while (pw = getpwent()) {
        rb_yield(setup_passwd(pw));
    }
    return Qnil;
}

According to the Linux Manual Page for getwpent():

The getpwent() function returns a pointer to a structure containing the broken-out fields of a record from the password database (e.g., the local password file /etc/passwd, NIS, and LDAP). The first time getpwent() is called, it returns the first entry; thereafter, it returns successive entries.

Thus, for users who have have LDAP or NIS configured, node['etc']['passwd'] will contain more than simply the contents of /etc/passwd. For some users, the data the default passwd plugin collects will produce large node objects. These large node objects place additional load on the Chef Server and other chef clients that have to process those objects later.

Using the passwd_min plugin

The easiest way to distribute the passwd_min plugin is via the Opscode Ohai cookbook. From your Chef Repository:

1
2
3
4
5
6
7

knife cookbook site install ohai
wget https://raw.github.com/stevendanna/ohai-plugins/master/passwd_min.rb
mv passwd_min.rb cookbooks/ohai/files/default/plugins/
knife cookbook upload ohai
# Add the recipe to the run list of the relevant nodes
# or roles
knife node run_list add NODENAME ohai

You will also want to disable the default passwd plugin by placing the following in /etc/chef/client.rb on the node:

 Ohai::Config[:disabled_plugins] = ["passwd"]

Note that if you use the Chef-client Cookbook, this can be done via an attribute.

Bug reports and improvements happily accepted on Github.

Recently, a coworker has been sharing some higher-order shell functions he has been writing, inspired by the first of these articles:

• Higher Order Functions in Zsh
• Higher-order shell

Both discuss small scripts and shell functions that allow you to more easily compose simple unix tools to complete complex task.

My own ~/bin contains similar higher-order functions and other simple tools. Like most unix tools, they do a single task and are composable with other tools on the command line. However, I am often surprised by the number of heavy command-line users I see who don’t regularly encapsulate repetitive tasks or constructions into shell functions or aliases.

In my experience, the majority of such tools

• take less than 5 minutes to write,
• require almost no maintenance, and
• prove repeatably useful after their initial creation.

This post shares two tools in my ~/bin that I used this weekend. I created both while ago for entirely different purposes, but was still able to easily use them together without modification.

do.times N COMMAND

do.times executes COMMAND, N times. For example,

1
2
3
4
5
6

$ do.times 5 echo hello
hello
hello
hello
hello
hello

The tool itself is little more than a wrapper to a shell for loop, but the typing it saves and the semantic value it provides when constructing a command line has proven valuable on numerous occasions.

summarize [OPTIONS] [FILE]

summarize reads columns of data from either a file given as an argument or its standard input and provides basic summary statistics. I originally wrote this to do quick spot checks of data files I was working on before sending them off to collaborators.

Here is an example of its output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

$ summarize --sep="," --header data_file
    wombats            frogs
 Min.   :0.01911   Min.   :0.02979
 1st Qu.:0.22101   1st Qu.:0.32504
 Median :0.51178   Median :0.50530
 Mean   :0.51318   Mean   :0.51650
 3rd Qu.:0.79354   3rd Qu.:0.71484
 Max.   :0.99561   Max.   :0.99827
 It requires
Covariance Matrix:
            wombats       frogs
wombats 0.084647845 0.006553818
frogs   0.006553818 0.074872790

Column Sums:
 wombats    frogs
51.31770 51.65021

Number of Rows:
[1] 100

First 6 Rows:
     wombats     frogs
1 0.15767016 0.3545217
2 0.42022851 0.5092576
3 0.22144630 0.6796292
4 0.84320607 0.6749124
5 0.07491303 0.3430995
6 0.65910419 0.3053271

While it doesn’t seem like much, these basic statistics are often all one needs to ensure they are sending the correct data to a teammate or to quickly answer a basic question.

summarize uses Rscript, an executable shipped with R that allows one to create scripts using R. Recently, I added a dependency on the CRAN package optparse to make handling options a bit more straightforward. With the exception of the option parsing, the R code itself is likely easily understood by anyone who has used R.

Smoke Testing Speed Improvements

This weekend I have been working on a small set of improvements to make some not-so-simple tools I use on a regular basis a bit faster. Combining do.times and summarize allowed me to quickly generate a smoke test for whether the speed improvements I was implementing were working. To protect the innocent, I’ll use git --version as an example of the command I wanted to test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

$ do.times 100 time git --version 2>&1 | awk '/real/ {print substr($2,3,5)}' | summarize
       V1
 Min.   :0.00100
 1st Qu.:0.00100
 Median :0.00100
 Mean   :0.00116
 3rd Qu.:0.00100
 Max.   :0.00400

Covariance Matrix:
             V1
V1 2.771717e-07

Column Sums:
   V1
0.116

Number of Rows:
[1] 100

First 6 Rows:
     V1
1 0.004
2 0.004
3 0.001
4 0.002
5 0.001
6 0.001

While this isn’t scientific testing by any means, it was enough to keep me moving in the right direction, and simple enough that the cost of creating and running the test was practically zero.

Recently, I’ve used Shef, the interactive Chef console, to get to the bottom of some rather tricky problems. In the process, I developed a number of tricks and tools for productively debugging problems using Shef. This is the first in a series of articles I’d like to write on the most useful of these tips.

In this article I will discuss how I use Shef and a handful of additional functions to debug Chef-client runs by stepping through a node’s run list, breaking before or after specific resources in order to inspect the state of the system. Note that this is not intended to be an introduction to Shef. For that, you should head over to the Chef Wiki.

Setup

Some of the functions I mention in this article are in the shef/ directory of my knife-hacks repository. If you’d like to use them, you can download this repository from Github:

1
2
3

mkdir ~/src
cd ~/src
git clone git://github.com/stevendanna/knife-hacks.git

To make it easy to include files from this repository inside Shef, you can use the following in the relevant configuration file:

1
2
3

unless defined?(Shef).nil?
  $: << File.expand_path("~/src/knife-hacks/")
end

All of the examples I will be doing will be on my local workstation. For ease, I have placed this in my knife.rb file, which I will pass to Shef using the -c command line option. However, a similar approach will work within a client.rb for a node or a custom shef.rb file.

Introduction

Shef is an interactive console for Chef. Essentially, it is IRB with support for recipe and attribute syntax and a number of Chef-relevant functions.

Here’s a typical Shef session that you might find in an introduction to Shef. We add some resources, including a breakpoint resource and start a chef-run.

1
2
3
4
5
6
7
8
9
10
11

chef > recipe
chef:recipe > echo off
chef:recipe > log "a"
chef:recipe > breakpoint "STOP"
chef:recipe > log "b"
chef:recipe > run_chef
DEBUG: Processing log[a] on sdanna
INFO:  Processing log[a] action write ((irb#1) line 3)
INFO:  a
DEBUG: Processing [/home/sdanna/.rvm/gems/ruby-1.9.3-p0/gems/chef-0.10.8/lib/chef/mixin/recipe_definition_dsl_core.rb:61:in `new'] on sdanna
INFO:  Processing [/home/sdanna/.rvm/gems/ruby-1.9.3-p0/gems/chef-0.10.8/lib/chef/mixin/recipe_definition_dsl_core.rb:61:in `new'] action break ((irb#1) line

The breakpoint stops the chef-client run, allowing you to investigate the state of the system and then resume the run with chef_run.resume

1
2
3

chef:recipe > chef_run.resume
INFO:  Processing log[b] action write ((irb#1) line 6)
INFO:  b

With these basic features, Shef can be great for running quick tests of recipe code. However, it is not immediately clear how you get from here to being able to step through an actual chef-client run. To effectively step through a chef-client run, we need to be able to:

• Load resources from the recipes in the node’s run_list, and
• Insert breakpoints between the loaded resources.

The remainder of this article covers how to accomplish these two tasks.

Loading Resources from the RunList

Invoking Shef with the -z argument enables client-mode, forcing it to download the relevant recipes from the node’s run_list just as chef-client would, but it does not processes these recipes and add their resources to the resource collection.

Because Shef’s recipe context allows you to use the recipe DSL, we can add the resources from an individual recipe using the include_recipe function. We can only use include_recipe on recipes that Shef downloaded when we started it up.

1
2
3
4
5

chef:recipe > resources
[]
chef:recipe > include_recipe "ssh_known_hosts"
chef:recipe > resources
["template[/etc/ssh/ssh_known_hosts]"]

In order to load all of the recipes we need to iterate over a list of all the recipes in the run_list and call include_recipe on each of them. Since the run_list can contain roles, we also need to ensure we get the expanded run_list. There are a number of ways to get the expanded run_list. Here is one:

1
2
3

node.run_list.expand(node.chef_environment).recipes.each do |r|
  include_recipe r
end

This simple bit of Ruby is much like the resource compilation phase of a chef-client run. Just as within a chef-client run, you may encounter an error as you try to include the recipes. In a future article, I may discuss better ways to step through this compilation phase.

Since I’d rather not type the above every time I want to debug a chef-client run, I encapsulated this into a function within my ShefExtras library: load_node_run_list.

Inserting Breakpoints between Resources

A large run_list can easily contain hundreds of resources. While we could step through the resources one-by-one, we often want to run through a large number of the resources, and then stop just before a resource that exhibiting some errant behavior.To do this, we need a way to insert breakpoints between the resources we are loading from recipes.

Since “breakpoint” is a fully-fledged Chef resource, we could place the breakpoint resources directly in the relevant recipe. Within a normal chef-client run, the breakpoint resources will have no effect, allowing us to do this without much fear of endangering other nodes using the same recipe.

However, I’ve found that it is more useful to be able to add the breakpoints via Shef, since we will often want to add new breakpoints as we gain new information.

To accomplish this, I’ve created an insert_break function to do just that within my ShefExtras library. Since it is a bit uglier than I would like and depends on mucking about in the depths of Chef’s data structures, I am not going to walk through how the function works. However, here is an example of how to use it:

• First, we load the ShefExtras library and switch to recipe mode.

1
2
3
4
5

chef > echo off
chef > require 'shef/extras'
chef > ShefExtras.load
INFO: ShefExtras loaded!
chef > recipe

• Next, we load the resources from the recipe in the node’s run list and insert a break point before one of them.

1
2
3
4
5
6
7
8
9
10

chef:recipe > load_node_run_list
chef:recipe > resources
["log[Please set the set_fqdn attribute to desired hostname]",
"template[/etc/ssh/ssh_known_hosts]"]
chef:recipe > insert_break :before, "template[/etc/ssh/ssh_known_hosts]"
INFO:  Breakpoint added before template[/etc/ssh/ssh_known_hosts]
chef:recipe > ordered_resources
=> ["log[Please set the set_fqdn attribute to desired hostname]",
"break[break-before-template[/etc/ssh/ssh_known_hosts]]",
"template[/etc/ssh/ssh_known_hosts]"]

• Finally, we run chef.

1
2
3
4
5
6

chef:recipe > run_chef
DEBUG: Processing log[Please set the set_fqdn attribute to desired hostname] on sdanna
INFO:  Processing log[Please set the set_fqdn attribute to desired hostname] action write (hostname::default line 64)
WARN:  Please set the set_fqdn attribute to desired hostname
DEBUG: Processing break[break-before-template[/etc/ssh/ssh_known_hosts]] on sdanna
INFO:  Processing break[break-before-template[/etc/ssh/ssh_known_hosts]] action break (/home/sdanna/src/knife-hacks/shef/extras.rb line 38)

As you can see, the chef-client run halts before running the template resources. At this point we could investigate elements of the system state that we believe may be having an effect on the template resource. When we are ready to continue, we can call chef_run.resume to continue the run.

In this example, I’ve used three custom functions from the ShefExtras module:

• load_node_run_list loads the resources from the node’s run list in the same way as we did in the previous section.

• ordered_resources returns a list of the resources in the order that chef-client would run them.

• insert_break(preposition, resource) will place a break resource either before or after the given resource.

When Do I Use This

Using these tools to step through the resource list is particularly useful in the following types of situations:

• The run list contains recipes which are heavily dependent on execute or script resources that may not be behaving as you expect.
• A recipe makes significant run time modifications of node attributes.
• A recipe’s behavior non-trivially branches based on pieces of the system’s state that is not easily observed before or after the chef-client run has completed.

While I still prefer to start with reading the relevant recipe and reasoning about what will happen, having the ability to quickly run a portion of the recipe and confirm a hypothesis about the state of the system has proven incredibly valuable.

More to Come

Future articles in this series may cover:

• Customizing your Shef configuration,
• Using Shef to make bulk changes to node data,
• Analyzing API responses using Shef,
• Running Shef as an inferior process in Emacs, and
• Debugging errors with resource compilation.

Ohai is a library used by Chef to collect information about nodes within your infrastructure. Information is collected by a set of Ohai plugins, most of which parse the output of system commands.

During lunch on Friday, I create a simple Ohai plugin to collect some basic information about R. You can find the plugin here:

https://github.com/stevendanna/ohai-plugins/blob/master/r.rb

Currently, this plugin collects:

• The version of R,
• The installed packages, and
• The output of capabilities()

To use this plugin right away, the easiest course of action is to use the Opscode Ohai cookbook. From your chef repository:

1
2
3
4
5
6
7
8
9

knife cookbook site install ohai
wget https://raw.github.com/stevendanna/ohai-plugins/master/r.rb
mv r.rb cookbooks/ohai/files/default/plugins/
knife cookbook upload ohai
# Add the recipe to the run list of the relevant nodes
# or roles
knife node run_list add NODENAME ohai
# The ohai plugin will run on the next chef-client run
knife ssh 'name:NODENAME'

In order to avoid unnecessary runs of Ohai within a chef-client run you can add the following line to the client.rb configuration file on your nodes:

1

Ohai::Config[:plugin_path] << '/etc/chef/ohai_plugins'

If you are using Opscode’s chef-client cookbook, this will already be taken care of for you.

The following is an example of the information it collects:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

> knife node show snow-master -a languages.r -Fj
{
  "languages.r": {
    "capabilities": {
      "tiff": true,
      "cledit": false,
      "tcltk": true,
      "X11": false,
      "sockets": true,
      "fifo": true,
      "iconv": true,
      "cairo": true,
      "png": true,
      "http/ftp": true,
      "libxml": true,
      "jpeg": true,
      "aqua": false,
      "NLS": true,
      "profmem": true
    },
    "packages": [
      {
        "name": "Rmpi",
        "version": "0.5-9",
        "built": "2.14.1"
      },
      {
        "name": "snow",
        "version": "0.3-8",
        "built": "2.14.1"
      },
      {
        "name": "KernSmooth",
        "version": "2.23-7",
        "built": "2.14.0"
      },
[ ... SNIP MANY MORE PACKAGE ...]
      {
        "name": "boot",
        "version": "1.3-3",
        "built": "2.14.0"
      },
      {
        "name": "utils",
        "version": "2.14.1",
        "built": "2.14.1"
      }
    ],
    "version": "2.14.1"
  }
}

Also, Happy New Year!

One of my major interests is programming in R. I took a few days off for Christmas break, and in between talking with family, watching old movies, opening presents, and spreading Christmas cheer, I decided it would be fun to use Chef to create a parallel computing environment suitable for use with R. As part of this project, I wrote:

• A Chef cookbook to install and configure R.
• An R package provider to easily install packages from CRAN.
• A Chef cookbook to install and configure MPI.

This blog post outlines some of the highlights of this fun little project. It isn’t a step-by-step tutorial for creating such an environment. I assume that already have:

• General familiarity with R and Chef. I talk about cookbooks, recipes, and roles and assume you know what to do with them.
• A means of provisioning computation nodes, each already configured with hostnames that are resolvable to network addresses by other computation nodes.

I used vagrant to provision a few VMs for testing these recipes and a small set of vagrant-specific cookbooks to ensure that each node had a resolvable hostname.

R Cookbook

The first item every node in the cluster will need is R. The R cookbook on the Opscode Community site is a bit out of date. I’ve created a newer version that can be found here:

http://github.com/stevendanna/cookbook-r

This cookbook does the following:

• Installs R from either the CRAN APT repository or from source depending on the platform.
• Defines a system-wide default CRAN mirror using an Rprofile.site template.
• Contains an R package provider that can install R packages available on CRAN.

Currently this cookbook is linux-centric and best suited for use on Ubuntu or Debian.

R Package Provider

My R parallel computing environment relies on the ‘snow’ and ‘Rmpi’ packages to manage communication with computation nodes. Further, I often need additional packages from CRAN when working with my cluster. The R package provider in my R cookbook allows for easy, automated installation of R packages. It was written using Chef’s Light-weight Resource and Provider DSL. The current version provides the minimal necessary functionality:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

action :install do
  execute "Install R Package" do
    command r_package_install(new_resource.package)
    not_if r_package_is_installed(new_resource.package)
  end
end

action :upgrade do
  execute "Upgrade R Package" do
    command r_package_install(new_resource.package)
  end
end

action :remove do
  r_package_remove = "remove.packages('#{new_resource.package}')"
  execute "Remove R Package" do
    command "echo \"#{r_package_remove}\" | R --no-save --no-restore -q"
    only_if r_package_is_installed(new_resource.package)
  end
end


# The following helper functions construct strings that can be run as
# Bash commands. For example, as the input of not_if or only_if
# statements

def r_package_is_installed(package_name)
  r_code = "if (any(installed.packages()[,1] == '#{package_name}')) { quit('no', 0, FALSE) }; quit('no', 1, FALSE)"
  "echo \"#{r_code}\" | R --no-save --no-restore -q"
end

def r_package_install(package_name)
  r_code = "install.packages('#{package_name}')"
  "echo \"#{r_code}\" | R --no-save --no-restore -q"
end

It supports three actions: :install, :upgrade, and :remove; however, it is not very feature-rich. A few features I would like to add include:

• The ability to specify the CRAN mirror or repository.
• The ability to install local packages
• The ability to install packages on a per-user basis.

Despite these shortcomings, the current provider works well enough for my purposes and is a good example of Chef’s high-level of ‘Whip-It-Up-itude.’ Within a few minutes, I was able to go from some quick-and-dirty R code to a Chef provider that I can iteratively improve as my needs become more complex. While I did not include it in the repository linked above, I was able to create a small R::snow recipe that does little more than use this provider:

1
2
3
4
5
6
7

R_package "snow"

case node['R']['snow']['cluster_type']
when 'mpi'
  include_recipe 'mpi'
  R_package 'Rmpi'
end

MPI Cookbook

The R package ‘snow’ can use various backends to communicate with computation nodes. The default method for most commands is “Message Passing Interface” (MPI). MPI is also generally useful outside of R as you can easily use MPI utilities such as mpirun to execute a command on multiple computation nodes and use MPI’s C libraries to write programs designed for parallel computation.

MPI has multiple implementation. I’ve written an MPI cookbook that can install openmpi on Ubuntu. It can be found here:

https://github.com/stevendanna/cookbook-mpi

This cookbook does the following:

• Installs openmpi on Ubuntu
• Constructs a list of computation nodes within your environment using search and then renders a default hostfile that is used by MPI commands such as mpirun.
• Sets basic configuration options that I have found useful.

While MPI does not natively contain the concepts of master and slaves (every MPI node can talk to any other MPI node), I found it useful to include the concept of a node’s “MPI role.” I use this role to populate the default hostfile, since there are often cases where you might want MPI installed on a machine within your environment but would not want that machine being used for computation.

Putting it all Together

I put this all together using two roles: snow_master and snow_slave

1
2
3
4
5
6
7
8
9
10
11

{
    "name" : "snow_master",
    "json_class" : "Chef::Role",
    "chef_type" : "role",
    "run_list" : [ "recipe[R]", "recipe[R::snow]", "recipe[mpi]", "recipe[ssh_known_hosts]" ],
    "default_attributes" : {
        "mpi" : {
            "role": "master"
        }
    }
}

1
2
3
4
5
6

{
    "name" : "snow_slave",
    "json_class" : "Chef::Role",
    "chef_type" : "role",
    "run_list" : [ "recipe[R]", "recipe[R::snow]", "recipe[mpi]", "recipe[ssh_known_hosts]" ]
}

By applying the snow_slave role to all of the nodes I wanted within my computation cluster and the snow_master role to the machine I will use to launch jobs in my cluster, I can quickly bring up a new cluster of machines suitable for doing parallel computation in R using snow.

The ssh_known_hosts recipe is directly from the Opscode community site and helps make ssh connections between the nodes a bit smoother.

Twelve Days of Chistmas

As an example of how easy it is to do simple parallel computation using snow, consider the following Chistmas-themed R function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

twelve_days_of_christmas <- function (n) {
  if (!any(1:12 == n)) stop("Not a day of Chirstmas!")
  LYRICS <- c("a Partridge in a Pear Tree.",
              "Two Turtle Doves",
              "Three French Hens",
              "Four Colly Birds",
              "Five Golden Rings",
              "Six Geese-a-Laying",
              "Seven Swans-a-Swimming",
              "Eight Maids-a-Milking",
              "Nine Ladies Dancing",
              "Ten Lords-a-Leaping",
              "Eleven Pipers Piping",
              "Twleve Drummers Drumming")
  append = c("st", "nd", "rd", rep("th",9))
  if (n > 1) LYRICS[1] = paste("and", LYRICS[1])
  paste("On the",
        paste(n, append[n], sep=""),
        "day of Christmas, my true love gave to me",
        paste(LYRICS[n:1], collapse=", "))
}

Assuming chef-client has been run on all of our nodes, we can simply log into our ‘snow master’, boot up R, and run the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

> library('snow')
> source('twelve_days.r')
> cl <- makeCluster(2)
> clusterApply(cl, 1:12, twelve_days_of_christmas)
[[1]]
[1] "On the 1st day of Christmas, my true love gave to me a Partridge in a Pear Tree."

[[2]]
[1] "On the 2nd day of Christmas, my true love gave to me Two Turtle Doves, and a Partridge in a Pear Tree."

[[3]]
[1] "On the 3rd day of Christmas, my true love gave to me Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[4]]
[1] "On the 4th day of Christmas, my true love gave to me Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[5]]
[1] "On the 5th day of Christmas, my true love gave to me Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[6]]
[1] "On the 6th day of Christmas, my true love gave to me Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[7]]
[1] "On the 7th day of Christmas, my true love gave to me Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[8]]
[1] "On the 8th day of Christmas, my true love gave to me Eight Maids-a-Milking, Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[9]]
[1] "On the 9th day of Christmas, my true love gave to me Nine Ladies Dancing, Eight Maids-a-Milking, Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[10]]
[1] "On the 10th day of Christmas, my true love gave to me Ten Lords-a-Leaping, Nine Ladies Dancing, Eight Maids-a-Milking, Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[11]]
[1] "On the 11th day of Christmas, my true love gave to me Eleven Pipers Piping, Ten Lords-a-Leaping, Nine Ladies Dancing, Eight Maids-a-Milking, Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

[[12]]
[1] "On the 12th day of Christmas, my true love gave to me Twleve Drummers Drumming, Eleven Pipers Piping, Ten Lords-a-Leaping, Nine Ladies Dancing, Eight Maids-a-Milking, Seven Swans-a-Swimming, Six Geese-a-Laying, Five Goldedn Rings, Four Colly Birds, Three French Hens, Two Turtle Doves, and a Partridge in a Pear Tree."

Conclusion

Overall this was a fun project to do over break. It took a handful of hours to complete, most of which was dedicated to a few problems not discussed in this post:

• Making hostnames resolvable and other network oddities within my Vagrant test environment.
• Deciding which implementation of MPI to use.

I personally would like to see further work done around using Chef to manage large R environments. Some work I hope to do in the not to distant future includes:

• Expand the R cookbook to better support OS X.
• Add support to detect the latest version when doing a source install.
• Expand the R package provider to include more of the features available when installing package from within R.
• Write an R Ohai plugin that would collect information about:
  • R version and capabilities
  • R packages installed

Happy Holidays!

I am in the process of rebooting my GTD workflow. Since I do most of my work in Emacs, I decided it was also time to refactor and improve my Emacs configuration. What was wrong with my configuration?

• I was living in the past (Emacs 23) and wanted to live in the future (Emacs 24).
• My configuration file was not under version control. And, because of the differences between Fedora and OS X, the configuration on my work laptop and personal laptop had drifted significantly.
• My .emacs file was poorly structured and easily broken.

I’ve documented my work on my Emacs configuration here, mostly as a means of reminding my future self what I did. Overall the experience was pleasant. Emacs 24 brings many long-needed features to Emacs.

The end result can be seen below.

Installing Emacs 24:

On Fedora, I installed directly from git:

1
2
3
4
5
6
7
8
9
10
11
12

# Install Dependencies
sudo yum-builddep emacs
sudo yum install
# Get source
cd ~/src
git clone git://git.savannah.gnu.org/emacs.git
cd emacs
# Install
./autogen.sh
./configure
make
sudo make install

On my Mac, I installed Emacs 24 via Homebrew:

1
2
3

brew uninstall emacs
brew install emacs --cocoa --use-git-head --HEAD
ln -s /usr/local/Cellar/emacs/HEAD/Emacs.app /Applications/

Rebuilding My Emacs Configuration

A number of high-quality, pre-built Emacs configurations are available online, including:

• Emacs Starter Kit
• Emacs Prelude
• Emacs Dev Kit

A few Opscoders use configurations based around Emacs Starter Kit. However, since I have fairly simple configuration needs, I decided to borrow a few ideas from each and forge ahead on my own. The result can be found in my emacs-config repository. It is still very much a work in progress.

I’ve structured my new .emacs.d directory as follows:

.
|-- init.el
|-- modules/
|-- snippets/
|-- themes/
`-- vendor/

• init.el: The main entry point of the Emacs configuration. I’ve taken a bit of code from the README of Emacs Starter Kit that automatically installs packages from ELPA.

• modules/: This is the heart of my emacs config. Any .el files in this directory are automatically loaded.

• snippets/: Used to store snippets for yasnippet. Currently empty.

• themes/: Color themes.

• vendor/: Any 3rd party Emacs packages that are not yet distributed via ELPA.

The auto-installation of packages is great for sharing my configuration between workstations; but, it can be brutally slow the first time you run it.

Highlights

The majority of my emacs configuration consists of minor customizations to popular, pre-built packages. The following are the packages that I use on a daily basis:

• ido-mode: I honestly don’t know how I used Emacs before discovering ido-mode.

• org-mode: Org-mode is my go-to application anytime I need to take notes in a meeting, outline a new project, or author a simple document.

• ess: Emacs Speaks Statistics (ESS) is a must-have for any user of R and Emacs. Kieran Healy has created a repository for ESS, making ESS easily installable via Emacs 24’s package management features.

• auctex: AucTeX provides a number a features useful to those who love LaTeX. If you haven’t fallen in love with LaTeX yet, I highly recommend trying it!

In addition to the tried-and-true packages above and a handful of popular programming modes, I added the following packages to my standard configuration:

• Gist: I use Github’s gists all the time; I can’t believe I didn’t look for this before.

• Deft: Deft allows you quickly create, edit, and view plain-text notes. I’ve configured it to use org-mode as its default text-mode and hope to use it to easily collect tasks throughout the day.

• hippie-expand: I previously used predictive-mode extensively, but eventually found it too slow to use productively. So far, hippie-expand has been filling the auto-completion void that I’ve been feeling since remove predictive-mode.

All of these packages are either shipped with Emacs or available as packages using the new package management features in Emacs 24. I was initially skeptical of the package management features. However, I’ve found that a large amount of the complexity in managing my previous configuration was the result of manually managing packages. With this complexity removed, moving to Emacs 24 was relatively painless.

I was also pleased to see improvements to clipboard integration. My old .emacs file had a number of ugly workarounds for properly interacting with the clipboard on multiple platforms. I’ve found this completely unnecessary in Emacs 24.

The only problem I encountered during this transition was finding a suitable color theme. Low-contrast color themes such as zenburn seem to be in vogue at the moment. I prefer higher-contrast themes. In Emacs 23, I used dark-laptop, but it appears that this theme has not yet been ported to Emacs 24’s built-in color theme support. My current theme is a small color theme based heavily on dark-laptop.

Deploy!

I’ve deployed my new configuration to all of my workstations via a simple git pull. My hope is that a few weeks of real-world use will find the rough edges that remain. Comments and suggestions welcome!

In preparation for moving more of my personal laptop’s setup into git and Chef, I spent a few minutes this morning taking stock of what I had floating around my home folder on my personal laptop.

I found a small script I wrote when working on my thesis to help me avoid distractions. Since I thought others might be interested, I put it up on Github and added a bit more user friendliness:

gsd: A small utility to get stuff done

gsd helps you get work done by replacing your host file with a custom host file kept in ~/.gsd/. The custom host file prevents you from visiting sites that distract you from the work you need to get done.

To start using the filter, simply clone the git repo and install the script

> git clone git@github.com:stevendanna/gsd.git
> cd gsd
> sudo make install

Then, enable the filter

> sudo gsd enable

Finally, add sites to the filter and reload when you find yourself being distracted by them:

> gsd add lkml.org
> sudo gsd reload

With a bit of searching, one can find a number of these types of scripts online. My favorite that I’ve come across so far is Get Shit Done, which works on Windows and includes some logic to restart your systems networking services.

Introductory blog posts are awkward. Come back in the future to read less awkward blog posts. Possible topics for such blog posts include:

• Linux and Unix systems administration
• Technical writing
• Programming projects I’m working on in my free time
• Free software
• Chef and Opscode
• Running
• Cooking

I recently finished school and am beginning a career in technology. Most of my formal education is in Mathematics and Economics; I hope to chronicle parts of my informal, on-the-job education here.

I live in Seattle, WA with my wife Theresa. I enjoy running, cooking, and movies and may occasionally write posts on these topics as well.

I work for Opscode, Inc. The opinions expressed here are my own and do not reflect those of my employer. Or, in other words,

1

   require 'disclaimer'