Salty Infrastructure, Part One

I recently attended PyCon 2013 out in beautiful California, and as expected I absorbed quite a bit of knowledge from all of the great speakers there. Yeah, I know that just about the only thing that people remember from this recent PyCon involved crude jokes, public shaming, and a crazy Internet storm of hate. It’s unfortunate that all of that occurred, but that’s all I’m going to say on the matter. Not “choosing a side” or anything of the sort. Instead, I’d like to talk about something positive that I got out of the trip.

While I’m currently a backend Python developer for my current employer, in the past I’ve been a Linux system administrator. I’ve always leveraged that experience along my various jobs throughout my career and continue to find myself interested in infrastructure development, maintenance, and administration. Just something about that line of work that I always find incredibly appealing. This all lead to me attending a great talk by Kate Heddleston titled “Chef: Automating web application infrastructure”.

Obviously, as the name implies, this talk covered Chef. It’s hard to be in the development or system administration communities and not at least be aware of Chef and/or Puppet. Now, both of these are great tools that I completely recommend for anyone to check out. I’ve played a bit with them both, but Kate mentioned another tool during her talk that wasn’t quite “mature” in her eyes. That tool was SaltStack, and it’s written in Python. I was immediately interested to see what this group was up to given that it’s relatively new and written in Python. I’ve spent a couple of weeks off and on using it to bootstrap the servers for a new side project of mine and would like to share my initial thoughts. I hope to make a couple of posts about it as I learn more and get more comfortable. For now I’ll warn that what I share here should not be seen as “Best Practices” or anything such as that.

As I started to sit down and sink my teeth into Salt, I had a pretty distinct goal in mind. I’m working on a new side project with a friend of mine, and I wanted to quickly be able to spin up new servers when needed. Frankly, scaling isn’t an issue at all for us yet, but half the fun of side projects is learning something new. I also wanted to run this on EC2. I made a quick list of requirements that my server(s) were going to need:

  • Python
  • Django
  • Virtualenv
  • nginx
  • gunicorn
  • postgresql
  • supervisord
  • misc packages (git, zip/unzip, imaging libs)

Naturally, I started with the Official Salt Walkthrough. This is a pretty great method of getting yourself rolling. After going through it, you should have yourself with a “salt master” and a “salt minion”. Their names are pretty self explanatory. The “master” is the center of control, managing keys, issuing commands, and so on. Meanwhile, minion(s) are the nodes that you’ll be configuring as web servers, database servers and so on. Of course, the “master” server can also have a “minion” running on it. In my current situation, I just have 1 server running, and thus it’s a master and a minion.

Getting through the installation and initial walkthrough of Salt is pretty straightforward. I’m not going to cover that aspect right now, and instead focus on the seemingly less talked about Salt Cloud. My only real complaint so far with this tool is the documentation, but it seems to be getting some love as we speak. What they have up now in terms of documentation is better than what I recall when I first started tinkering with the tool. Still, you’ll notice that the docs I linked you to don’t have an “installation” guide. This can be handled in a couple of different ways.

The simplest way is through your operating system’s package manager. If you followed the walkthrough earlier by wgetting a script and piping it through sh then you’re already halfway there. For instance, on Ubuntu you should then be able to do:

sudo apt-get install salt-cloud

If you decided to go through a less conventional mean, then there’s always pip:

sudo pip install salt-cloud

I will say that you’re really best off if you install salt-cloud on whichever server you plan on being your “Salt Master”. Doing so will ensure that newly spun up nodes will automagically have their keys accepted, and they’ll be instantly a proper part of your Salt ecosystem. You certainly can spin up servers from servers other than your master, but you’ll eventually have to pop into your master node to accept the Salt keys.

After you have salt-cloud installed, you’ll want to edit /etc/salt/cloud on your Salt Master. Now the changes you make to this file will depend on your cloud provider of choice. They currently cover a variety of providers, but their documentation really only discusses AWS, Rackspace and Parallels. I’ll be discussing AWS strictly, as that’s who I use, but there aren’t many differences.

Assuming you have /etc/salt/cloud open, the first thing you’ll want to do is find the minion option in the config. You’ll want to tell it who the master is, and then you’ll also want to set the provider, like so:

provider: aws

Now we’ll want to find the AWS section in the config, which will look like so: YOUR_AWS_ID
AWS.private_key: /PATH/TO/PRIVATE_KEY.pem
# Specify whether to use public or private IP for deploy script
# private_ips or public_ips
AWS.ssh_interface: public_ips 
AWS.location: us-west-2
AWS.ssh_username: ubuntu

That’s pretty much all there is to the configuration. The rest of that configuration file is mostly for configuring other cloud providers. Now we need to define a cloud profile. Open up /etc/salt/cloud.profiles in your editor of choice, and add something like the following:

    provider: aws
    image: ami-YOURIMAGEID
    size: Micro Instance
    ssh-user: ubuntu

Now you should be able to run the following command:

salt-cloud -p base_aws NAME_TO_GIVE_INSTANCE

Afterwards, you should have a brand new EC2 instance running. If you ran that command from your “Salt Master” then the new node should already be configured as a minion that’s ready to receive commands.

You can find a Getting Started With AWS tutorial via Salt’s documentation, which can provide you with more fine tuning options than my bare bones tutorial provides here. I hope, in the coming weeks, to get more in depth with some of the things I’m using Salt for. For now, feel free to take a look at my Salt configurations in my GitHub Repo.