This installation guide describes how to get up and running with Chef as quickly as possible, and will walk you through adding a client node on Amazon's EC2 to your organization in Hosted Chef.

You can run your own Chef Server, but for this guide we will use Hosted Chef as the Chef Server, so you don't have to set one up.

Clients are able to run chef-client, but will be unable to download, edit, and upload recipes with knife.
Management Workstations are able to download, edit and upload recipes, and can be configured to work as clients as well so they can run chef-client.

You can find directions for setting up workstations in the Fast Start Guides:

Assumptions
Steps

Step 1: Install the Knife-EC2 Plugin
Step 2: Configure AWS settings and the knife.rb file
Step 3: Bootstrap the EC2 Instance
Step 4: Verify the install completed
Step 5: Manage recipes on the new client node
Step 6: Run chef-client on the new client node
Step 7: Running chef-client as a daemon

If something goes wrong
What should I look at next?

Assumptions

In order to make this as quick as possible, we make some assumptions. If your system does not meet these assumptions, you will need to use the Installation instructions that apply for the "flavor" of Chef you are installing.

You already have Management Workstation configured

In this guide we will be creating an EC2 Instance and setting it up as a client. We assume you already have another node configured as your Management Workstation.

If you do not already have a workstation configured, you can find directions in the Fast Start Guides:

You'll want to follow one of these guides through at least step 4 and confirmed the knife commands worked properly. Once this is completed, you can return to this guide to set up another node as a client.

All of the commands in this guide will be run from the Management Workstation using knife commands to configure the client node remotely.

Amazon's EC2

This guide assumes you are using AWS EC2 to set up instances. You should already have an account created with AWS, and this guide assumes that you do not have the EC2 instance configured yet. If you've already created the EC2 instance, you can either delete it or use the regular Client Bootstrap Fast Start Guide to configure it.

Operating System

Chef runs on many popular Unix and Linux platforms as well as Mac OS X and Windows.

Management Workstation OS: We will describe how to set up using Ubuntu 10.04 as the workstation by default, but these are general directions that will work on most workstations.

Client EC2 Instance OS: There are directions included on setting up the EC2 client as Ubuntu, Debian, RHEL, or CentOS but as long as you have an AMI and a custom bootstrap file that works you can use any OS as the EC2 client.

Steps

We will complete the following steps from the Management Workstation:

1. Install the Knife-EC2 Plugin
2. Configure AWS settings and the knife.rb file
3. Bootstrap the EC2 Instance
4. Verify the install completed
5. Manage recipes on the new client node
6. Run chef-client on the new client node remotely
7. Running chef-client as a daemon

When you are done with this guide you will have a workstation configured to work with EC2. You will also have a node on EC2 added as a client to the Hosted Chef organization and AWS, and the getting-started recipe will have been run on it. It will also explain the basics of how you can manage this new client node remotely from your workstation.

For more in-depth information, please review the Launch Cloud Instances with Knife page.

Step 1: Install the Knife-EC2 Plugin

If you have already installed and configured the knife-ec2 plugin then you can skip to Step 3. All of the steps in this guide are done from the management workstation you've already set up.

Install dependencies

First you will need to install the dependencies needed for the knife-ec2 plugin on your Ubuntu management workstation where you use knife:

More information on why these are needed can be found on the Launch Cloud Instances with Knife page.

Install the plugin

Once the dependencies are installed you can install the knife-ec2 plugin with this command:

Step 2: Configure AWS settings and the knife.rb file

To authenticate with AWS, you'll need to set up a few things within your AWS account and on your local workstation.

Configure your Amazon security group

You will need to configure SSH access to your EC2 instances from external IPs, so knife will be able to reach them. You can do this by following these steps:

Log into the AWS Management Console and select the EC2 tab
Click on "Security Groups" from the menu on the left side of the screen
Select the default group. (You can select or create a new group in this step, but you will need to remember to add it to your knife ec2 server create command later with -G as it will use 'default' otherwise.)
Click on the "Inbound" tab
Choose "SSH" and enter "0.0.0.0/0" as the source to accept all traffic, then click on "Add Rule":

Create your SSH key pair for EC2 and save your identity file on the local workstation

Log into the AWS Management Console and select the EC2 tab
Click on "Key Pairs" from the menu on the left side of the screen
Select the "Create New Pair" button
Enter a name for your key pair and create it, this will download a .pem file to your local machine
Save the key pair to ~/.ssh/ and remember the name of it
Make sure ssh-agent is running. You need to run this from the console, not from Putty or other access. To start it:

If you do not use exec, you will get the following error message :
Could not open a connection to your authentication agent
Change the permissions of key pair file to write only for you. To do so please type this, substituting KEY_PAIR_NAME for the name of your key pair:
Now you can add this key pair to ssh with this command, substituting KEY_PAIR_NAME for the name of your key pair:

Note: This step is optional, if you skip it you'll need to remember to add the path to the identity file to your knife ec2 server create command later with -i.
You will also want to add this line to the bottom of your knife.rb, substituting KEY_PAIR_NAME for the name of your key pair:

Note: This step is optional, if you skip it you'll need to remember to add the name of your identity file to your knife ec2 server create command later with -S.

Configure knife.rb with your AWS Cloud Credentials

Add your AWS Cloud Credentials to the bottom of your knife.rb:

Step 3: Bootstrap the EC2 Instance

Before bootstrapping the instance you'll first need to choose an AMI to use. You can use an AMI you've created, an AMI found online, and we also have an unsupported AMI List for Developers that is useful for testing.

You will need to know the login name for the AMI, as well as the bootstrap file you want to use.

Where are the bootstrap files located?
The knife bootstrap sub-command looks in three locations for the template to use:

bootstrap directory in the installed Chef Knife library (Location varies depending how you installed Chef).
bootstrap directory in the $PWD/.chef/bootstrap, e.g. in ~/chef-repo/.chef/bootstrap.
bootstrap directory in the users $HOME/.chef/bootstrap.

You can then launch the instance with a command like this, substituting in the AMI ID, the LOGIN_NAME and the BOOTSTRAP_FILE as needed:

Knife will automatically generate a node name based on the instance id. If you'd like to specify a name you can do this with the -N flag.

Further examples using the knife-ec2 plugin
If you were creating an Ubuntu instance your command may look like this:

Note that you do not need to use -d with ubuntu as this will be used as the default.

If you were creating a CentOS instance your command may look like this instead:

Note that you do not need to add -x root to this command as this will be used as the default.

If you wanted to launch a large instance, you can specify this with the -f switch. Here's an example bootstrapping a 64-bit RHEL instance:

Here's an example where it's bootstrapping an instance named "server01" with the base and webserver roles. In this example it is also being added to the "www" security group:

Note that you do not need to use -d with ubuntu as this will be used as the default.

If you did not setup the SSH key pair/EC2 identity file in step 2, you can specify the location of your .pem file by adding the -i and -S switches to your command. You'll want to substitute KEY_PAIR_NAME for the name of your key pair:

Note: The AMIs used above are just an example. You should use the most current AMIs for your OS. For instance: the Ubuntu AMIs are found at their site, in the version subdirectories under "releases".

Configuration options

You can see the full list of options that you can add to the knife.rb config file by reading the readme for the knife-ec2 plugin. You can see the full list of switches available at the command line by using this command:

Here is a partial list, of the commands you are most likely to need:

-I - The AMI for the server
-x - The ssh username, by default it will use "root"
-d - Bootstrap a distro using a template, by default it will use ubuntu
-N - The Chef node name for your new node, if this is not specified it will use the instance name. For example "i-91f66cf4".
-r - Comma separated list of roles/recipes to apply. For example: knife ec2 server create -r 'role[webserver],recipe[users]' ...
-G - Comma separated list of security groups for this server, by default it will just use "default"
-f - The flavor of server (m1.small, m1.medium, etc), this will default to m1.small if not specified
--region - Your AWS region, by default it will use us-east-1
-Z - The Availability Zone

Expected Output

This is the output you can expect after entering this command. First you will see it create the instance:

It may pause at this step for a few minutes while SSHing into the instance. If it gets stuck at this step, make sure that you've properly set up the security group and if you are not using the "default" group you'll want to specify it with the -G switch. You'll also want to confirm you've followed all of the steps to create and save your SSH Key.

Once it has logged in it'll start running the bootstrap script, installing Ruby and Chef-client:

After a few minutes you should see it report that the Chef install was successful, and it'll run chef-client for the first time:

You should see the current version of chef in the output of this command, Chef 0.10.10. Now chef-client has been installed on the EC2 Instance and it is ready for recipes to be added to it. You can also add to the run_list when creating the instance by using the -r switch.

This process does not set up the Chef Client service as a daemon, you'll need to add a recipe in the run list that does this if needed. Step 7 also explains this further.

Step 4: Verify the install completed

Now that the client has registered with Hosted Chef, it should show up on the ec2 server list as well as the node list. You can use knife to display a list of ec2 servers:

Instead of i-XXXXXXXX you should see the instance id of your new node here. If it does not show up on this list, that means the EC2 instance was not created.

You can also check that the client shows up on the node list with Hosted Chef:

Instead of NODENAME you should see the name of your new node here. If it does not show up on this list, that means it was unable to finish convergence and save the node at the end of the chef-client run. You can review your logs, or SSH into the node and try running sudo chef-client -l debug to get more information from the debug logs on why this failed.

The Client install is now complete

At this point, your instance has been created and is now configured as a client. The next steps explain how to add a recipe and run it on this node remotely from the workstation, as well as run the client as a daemon. If you are already familiar with Chef, you can skip the rest of the guide and it will have no impact on your install.

If you need to set up another EC2 instance, you can just repeat Step 3 as many times as needed. If you need to delete the instance you've created, there are instructions on how to do this on the Launch Cloud Instances with Knife page.

Step 5: Manage recipes on the new client node

Now that your client is configured, you'll need to know how to add a recipe to it so you can start configuring it. This step will walk you through adding the getting-started recipe to it as an example. These steps are also ran from the management workstation.

We will need to download the getting-started cookbook from the Community Site:

And then upload the recipe to Hosted Chef so it is available for our nodes:

Once this is completed, you can add this new recipe to the new nodes run list:

Step 6: Run chef-client on the new client node

You can just SSH into the node at this point and run sudo chef-client, but it may be easier to run this command remotely from the workstation with knife. This way, once you have multiple nodes added you can run chef-client on all of them at once if needed by using a wildcard (*) as the NODENAME.

You can run chef-client remotely from the management workstation with knife with the SSH subcommand:

By default, knife will use the internal FQDN to connect to nodes. When connecting from outside of your cloud you may need to force this to use the external FQDN. You will want to avoid doing this unless necessary, as this will route all traffic out through the NAT infrastructure and could cause performance hits or additional usage charges.

If you're sure you want to do this, you can do it with the -a option, such as in this example:

You do not want to edit the -a ec2.public_hostname portion when using this, as this is telling it to connect using the public_hostname attribute instead of the specific FQDN to connect to. For more information on the knife SSH subcommand, review the Knife Built In Subcommands.

Running chef-client will run the getting-started recipe you've added to the client node's run_list. The getting-started recipe added a template to the client node in ~/chef-getting-started.txt. Once Chef has completed on the client, you can log into the client and look at the template:

Your output will of course be dependent on the data we have stored about your node.

You can also see what code the recipe contained by opening it with a text editor from the Management Workstation:

For more information on editing and creating recipes, see the Cookbook Fast Start Guide.

When you are ready to start configuring this node, you can remove this test recipe from the run_list with this command:

Step 7: Running chef-client as a daemon

You can set up chef-client to run as a daemon, so it will periodically run chef-client and get any updates you have submitted without needing to SSH to it every time. The easiest way to do this is with the chef-client cookbook. This will configure the chef-client as a daemon and add a time splay, you can add it the same way you added the getting-started recipe:

Once it is added you'll need to run chef-client on the node once more for the recipe to configure the server:

Next time you bootstrap a new client node, you can include the chef-client cookbook in the Knife Bootstrap command from Step 3 if desired. This will add it to the run_list for the node and run it when installing Chef. You can do this with the -r switch:

You can read the cookbook's README to get more information on altering the default attributes.

If something goes wrong

It only takes a small change to a dependency or environment to cause these instructions to not be exact. If you need to delete the instance you've created, there are instructions on how to do this on the Launch Cloud Instances with Knife page.

If you want to troubleshoot the issue, first check the Installation page and refer to documents specific to your OS. You may also want to try SSHing into the new instance and seeing if sudo chef-client -l debug gives you any additional debug information. If this doesn't help, head over to the Support page for information on where and how to find help from Opscode and the rest of the Chef community.

What should I look at next?

More information on Knife Cloud Plugins...

We begin with an Architecture Introduction, covering the basic functions of the Chef Server, Nodes, and Chef Workstations and how these components communicate.
Then an overview of the Core Components, which introduces all of the aspects and components of modeling your infrastructure, Configuring Nodes and Managing Chef.
Onward to Cooking School and begin an Introduction to Cookbooks and More. Cookbooks are Chef's fundamental units of distribution, the way Chef users package up, distribute and share configuration information. Recipes, Resources, Attributes, Roles and more are also introduced.
The final basic section is an Introduction to Search and Data Bags, two of Chef's most powerful features allowing you to dynamically change the configuration of your infrastructure based on data.

Chef Architecture

With all that under your belt, it's time to tackle the dirty secrets of what's happening behind the scenes with Chef Architecture. We'll give you the scoop on Chef's Authentication and Authorization system and go over the Anatomy of a Chef Run, where we go in-depth with the process by which your systems get configured. From there we'll review all the executable parts of Chef - Chef Client, Chef Solo, Chef Server, Chef Indexer, and Server API and Cookbook Site API interaction.

Or view video tutorials and training...

Guides

We have a number of Walkthrough Guides available on the building of common stacks - including: Rails, Java, LAMP, and more. There are also How To Guides such as Deploying OpenStack with Chef, Guide to Creating A Cookbook and Writing A Recipe, and more - as well as a full 2.25 hour Webcast of a Chef Tutorial Session. These should be good resources for you as you move forward in the automation of your infrastructure.

Installation

Cookbook Fast Start Guide

EC2 Bootstrap Fast Start Guide