Overview

This page describes how to launch API-driven "Cloud" servers with Knife and automatically run Chef on them.

This page covers the general use of the available cloud plugins. Reference documentation for the individual plugins is maintained in the README of the plugin's code repository, linked below where available.

Currently supported services (version 0.10.0 with plugins):*

Amazon EC2
Rackspace Cloud
Openstack
Bluebox
Terremark vCloud
Eucalyptus

These are supported via sub-commands in knife that use the Ruby library fog. During the "bootstrap" process that runs Chef, knife will SSH to the systems, and uses the net-ssh-multi library. Finally, knife uses colorized output via highline.

You will need to install the fog, net-ssh-multi and highline gems / libraries on your local management workstation where you use knife.

On Ubuntu you will also need to install some additional apt packages in order to compile the nokogiri gem (a fog dependency):

Once complete, you'd move on to continuing to setup knife with your "cloud" credentials.

Cloud Credentials

Add the Cloud Credentials to your knife.rb. They are specifically named for each of the service providers Knife supports as keys under the knife configuration option.

Add the appropriate lines for your preferred Cloud Computing service to your knife.rb.

Chef 0.10: Command examples on this page The command examples on this page now use Chef 0.10's knife plugin syntax. See the Opscode Blog Post for an overview.

Cloud Sub-commands

Cloud sub-commands for knife are available as Knife Plugins. The following plugins are provided by Opscode as RubyGems. Install the desired plugin to make the sub-commands described below available.

• knife-ec2
• knife-rackspace
• knife-openstack
• knife-bluebox
• knife-terremark
• knife-eucalyptus

The sub-commands for cloud management in knife follow the pattern:

Where service is ec2, rackspace, openstack, bluebox, openstack, or terremark, euca each of the services has a server create sub-command that uses their API to launch a new instance of the specified flavor or image with a run list.

Enclose the run list items in quotes, as the brackets for role[thing] or recipe[thing] may be interpreted by your login shell. Multiple items can be specified as a comma separated list.

Examples

EC2 Note: The AMI used above is just an example. You should use the most current AMI's for your OS. For instance: the Ubuntu AMIs are found at their site, in the version subdirectories under "releases".

Node name If no node name is specified with the -N or --node-name flags, knife will automatically generate a node name based on information supplied by the cloud provider, such as a numeric server id. These machine generated names are generally less descriptive than is usually desirable.

Syntax for Run List in Chef 0.10 In the knife plugins used for these cloud computing commands with Chef 0.10, the run list must be specified with the -r option. Earlier versions of Chef did not use -r.

See the Knife documentation for more information on the syntax for the subcommands and further options.

Instance Bootstrap

The server creation sub-command bootstrap the instance. When the instance is launched, knife determines whether it is available, and once it is, ssh's to the instance using the specified username (or your local user if none specified) and does the following:

1. Installs Ruby and packages to support installing RubyGems with native extensions.
2. Installs RubyGems from source.
3. Installs Chef from RubyGems.
4. Creates the /etc/chef directory.
5. Writes your validation certificate to /etc/chef/validation.pem.
6. Writes an /etc/chef/client.rb config file.
7. Writes a JSON file, /etc/chef/first-boot.json with the roles and recipes specified as the run list.
8. Executes chef-client with the first-boot.json, connecting to the server in the client.rb.

Once the Chef run is complete, the new instance will be registered with the Chef Server and be saved as a node object. This process does not set up the Chef Client service, you'll need to add a recipe in the run list that does this.

You can see the config file that will be copied by creating one with:

The client.rb and validation.pem will be copied to the specified directory, in this case client-config.

If The Run Does Not Complete

The run list for the first run of a newly provisioned node is passed in by knife as a file with the special command line:

At the end of the first run, the node object is saved with this list, shows up on the console and knife commands, and is used on subsequent chef-client runs. If the run does not complete, (the instance is created but the installs are not completed), then the node with the run list applied to it is never saved.

Run the above command if this occurs. The normal state is for the first chef-client run to complete - an incomplete run is the sort of thing that will happen while debugging changes and new recipes. A typical cause is an unmet dependency in cookbooks. Review the metadata.rb files of the cookbooks in your base role, or any other roles being established in the initial run for any dependencies.

Bootstrap Subcommand

Currently, the bootstrap subcommand is used on EC2 and Rackspace instances. This isn't cloud-provider specific, and can be used on any kind of system, like a bare metal server or a VMware machine.

The bootstrap sub-command is not used by the terremark or openstack or euca server creation yet:

• CHEF-1731 is an open ticket for Terremark to have this functionality
• Use the Deploying OpenStack with Chef Guide for Openstack

SSH Configuration

The bootstrap process uses SSH, so you'll need to configure your system to SSH in a particular way. The easiest way to make sure that SSH uses your Cloud service's SSH key is to add it to your SSH key agent. For example, if you have saved the private key for EC2 to ~/.ssh/ec2_ssh_key:

On Mac OS X Snow Leopard, ssh-agent is already running by default when you use Terminal.

You also need to specify the current user to SSH as:

• On Canonical's Ubuntu AMIs, this is ubuntu.
• If you're using an image that uses a different user, specify it with -x in the server create command.

Connecting to your cloud from its external FQDN

By default, knife will use the internal FQDN to connect to nodes. When connecting from outside of your cloud you may need to force this to use the external FQDN. You will want to avoid doing this unless necessary, as this will route all traffic out through the NAT infrustructure and could cause performance hits or additional usage charges.

If you're sure you want to do this, you can do it with the -a option, such as in this example:

Deleting a server

To delete a server once you are done using it, you will want to use this command:

The instance id can be located with the command 'knife <plugin name> server list' or 'knife status'. For example, you could delete a server using these commands if you were using the rackspace plugin:

Make sure that you also clean up the associated 'node' and 'client' objects from hosted chef:

Knife Exec

Managing API Clients With Knife