Jenkins Delivery

At Sky Betting and Gaming, we use Chef and Jenkins extensively to manage our infrastructure and build our software. 

We are starting to use public cloud services where it can either help us with scale or increase the speed of our service delivery. We wanted a common toolchain for our software delivery and operational developers across both our on premises VMWare estate, and the public cloud. 

Enter Jenkins Delivery, our take on continuous delivery using Jenkins 2, Chef, Test Kitchen and Docker. It is a continuous delivery platform with deep Chef and Chef Supermarket integration. It encourages and enables building and testing software right alongside configuration management, compliance and the deployment pipeline itself. It makes heavy use of the Pipeline features shipped with Jenkins 2.0, and includes custom extensions to the Pipeline DSL that integrate Jenkins with Chef.

What follows is a walkthrough of standing the system up in a VPC, and an example of using it to build and deploy software in an EC2 instance. The repositories for what follows are all available publically, so feel free to try this out and create your own CD platform!

Pre-Requisites

  • AWS Account

First you will need an Amazon Web Services account. Your API key should be in ~/.aws/credentials in Amazon's shared credential format.

  • The ChefDK installed

The most recent version of the ChefDK for your platform should work. This has been tested using ChefDK 0.14. Download and install the ChefDK from here: https://downloads.chef.io/chef-dk/

  • IAM Role

You will need an IAM role configured in AWS that has permission to modify Route53 DNS records and permission to create EC2 instances

Creating the IAM Role

  1. Select 'Identity and Access Management (IAM)'
  2. Select Roles
  3. Create new role
  4. Call it 'jenkins-delivery'
  5. Select a role type of 'Amazon EC2'
  6. Attach policies 'AmazonRoute53FullAccess' and 'AmazonEC2FullAccess'
  7. Create Role
IAM role setup

IAM role setup

  • AWS Virtual Private Cloud (VPC)

You will need an AWS VPC to create the system in.

Creating the VPC

  1. Select the destination Amazon region, and select Virtual Private Cloud (VPC)
  2. Start the VPC configuration wizard and choose 'VPC with Single Public Subnet'
  3. Choose a VPC name. Make sure 'Enable DNS hostnames' is enabled.
  4. Go to 'Subnets' and select the public subnet you just created with the wizard. Select 'Subnet Actions' and choose 'Modify Auto-Assign Public IP', make sure it is enabled.
Auto-assign public IP

Auto-assign public IP

  • EC2 Security Groups

You will need two EC2 security groups for the delivery system. One to allow inbound access from your administrative location (Access to the Jenkins, Chef and Supermarket UI/APIs) and one to allow access between the delivery system and instances you create using the system.

Creating the Security Groups

  1. Select the region in which you are deploying the system, and EC2
  2. Security Groups
  3. Create, name "admin-access' and description 'Administrative access to Jenkins Delivery'. Make sure the group is created in the correct VPC. Create.
  4. Select the new group, allow inbound access on ports 22 and 443 from your administrative address range.
  5. Create, name 'delivery-system' and description 'Access internal to Jenkins Delivery'. Make sure the group is created in the correct VPC. Create.
  6. Select the new group. Add a new inbound rule, allowing ALL TCP and set the source to the security group created in the previous step
  7. Add a second rule to 'delivery-system'. Allow HTTPS from 0.0.0.0/0.

This is for annoying OAuth related reasons. The Supermarket server needs to be contactable on its public IP address from the other instances. We don't know what the public IP addresses will be yet....

You should further restrict this once the system is built by restricting HTTPS access to just your administrative location, and the three Delivery instance public IP addresses. 

delivery-system security group

delivery-system security group

admin-access security group

admin-access security group

  • Delegated Route53 Domain

You will need a DNS zone that is delegated to Amazon's Route53 service. In the example we are using the subdomain test.gharris.uk

Delegating the Domain

  1. In the AWS console, select Route53
  2. Select 'Hosted Zones'
  3. Create Hosted Zone
  4. Choose your domain name and add a comment. The type should be 'Public Hosted Zone'
  5. Click Create.
  6. Save the Route53 Zone ID for later on
  7. Make sure the name servers for the newly delegated subdomain are correctly set to Amazon's
Route53 delegated subdomain

Route53 delegated subdomain

These are the pre-requisites for building the system. Once these are in place, you can continue.

building the system

 

Fork the Jenkins Delivery Repository

First, you will need to fork the Jenkins Delivery git repository. You can use the git server of your choice, in this example we're using public Bitbucket. The repository is available here:

https://bitbucket.org/harriga/jenkins_delivery

Fork this and then clone your repository to your local system. Then:

cd jenkins_delivery

Create SSH Keys

The system will need some SSH keys in order to operate. Keys are needed for pushing updates back to your git repositories, SSH access to newly created EC2 instances and SSH orchestration within the environment once it is running. Create these SSH keys:

  • An EC2 SSH Keypair
  1. Select the destination Amazon region, and select EC2
  2. Select Key Pairs
  3. Create a Key Pair and call it 'jenkins-delivery'
  4. Save the downloaded key to:
jenkins_delivery/keys/aws
  • An SSH key for pulling and pushing updates to git repositories
ssh-keygen -f keys/git
  1. Give this key access to clone your jenkins_delivery repository
  2. Instructions for Bitbucket: https://confluence.atlassian.com/display/BITBUCKET/Use+deployment+keys
  3. Instructions for Github: https://developer.github.com/guides/managing-deploy-keys/
  • An SSH key for orchestration inside AWS post provisioning
ssh-keygen -f keys/knife

Populate Data Bags

Included in the jenkins_delivery respository is a convinience script that will convert the three SSH keys stored in jenkins_delivery/keys into JSON format and update the corresponding data bags in data_bags/keys/. Run this script to update the supplied (empty) data bags:

Updating data bag content

Updating data bag content

Configure Delivery Environment

Before installing the system, the install needs to be customised. Edit the management environment configuration to match your install

vi envs/management.json
"default_attributes": {
 "resolver": {
 "nameservers": ["10.0.0.2"]
 },
 "route53_domain": "your.route53.domain",
 "route53_zoneid": "XXXXXXXXXXXXXXX",
 "delivery_ssh_uri": "git@bitbucket.org:<user>/jenkins_delivery.git"
},

The name server on Amazon networks generally lives on .2 in the same subnet, so you shouldn't need to change this unless you changed the default IP allocation when creating the VPC using the VPC creation wizard.

Set Up Local Test Kitchen Configuration

Test Kitchen needs to know some specifics about your Amazon setup before installing. Configure .kitchen.local.yml to match your environment. See .kitchen.local.yml.example in your jenkins_delivery repository

 

Setup ChefDK

Run the following to setup your ChefDK environment (paths to gem, bundler, test kitchen, chef, etc)

eval $(chef shell-init sh)

Install the kitchen-sync rubygem, which gives Test Kitchen rsync transport support. Also install the Test Kitchen EC2 driver:

gem install kitchen-sync kitchen-ec2

Setup SSH Agent

Set up a local SSH agent and add the AWS EC2 SSH key to it

chmod 600 ./keys/aws
eval $(ssh-agent)
ssh-add ./keys/aws

Create the System

We are now ready to install the system. Running the Test Kitchen converge will create 3 EC2 instances. One Chef server, one Chef Supermarket server and one Jenkins instance to tie everything together. Although in this example we're creating new Chef and Supermarket servers to integrate with, it is possible to use Jenkins Delivery with existing Chef and Supermarket servers. This is left as an exercise for the reader, though!

kitchen converge

Note: If you see something similar to the below either a) you forgot to add your EC2 ssh key to your ssh-agent, or b) You were unlucky and Kitchen failed to authenticate with the instance before cloud-init sorted out SSH keys on Amazon's side. tl;dr, press enter and Kitchen will re-auth

EC2 instance <i-0f27541d512486357> ready.
centos@ec2-52-58-222-128.eu-central-1.compute.amazonaws.com's password: 

Time passes. The amount of time is variable depending on a few factors. I'm using an instance type of m4.large and creating the three instances usually takes around 30 minutes. When it has finished, it should look something like this:

 

A lot has happened with the Test Kitchen converge.

It has created Chef, Chef Supermarket and Jenkins 2.0 instances, configured them and started the Jenkins pipeline called InitialSetup. This pipeline configures the first Chef organization called 'management', bootstraps all 3 instances and sets them up as Chef Server clients in the management organization. This means that the delivery system itself is managed in the same way as any other environment that is managed by Jenkins Delivery. 

kitchen list

Login to Jenkins

Login to the jenkins console, at https://jenkins-delivery.<your route53 domain> using the 'admin' user. The password can be obtained by logging into the Jenkins instance:

Once logged in, you should see that the InitialSetup pipeline has begun. This pipeline sets up the system ready to start building custom software. The Jenkins console should look something like this when InitialSetup has completed successfully:

Login to Supermarket

Login to Chef Supermarket at https://chef-supermarket.<your route53 domain> with the 'chefadmin' user and authorise Supermarket to use the chefadmin account. The default password for the chefadmin user can be set in the .kitchen.local.yml configuration, check .kitchen.yml for the default.

Building and Deploying Software

Now we can demonstrate using the system to deliver software. In this (fairly trivial) example, we are using our public engineering blog as 'software' that we will be deploying onto an EC2 instance. In order to do this, there are two more git repositories you will need to fork. 

  1. skybet_public. This repository is an example Jenkins Delivery repository. It contains a Chef repository, along with both build and deployment pipelines for the project. Fork the repository from https://bitbucket.org/harriga/skybet_public and give the Jenkins Delivery git SSH key access to pull/push to your fork.
  2. sbg_engineering: This repository is a Chef cookbook and associated Jenkins Delivery Pipeline used to build the cookbook. Fork the repository from https://bitbucket.org/harriga/engineering and give the Jenkins Delivery git SSH key access to pull/push to your fork.

This example only uses one Chef cookbook, but could just as easily be using many more in order to build complex environments composed of many different services. There is no real limit except your imagination.

Create an Organization

Next, we use the Jenkins console and the 'CreateChefOrg' job to create a container Organization for our code to be uploaded to, and our EC2 instances to be driven from. CreateChefOrg will have already been run once for the setup of the 'management' organization during the InitialSetup pipeline. Run it again for our new 'skybet_public' organization:

The pipeline will run, create the new organization and make it available in the Jenkins menus for use with other jobs.

Build a Cookbook Version

Before starting this step, make sure you have logged into Supermarket once and allowed the chefadmin account to be linked and used for Chef Supermarket (above). Jenkins uploads to Chef Supermarket using this account so it must already be authorised to do so.

Run the ChefCookbook job, and fill in the parameters appropriately. Make sure the Jenkins git SSH key you configured earlier has access to clone and push to the cookbook repository

This job will checkout the Cookbook repository using the supplied git reference, then execute pipeline/build.groovy directly from the cookbook repository. In this example, our cookbook pipeline performs the following tasks:

  • Check out the Cookbook repository using the supplied git reference
  • Run Test Kitchen (Chef recipe both builds and tests the software, as well as building and testing the configuration management)
  • Tag the repository according to the supplied $VERSION_TYPE. This is semantic versioning, so minor major or patch revisions
  • Upload the built and tagged cookbook to Chef Supermarket. The Supermarket cookbook is versioned and contains a bundled, pre-built version of the application

The completed pipeline

The built cookbook will now be available on Chef Supermarket. Login to the Chef Supermarket and verify that you can now see a version of sbg_engineering hosted:

The cookbook as it appears in Chef Supermarket

Build a Jenkins Delivery Repository

Next it is time to build and test a Jenkins Delivery repository. This repository contains the Berksfile, Chef roles, environments, data bags and the Jenkins pipelines used for build and deployment of the service.

Run the 'BuildIntegration' job. This will execute pipeline/build.groovy from the source repository. In this example, our test is a simple 'berks update'.

In this example, our integration build pipeline performs the following tasks:

  • Check out the repository using the supplied git reference
  • Run 'berks update', verifying that all required cookbook versions are available
  • Tag the repository according to the supplied $VERSION_TYPE. This is semantic versioning, so minor major or patch revisions

Once finished, the integration build will look something like this:

This has built version 0.0.2 of the integration which is ready to be deployed.

Deploy the Integration

The 'DeployIntegration' job will checkout a Jenkins Delivery repository, then execute pipeline/deploy.groovy. In the supplied example, this uploads all Chef code to the Chef server (Roles, environments, data bags) and then installs the Berksfile to the Chef server (which includes the sbg_engineering cookbook we built earlier). Execute the 'DeployIntegration' job: