Continuous Deployment

The Reliability Engineering team uses Concourse for testing and deploying code to production.

Concourse is a versatile, open-source, continuous thing-doer which can be used for task automation, running tests and status checks against pull-requests, and deploying code.

The Reliability Engineering team help you get started with Concourse, and ensure that the infrastructure is available.

Tenancy model

The Reliability Engineering team’s Concourse is multi-tenanted:

Concourse is configured with many teams
Each teams workloads run on separate infrastructure
Users in one team cannot interact with the pipelines of another team
- (unless the user is a member of both teams)

By default, all Concourse workers share the same egress IP addresses, unless configured otherwise. If your team needs separate egress IP addresses, then please ask Reliability Engineering to make the configuration changes.

The Reliability Engineering team’s Concourse is managed as follows:

Concourse is configured to delegate secrets management to AWS Systems Manager Parameter Store
Concourse worker instances are recreated every morning to ensure they have the latest security patches
Users authenticate using GitHub SSO
- (if you need Google SSO instead of GitHub, ask the Reliability Engineering team)
Concourse is accessible from the GDS network (Whitechapel and the VPN)

Getting started with Concourse

Concourse has comprehensive documentation and interactive examples.

Reliability Engineering run a short “Introduction to Concourse” workshop, if you are interested then let us know.

Teams

The Reliability Engineering team configured Concourse so that the workloads of different teams run on different infrastructure. This design decision ensures that:

the time taken to schedule and run workloads stays predictable
the workloads of each team are isolated to separate pools of resources

Teams do not have to represent actual organisational units, and can instead be used for fine-grained control over permissions.

For example: a service team could have two teams:

service-team-dev
service-team-deploy

Where:

service-team-dev is for pull-requests and management of development environments
service-team-deploy is for deploying trusted code to production

Where:

service-team-dev includes all service team members
service-team-deploy includes only team members who are trusted to deploy code

Roles

Concourse has 5 roles, which are explained in depth in the documentation.

When the Reliability Engineering team configure your Concourse team, you should decide which Concourse team role should be configured. The role you choose will be allocated to GitHub teams, so that anyone in the GitHub team is given that role.

The roles you should choose between are usually:

member
pipeline-operator

Members can manipulate pipelines and update the configuration of pipelines from the command line. This role should be used in most cases.

Pipeline-operators can only interact with pipelines, but cannot update their configuration. This role should be used where a strict 2-pairs-of-eyes code review policy is enforced.

Services

For convenience, the Reliability Engineering team provision the following resources for each Concourse team:

A public AWS S3 bucket (publicly readable from the internet by anyone)
A private AWS S3 bucket (readable only by the team’s Concourse workers)
A private AWS Elastic Container Registry (ECR) (readonly only by the team’s Concourse workers)

which can be accessed only by the individual Concourse team’s worker AWS IAM roles.

Each Concourse team’s worker instances have a specific AWS IAM role, this allows Concourse to assume roles in other AWS accounts, if those accounts have a role which Concourse is permitted to assume.

Secrets

The Reliability Engineering Concourse delegates secrets management to AWS Systems Manager Parameter Store and Key Management Service.

Secrets can be managed by users with the member role using the GDS CLI:

gds cd secrets add

and

gds cd secrets rm

Concourse injects secrets into pipelines at runtime using double parentheses syntax:

resources:
  - name: my-repository
    type: git
    source:
      uri: git@github.com:alphagov/my-repository.git
      branch: master

      # This is a secret
      private_key: |
        ((github-ssh-key))

For your convenience when writing Concourse pipelines, there are also some variables provided for you which are generated automatically and are immutable (hence the readonly_ prefix).

readonly_private_bucket_name - the name of the private AWS S3 bucket
readonly_private_bucket_arn - the ARN of the private AWS S3 bucket
readonly_private_bucket_domain_name - the domain name of the private AWS S3 bucket
readonly_public_bucket_name - the name of the public AWS S3 bucket
readonly_public_bucket_arn - the ARN of the public AWS S3 bucket
readonly_public_bucket_domain_name - the domain name of the public AWS S3 bucket
readonly_private_ecr_repo_name - the name of the private AWS ECR
readonly_private_ecr_repo_arn - the ARN of the private AWS ECR
readonly_private_ecr_repo_url - the URL of the private AWS ECR
readonly_private_ecr_repo_registry_id - the ID of the private AWS ECR
readonly_team_name - the name of the Concourse team
readonly_local_user_password - the password for a local user of the Concourse team, which is used for updating pipelines
readonly_secrets_path_prefix - the secrets path prefix, which is used for managing secrets
readonly_secrets_kms_key_id - the AWS KMS Key ID, which is used for encrypting secrets at rest

Monitoring

The Reliability Engineering team’s Concourse is monitored using:

Prometheus - collection of metrics
Grafana - alerting and dashboards
Splunk - audit logs

Concourse emits metrics which can be queried to get metrics about pipelines.

For example, the following query shows how often the internal-apps pipeline is run over a 12 hour period.

sum
  without (instance, status) (
    increase(
      concourse_builds_finished{team="autom8", pipeline="internal-apps"}[12h]
    )
  )

Example pipeline

Deploy an application to GOV.UK PaaS

This pipeline clones a public open source git repository, and deploys it to GOV.UK PaaS

Requirements

There must be the following variables set:

paas-username
paas-password

Pipeline

---
resources:
  - name: my-git-repo
    type: git
    source:
      branch: master
      uri: https://github.com/alphagov/my-repo.git
  
  - name: my-paas-app
    type: cf
    source:
      api: https://api.london.cloud.service.gov.uk
      organization: my-service-team
      space: my-space
      username: ((paas-username))
      password: ((paas-password))

jobs:
  - name: test-and-build
    public: false
    plan:
      - get: my-git-repo
        trigger: true
  
      - task: build
        config:
          platform: linux
  
          image_resource:
            type: docker-image
            source:
              repository: ruby
              tag: 2.7

          inputs:
            - name: my-git-repo
          outputs:
            - name: my-git-repo

          run:
            path: sh
            dir: my-git-repo
            args:
              - -exc
              - |
                bundle install
                bundle exec middleman build
  
      - put: my-paas-app
        params:
          manifest: my-git-repo/manifest.yml
          path: my-git-repo

Results