Skip to main content
Table of contents

This documentation is intended for internal use by the GDS community.

Continuous Deployment

The Reliability Engineering team uses Concourse for testing and deploying code to production.

Concourse is a versatile, open-source, continuous thing-doer which can be used for task automation, running tests and status checks against pull-requests, and deploying code.

The Reliability Engineering team help you get started with Concourse, and ensure that the infrastructure is available.

Tenancy model

The Reliability Engineering team’s Concourse is multi-tenanted:

  • Concourse is configured with many teams

  • Each teams workloads run on separate infrastructure

  • Users in one team cannot interact with the pipelines of another team

    • (unless the user is a member of both teams)

By default, all Concourse workers share the same egress IP addresses, unless configured otherwise. If your team needs separate egress IP addresses, then please ask Reliability Engineering to make the configuration changes.

The Reliability Engineering team’s Concourse is managed as follows:

  • Concourse is configured to delegate secrets management to AWS Systems Manager Parameter Store

  • Concourse worker instances are recreated every morning to ensure they have the latest security patches

  • Users authenticate using GitHub SSO

    • (if you need Google SSO instead of GitHub, ask the Reliability Engineering team)
  • Concourse is accessible from the GDS network (Whitechapel and the VPN)

Getting started with Concourse

Concourse has comprehensive documentation and interactive examples.

Reliability Engineering run a short “Introduction to Concourse” workshop, if you are interested then let us know.

Teams

The Reliability Engineering team configured Concourse so that the workloads of different teams run on different infrastructure. This design decision ensures that:

  • the time taken to schedule and run workloads stays predictable
  • the workloads of each team are isolated to separate pools of resources

Teams do not have to represent actual organisational units, and can instead be used for fine-grained control over permissions.

For example: a service team could have two teams:

  • service-team-dev
  • service-team-deploy

Where:

  • service-team-dev is for pull-requests and management of development environments
  • service-team-deploy is for deploying trusted code to production

Where:

  • service-team-dev includes all service team members
  • service-team-deploy includes only team members who are trusted to deploy code

Roles

Concourse has 5 roles, which are explained in depth in the documentation.

When the Reliability Engineering team configure your Concourse team, you should decide which Concourse team role should be configured. The role you choose will be allocated to GitHub teams, so that anyone in the GitHub team is given that role.

The roles you should choose between are usually:

  • member
  • pipeline-operator

Members can manipulate pipelines and update the configuration of pipelines from the command line. This role should be used in most cases.

Pipeline-operators can only interact with pipelines, but cannot update their configuration. This role should be used where a strict 2-pairs-of-eyes code review policy is enforced.

Services

For convenience, the Reliability Engineering team provision the following resources for each Concourse team:

  • A public AWS S3 bucket (publicly readable from the internet by anyone)
  • A private AWS S3 bucket (readable only by the team’s Concourse workers)
  • A private AWS Elastic Container Registry (ECR) (readonly only by the team’s Concourse workers)

which can be accessed only by the individual Concourse team’s worker AWS IAM roles.

Each Concourse team’s worker instances have a specific AWS IAM role, this allows Concourse to assume roles in other AWS accounts, if those accounts have a role which Concourse is permitted to assume.

Secrets

The Reliability Engineering Concourse delegates secrets management to AWS Systems Manager Parameter Store and Key Management Service.

Secrets can be managed by users with the member role using the GDS CLI:

gds cd secrets add

and

gds cd secrets rm

Concourse injects secrets into pipelines at runtime using double parentheses syntax:

resources:
  - name: my-repository
    type: git
    source:
      uri: git@github.com:alphagov/my-repository.git
      branch: master

      # This is a secret
      private_key: |
        ((github-ssh-key))

For your convenience when writing Concourse pipelines, there are also some variables provided for you which are generated automatically and are immutable (hence the readonly_ prefix).

  • readonly_private_bucket_name - the name of the private AWS S3 bucket
  • readonly_private_bucket_arn - the ARN of the private AWS S3 bucket
  • readonly_private_bucket_domain_name - the domain name of the private AWS S3 bucket

  • readonly_public_bucket_name - the name of the public AWS S3 bucket

  • readonly_public_bucket_arn - the ARN of the public AWS S3 bucket

  • readonly_public_bucket_domain_name - the domain name of the public AWS S3 bucket

  • readonly_private_ecr_repo_name - the name of the private AWS ECR

  • readonly_private_ecr_repo_arn - the ARN of the private AWS ECR

  • readonly_private_ecr_repo_url - the URL of the private AWS ECR

  • readonly_private_ecr_repo_registry_id - the ID of the private AWS ECR

  • readonly_team_name - the name of the Concourse team

  • readonly_local_user_password - the password for a local user of the Concourse team, which is used for updating pipelines

  • readonly_secrets_path_prefix - the secrets path prefix, which is used for managing secrets

  • readonly_secrets_kms_key_id - the AWS KMS Key ID, which is used for encrypting secrets at rest

Monitoring

The Reliability Engineering team’s Concourse is monitored using:

  • Prometheus - collection of metrics
  • Grafana - alerting and dashboards
  • Splunk - audit logs

Concourse emits metrics which can be queried to get metrics about pipelines.

For example, the following query shows how often the internal-apps pipeline is run over a 12 hour period.

sum
  without (instance, status) (
    increase(
      concourse_builds_finished{team="autom8", pipeline="internal-apps"}[12h]
    )
  )

Example pipeline

Deploy an application to GOV.UK PaaS

This pipeline clones a public open source git repository, and deploys it to GOV.UK PaaS

Requirements

There must be the following variables set:

  • paas-username
  • paas-password

Pipeline

---
resources:
  - name: my-git-repo
    type: git
    source:
      branch: master
      uri: https://github.com/alphagov/my-repo.git
  
  - name: my-paas-app
    type: cf
    source:
      api: https://api.london.cloud.service.gov.uk
      organization: my-service-team
      space: my-space
      username: ((paas-username))
      password: ((paas-password))

jobs:
  - name: test-and-build
    public: false
    plan:
      - get: my-git-repo
        trigger: true
  
      - task: build
        config:
          platform: linux
  
          image_resource:
            type: docker-image
            source:
              repository: ruby
              tag: 2.7

          inputs:
            - name: my-git-repo
          outputs:
            - name: my-git-repo

          run:
            path: sh
            dir: my-git-repo
            args:
              - -exc
              - |
                bundle install
                bundle exec middleman build
  
      - put: my-paas-app
        params:
          manifest: my-git-repo/manifest.yml
          path: my-git-repo