Table of contents

This documentation is intended for internal use by the GDS community.

How to create Grafana dashboards

Logging into Grafana

Visit grafana-paas.cloudapps.digital and sign in with Google:

google-auth

Log in with your digital.cabinet-office.gov.uk email address.

Viewing metrics for your app

Click ‘Home’ in the top left of the screen:

dropdown

Select 'GDS Default Dashboard’ from the dropdown:

Dashboard List

Select your application from the 'Available Apps’ dropdown:

Available Apps

If you can’t see your application in the list, make sure you’ve followed the instructions in Setting up metrics and that your changes have been deployed to the GOV.UK PaaS.

You should now see the GDS Default Dashboard applied to your application.

Understanding the dashboard

This is the default dashboard for every application using the Reliability Enginering team metrics solution:

Dashboard

What’s powering this dashboard?

This dashboard has been built using the following metrics provided by Prometheus:

http_server_exceptions_total
http_server_request_duration_seconds_bucket
http_server_request_duration_seconds_count
http_server_request_duration_seconds_sum
http_server_requests_total

The widgets on the dashboard are the result of queries over these metrics. For example, the 'Number of requests per minute’ widget is built from http_server_requests_total using Prometheus’ sum aggregator and rate function. TODO - link to PromQL section

What does the dashboard tell me?

There are eleven widgets showing information about your application:

Template dashboard widgets

  1. A brief description of the dashboard.
  2. The number of healthy instances the application has running right now. The range of colors is: Green (more than 1 instance available), Orange (1 instance available), Red (no instances available)
  3. The number of total instances for your application.
  4. The percentage of HTTP Status code 2xx responses.
  5. The percentage of HTTP Status code 3xx responses.
  6. The percentage of HTTP Status code 4xx responses.
  7. The percentage of HTTP Status code 5xx responses.
  8. The number of request per minute your application is serving.
  9. The number of HTTP errors (4xx, 5xx) per minutes your application is throwing.
  10. A graph showing the average response times for your application.
  11. A table of the list of the seven slowest endpoints in the application.

Requesting Grafana edit permissions from Reliability Engineering

Contact us on Slack in the #reliability-eng channel to be granted edit permissions.

Refer to the instructions below to help you orientate the Grafana interface:

Dashboard

  1. Click these icons to configure Grafana, e.g. manage profile updates, alerts, permissions, dashboard and folders. You’ll see different options depending on your level of permissions.
  2. Click this dropdown to see the list of available dashboards, some of which you can apply to change the view of your application.
  3. Click this dropdown to see the list of applications currently instrumented with GDS metrics. In the screenshot, we have selected registers.cloudapps.digital:80.
  4. Click these icons to interact with the dashboard. You can create a copy of the dashboard or share it, as well as changing the time period displayed.
  5. View the widgets and graphs currently measuring the performance of your application. Click and drag on any of the graphs to zoom in on a particular time period. If you have editing permissions, you can click on any widget to configure or remove these.

How to view dashboards on your team TV

Use your TV’s Google Chromebit user to display your Grafana dashboard because the Google Chromebit user will only be able to access Grafana with read permissions.

You should not use your personal Google account to display your Grafana dashboard on your team’s TV. This is because your personal Google account may have editing or admin permissions.

More instructions for navigating Grafana

Selecting different dashboards, and creating new dashboards

The dashboard dropdown contains all dashboards currently in use. Use the filters to find dashboards by name or by tag. If you have edit permissions, you can also create new dashboards.

Dashboard selector