Table of contents

This documentation is intended for internal use by the GDS community.

Sending logs securely to

Logit offers logstash endpoints. However in their default mode there is no authentication on them, this means anyone who has the logstash address could inject fake logs into your stash.

This guide takes you through setting up authentication for logstash on Logit using mutual TLS authentication.

Provision a new stack

After signing into create a new stack by selecting Add stack from the Stacks section of the dashboard.

 Stack name

When choosing a stack name, consider that other teams will need to use this name to differentiate your stack from one of their own.

Your own team will need to differentiate between stacks they own, so your stack name should be specific, concise, and convey information about who owns it and which environment it serves.

For example:

  • Good: GOV.UK Verify Hub Production (Green)
  • Bad: New dev environment 2

Stack version

In the Stack version field select the latest Elasticsearch, Logstash and Kibana (ELK) version from the displayed drop-down menu, unless you have specific reasons to prefer an earlier version.

Complete the Daily logs and Retention fields as appropriate then choose continue, confirm your stack details are correct and then Create Stack.

Once created, return to the Dashboard.

Configuring a new stack

Id and Stack API Key

An Id and Stack API Key will have been generated for your new stack. To view this information, choose Settings from the Dashboard.

The Id and Stack API Key will both be UUIDs. We’ll use the UUID below in our examples for the remainder of this manual.


Logstash endpoint

The Logstash endpoint is formed from the stack ID. Our example stack’s Logstash endpoint is

Logstash speaks many different protocols and these are all assigned to sequential TCP ports which are visible from Logstash Inputs under the stack settings page.

TCP ports are assigned on a per-stack basis at stack creation time and will be different per stack.

Shipping logs via Amazon S3

In some cases such as logging for Amazon services (CloudTrail, VPC FlowLogs etc) it can make sense to use an S3 bucket as a log “source” for your Logit stack. This differs from other ways of using Logit as logs are “pulled” into your stack by a process in Logit infrastructure.

Sharing access to resources in your Amazon account

Amazon recommend using IAM Roles to share access to resources in your account. This method doesn’t require the creation of shared secrets.

Currently Logit don’t support this, however they do plan to support it in the future. Right now the only way to grant them access is by issuing them access keys which can access the S3 bucket in question.

When setting this up you make sure that

  • The access key is only able to ListObjects and GetObject from the S3 bucket containing the logs
  • You are able to migrate to Role based access in the future when it is supported by Logit

Configuring Logit

Currently pulling logs from an S3 bucket requires a Logit support engineer to configure it for your stack. To start this process, add a “Source” to your Logit stack and select “S3”. Complete the form and a Logit support engineer will contact you once it has been configured.

Logs pulled from S3 will be processed as usual by Logstash.

Shut down unsecure inputs

Logit stacks have a non-restrictive security configuration by default. Each of the inputs listed will be behind a TCP port that is open to the internet at large, and will not authenticate connecting clients.

The only inputs we are interested in are Beats-SSL and Syslog-SSL, you will need to log a support request with Logit to close the TCP or UDP ports for all other protocols.

If you know in advance which IP addresses or address ranges you will be sourcing logging traffic from, include these in your request so that the Beats-SSL and Syslog-SSL ports can be whitelisted.

Request TLS mutual authentication

Request TLS mutual authentication is switched on for your stack. You can do this through a support request to Logit.

Logit provides two methods to achieve this:

  • provide Logit with a CA certificate of your own and instruct them to trust client certificates signed by that CA for access to the stack

  • the second, simpler method involves Logit generating a client private key and certificate on your behalf, signing it with their CA and sending it back

This manual will cover just the second, simpler method.

Please note, once TLS authentication is configured the ports you have assigned for rsyslog / beats will change after logit enable mTLS.

Logit provided keys

Once your request to enable mutual TLS authentication has been carried out Logit will send you the following files.

  1. f046ca39-8388-4c49-a536-19f95a555905.cert.pem This is the client certificate, signed by the stack-specific intermediate CA.

  2. f046ca39-8388-4c49-a536-19f95a555905.key.encrypted.pem the corresponding RSA private key for the above certificate. This private key needs to be provided to any process that wants to authenticate to the Logit service to initiate a log stream, for example, Filebeat or Rsyslog. The private key should be encrypted and Logit will provide the password to decrypt via a separate channel.

Logit CA Certificate

When mutual authentication is enabled, Logit will behave as a Root Certificate Authority. Here is the Ltd Root CA certificate your service will be trusting:


This was retrieved by running: openssl s_client -showcerts -connect

Configuring Filebeat

Filebeat is a Logstash client that reads log files from the filesystem and ships them to Logstash as they are being written to. Please refer to Filebeat documentationfor details on installation, general configuration and operation.

The following is a sample configuration fragment for mutual auth to Logit. The port number in the hosts specification must be the TCP port assigned to the Beats-SSL input in your stack.

  enabled: true
    enabled: true
      - /etc/logit/certs/logstash-ca.crt
    certificate: "/etc/logit/certs/logstash-client.crt"
    key: "/etc/logit/private/logstash-client.key"

Puppet example with optional mutual auth - using the pcfens-filebeat module:

$logstash_output_config = {
  'logstash'                    => {
    'enabled'                   => 'true',
    'hosts'                     => ["${logstash_host}:${logstash_port}"],
    'ssl'                       => {
      'enabled'                 => true,
      'certificate_authorities' => [ $logstash_ca_cert_path ],

if $mutual_auth {
  $logstash_mutual_auth_config = {
    'logstash'        => {
      'ssl'           => {
        'key'         => $logstash_client_key_path,
        'certificate' => $logstash_client_cert_path,
else {
  $logstash_mutual_auth_config = {}

$filebeat_outputs = deep_merge($logstash_output_config, $logstash_mutual_auth_config)

class { '::filebeat':
  manage_repo   => false,
  major_version => 5,
  outputs       => $filebeat_outputs,

Configuring Rsyslog

Rsyslog version 8.x is known to work, we have not tried earlier versions. Although Ubuntu Trusty ships with 7.x, packages of 8.x for Trusty are available on the Rsyslog vendor’s PPA. You must also install the optional rsyslog-gnutls package.

Then, load the gtls “stream driver” and configure it for operation with Logit:

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/logit/certs/logstash-ca.crt
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverMode 1
$DefaultNetstreamDriverCertFile /etc/logit/certs/logstash-client.crt
$DefaultNetstreamDriverKeyFile /etc/logit/private/logstash-client.key

Declare an action on matching messages that uses the omfwd module to forward them to Logit:

*.*;local2.none action (

Note the streamdriver="gtls" and streamdrivermode="1" are required here, despite the earlier config fragment implying that the default behaviour has been changed.

Proxy patterns

Architectural constraints around egress security often mean that having each of your servers connect directly to Logit’s endpoint is not an option.

This section covers some possible approaches to securely relaying Beats and Syslog traffic to Logit at your environment’s boundary.

Proxying Beats-SSL (Lumberjack protocol)

The Lumberjack protocol is not HTTP based, and as such, you can’t use an HTTP proxy. Filebeat does, however, support SOCKS negotiation.

In the following example, is the private network CIDR range of the log-generating servers that you’re proxying for, and 12345 is the TCP port of the Beats-SSL input on the Logstash endpoint.

Sample, minimal configuration for a Dante socks proxy:

logoutput: syslog

internal: eth0 port = 1080
external: eth0

method: none
clientmethod: none

user.privileged: proxy
user.notprivileged: nobody
user.libwrap: nobody

client pass {
  method: none
  log: connect disconnect error

client block {
  log: connect error

pass {
  from: to: port = 12345
  protocol: tcp
  command: connect
  log: connect disconnect error

Proxying Syslog

This section may need updating as the examples have not yet been tested

The Syslog protocol is designed to be relayed; Rsyslog fully supports this.

The configuration example in the above section will already do the egress-proxy-to-Logit part of the relay unmodified, and this should be applied to your egress proxy’s rsyslogd config.

However, you may need some additional configuration to allow your egress proxy’s rsyslogd to accept syslog messages from your internal hosts. For example:

$ModLoad imudp
$UDPServerRun 514