Private key errors from Logstash when shipping logs over SSL using filebeat

We use the filebeat shipper to ship logs from our various servers, over to a centralised ELK server to allow people access to production logs.

Because the logs are shipped from various data centres, the filebeat shippers are configured to send logs using SSL. To achieve that, we followed the how to guide. For the curious, that means that each of our servers has a filebeat.yml configuration file containing the following excerpt,

output.logstash:
    hosts: ["hostname:5044"]
    ssl.certificate: "/etc/ssl/certs/our.crt"
    ssl.key: "/etc/ssl/private/our.key"

and the inputs config file on our ELK server contains the following,

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/ssl/certs/our.crt"
    ssl_key => "/etc/ssl/private/our.key"
  }
}

I upgraded Logstash this morning, from 6.2 to 6.4 and found that Logstash wouldn’t start. When I inspected the Logstash logs I found errors as follows:

[2018-08-30T10:58:50,842][ERROR][logstash.inputs.beats ] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

I checked the filebeat shipper logs too, and found errors similar to the following:

2018-08-30T11:26:02Z ERR Connecting error publishing events (retrying): read tcp ip.address.here:45896->ip.address.here:5044: read: connection reset by peer

In actual fact, I needn’t have gone further than the error that I found in the Logstash error log which said, “your private key was not in PKCS8 format”.

The SSL key that we had shipped to the box was not in PKCS8 format, which according to the documentation is a requirement. The fix in the end was simple, and was provided in this comment against a github issue. I needed to create a PKCS8 version of our existing key. To do that, I ran the following command,

openssl pkcs8 -in /etc/ssl/private/our.key -topk8 -nocrypt -out /etc/ssl/private/our.p8

To use the new PKCS8 version of the key, I also needed to update the beats input file to use the new our.p8 key file and then restart Logstash. The revised config is as follows:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/ssl/certs/our.crt"
    ssl_key => "/etc/ssl/private/our.p8"
  }
}

Hope this helps someone else!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s