W3tTr3y's blog

A personal technology focused blog

Splunk SSL Chain

On Monday March 31 my boss forwarded an e-mail stating the SSL certificates for our Splunk servers were expiring; my employer participates in InCommon’s SSL certificate program, so I thought no big deal generate a CSR (certificate signing request), fill out the form and upload the csr, wait for the e-mail, download it, restart splunk. Sounds like a lot of work but no sweet.

Generating CSR

We currently have six search heads so filling out all the information and ensuring I didn’t make a typo seemed like a daunting task (I’m a notriously bad typer). Luckily, a quick Google search turned up openssl’s x509toreq option which will generate a CSR based on an existing certificate.

Since its bad practice to access the web from a server and I needed to upload the CSR to the CA, I wanted to: 1. Generate a CSR 2. Copy it to my local workstation * the CSR doesn’t contain the private key, so confidentiality isn’t a concern; the primary security concern is in protecting the integrity of the request.

We run Splunk as non-root and the permissions on the private key and curent certificate pervent anyone other then that user (except root of course) from reading those files, so I needed to run sudo; if you’ve tried sudo generally prevents you from running multiple commands (there’s a good stackoverflow thread on it)

In the end I used (line breaks added for readability):

for server in splunk1 splunk2 splunk3 splunk4 splunk5; 
    ssh -t -t ${server} "sudo -u splunkuser -- sh -c '
      cd /opt/splunk/etc/auth/splunkweb; 
      [ -f ${server}.csr  ] && rm ${server}.csr ; 
      [ -f ${server}.pem -a ! -f ${server}.csr -a -f ${server}.key ] 
        && openssl x509 -x509toreq -in ${server}.pem -out ${server}.csr -signkey ${server}.key; 
      cat ${server}.csr'" 
      | tail 18 > ${server}.csr ; done

Note: For each server it will run sudo and you must type your password for that server. You will not see a password prompt as the output is swallowed by the tail command. If nothing happens, you may have missed typed your password and need to retype it. The first time is the first server, then wait to see a message about the connection being closed and you should be good to type the password for the sudo command on the next server.

ssh -t -t
-t forces pseudo-tty allocation; the second -t forces the allocation even if there is no local tty.
This was required in order to get sudo to prompt for my password; otherwise it sees no tty and won’t prompt
sudo -u splunkuser
runs the commands as splunkuser – you should adjust it to be whatever user you run splunk as
– sh -c ‘[…]’
this is the “hack” to run multiple commands in one invocation
– makes it so sudo will quit looking for arguments
sh runs the sh shell
-c tells ssh we’d like to run one “command” and exit
’[…]’ then needs to contain one “command group” – something that you could type on one line of an interactive shell and type enter. Thus you need to join commands with things like `&&` or `;`
[ -f ${server}.csr ] && rm ${server}.csr
Check to see if a csr exists and if so delete it; I added this as my first run had an error and I needed to re-run it for a couple of servers
[ -f ${server}.pem -a ! -f ${server}.csr -a -f ${server}.key ]
ensure that the server’s certificate exists (${server}.pem so splunk0.pem, splunk1.pem as appropriate), that no csr exists, and that the private key file exists
openssl x509 -x509toreq -in ${server}.pem -out ${server}.csr -signkey ${server}.key;
Generate a new certificate signing request (CSR) based on the existing certificate utilizing the key ${server}.key (e.g. splunk0.key, splunk1.key etc. as appropriate)
cat ${server}.csr
This takes the newly generated certificate signing request and sends it to stdout (which we’ll use next)
tail 18 > ${server}.csr
This reads stdin (which is the data from cat’s stdout) and sends the last 18 lines to the output
I’m a natoriously bad typer, so I’m probably going to typo my password. Since the certificate signing request is 18 lines, this throws away those password prompts and only sends the CSR on
`> ${server}.csr` taks the 18 lines from the tail and puts them in an appropriately named file (e.g. splunk0.csr splunk1.csr etc)

Installing the new certificates

So I uploaded and waited; today I finally got notification that the certificates were ready.

So I click the link for the “X509, Base64 encoded” version. I copied the certificate over, named it appropriately, restarted splunk. Despite Splunk’s claim that it started correctly, Splunkweb wasn’t responding to requests. Great!

Chacking the Splunk wiki, there’s an article on 3rd Party CA’s and under the Complex certificate chains section it states:

If you are using a certificate chain, you need to bundle the intermediate and the server certificate into a single certificate, by concatenating the certificates together (the right type, and in the right order)[…]

   ... (certificate for your server)...
   -----END CERTIFICATE-----
   ... (the intermediate certificate)...
   -----END CERTIFICATE-----
   ... (the root certificate for the CA)...
   -----END CERTIFICATE-----

In checking our certificate bundle, it was in the opposite order.

Correcting the Certificate Chain Order

Luckily while searching how to get openssl to display the common name for more then one certificate I came across a stackexchange post which linked to a perl script which lists the common name and then will save them in seperate files (http://gagravarr.org/code/cert-split.pl). I started to modify it, but for various reasons switched over to python and wrote a script inspired by it.


Update: On April 23, 2014 I removed the direct listing of the source of the script and moved it into its own github repository. Hopefully this makes it easy to make contributions, log issues (in case you don’t know how to code or don’t have time), and to track updates/changes.

Using Splunk’s Bundled Python

This script utilizes PyOpenSSL; the python bundled with Splunk includes this library, so it should be easy fairly easy to run. By default the script utilizes the standard python shebang line #!/usr/bin/env python which should invoke the systems version of python. It also includes an example on the third line of directly utilizing the python bundled with Splunk (assuming $SPLUNKHOME is /opt/splunk) #!/opt/splunk/bin/python If you remove the first two lines and modify /opt/splunk to point to your $SPLUNKHOME if required, it should work.


To reverse the certificate order in the pem file:

./cert-reorder.py -r certificatechain.pem

While that was my first attempt and worked fine for us — our CA always provides them in server, intermediate, CA order — this seemed potentially error prone. What if its ran multiple times, the order changes, etc. Since Splunk bundles PyOpenSSL it was fairly easy to implement a version that finds the root certificate and then moves along the chain to the server certificate.

To analyze the certificates and put them in CA, intermediate, server order regardless of input order

./cert-reorder.py -a certificatechain.pem

If you want to verify the order of the certificates after a command, you can add the -p option

./cert-reorder.py -pa certificatechain.pem