W3tTr3y's blog

A personal technology focused blog

Heartbleed: Splunkweb Certificate Regeneration

Splunk has released Splunk Enterprise 6.0.3 which addresses the heartbleed vulnerability; all users of Splunk Enterprise 6.x should upgrade.

In my opinion you should upgrade your core infrastructure ASAP and then regenerate your private keys, generate new certificates, and once you deploy the new certificates revoke the previous ones.

While this isn’t a small task, its not particularly hard and most CA’s will re-issue certificates for free so the only cost is a soft cost of your time. While heartbleed attacks are easy to detect now that we know what to look for, exploitation doesn’t cause any unusual log entries so its virtually impossible to examine logs and tell if you have been exploited.

Generating a New Key

It is quite easy to generate a new RSA key:

openssl genrsa -out mykey.pem 2048

In the case of Splunk Web, you want to generate the key in $SPLUNKHOME/etc/auth/splunkweb and Splunk bundles a version of OpenSSL

$SPLUNKHOME/bin/openssl genrsa -out 2014Splunk1.key 2048

Building on Previous Work

Luckily our certificates for Splunk web expired last week, so I’m familiar with what I need to do to re-issue the certificates. The only difference is I need to generate new keys and then utilize the new keys when generating the CSR. In last week’s post, Splunk SSL Chain, I listed a command to easily generate and collect CSR’s for your search headers:

for server in splunk1 splunk2 splunk3 splunk4 splunk5; 
  do 
    ssh -t -t ${server} "sudo -u splunkuser -- sh -c '
      cd /opt/splunk/etc/auth/splunkweb; 
      [ -f ${server}.csr  ] && rm ${server}.csr ; 
      [ -f ${server}.pem -a ! -f ${server}.csr -a -f ${server}.key ] 
        && openssl x509 -x509toreq -in ${server}.pem -out ${server}.csr -signkey ${server}.key; 
      cat ${server}.csr'" 
      | tail 18 > ${server}.csr ; done

where splunk1 splunk2 (etc) is a space seperated list of your search heads.

Modify to generate a new key

Before generating a new certificate, add the line in that genereates the new key:

for server in splunk1 splunk2 splunk3 splunk4 splunk5;
  do
    ssh -t -t ${server} "sudo -u splunkuser -- sh -c '
      cd /opt/splunk/etc/auth/splunkweb; 
      [ -f ${server}.csr  ] && rm ${server}.csr ; 
      [ -f ${server}.pem -a ! -f ${server}.csr]
        && openssl genrsa -out ${server}2014.key 2048 
        && openssl x509 -x509toreq -in ${server}.pem -out ${server}.csr -signkey ${server}2014.key;
      cat ${server}.csr'" 
      | tail 18 > ${server}.csr ; done

I specifically do not delete the old key as it is currently in use with the certificate and I would like to continue utilizing Splunk’s web interface while I wait for our CA to sign the new requests.

To avoid overwriting the old key, I give the new key a different name; in this case I chose to append 2014 assuming I’ll remember that 2014 was when heartbleed occured.

Next Steps

Once the CA signs the certificates, I’ll put them in place (if you have errors, see my previous Splunk SSL Chain post about the order of certificate chains). I’ll then delete the old certificates and the old keys.