W3tTr3y's blog

A personal technology focused blog

Cron Jobs Not Executing

Last week we started getting some interesting error messages on the servers I manage:

Dec  1 07:32:28 hostexample kernel: type=1101 audit(1448977801.109:10043532): pid=28074 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=failed'

Background

My first reaction was to assume that it was due to some of the system hardening stuff that I have been implementing. I was a bit confused as the last change was nine days before we started getting this error messaging. Since I’m implementing the changes via Puppet, I was concerned not just with the most recent changes, but was something implemented earlier working (or breaking) inconsistently. One source of comfort was the last puppet update had been 9 days before the log messages started.

Root’s Password Expired

While the leeds I’d found online weren’t incorrect that pam was causing cron to fail, the initial issue was that root’s password had expired.

# chage -l root
Last password change                                   : Aug 26, 2015
Password expires                                      : Nov 24, 2015
Password inactive                                     : never
Account expires                                       : never
Minimum number of days between password change     :  7
Maximum number of days between password change     : 90
Number of days of warning before password expires  :  7

We only had distribution stock cron jobs running and they were all configured to run as root. If you’re lucky you’ll have a more diverse environment that will help you narrow it down to being specific to one account.

If an expired password isn’t your issues, I found some other interesting links suggesting adding [+ : root : cron crond to /etc/security/access.conf] (http://linuxfollies.blogspot.com/2014/06/root-cron-jobs-and-etcsecurityaccessconf.html), [adusting /etc/pam.d/system-ac] (TODO: find link), or editing /etc/cron.allow (although /etc/cron.allow won’t block root)

Immediate Action

After confirming that this was the issue, I set root’s password way into the future. This will prevent the account from expiring which breaks things.

I also configured our monitoring system to check for root’s password expiring once a day. This will provide a safety net in case things get misconfigured again. TODO: add a link to a new article about Nagios monitoring

Planned Actions

Our curent security requirements dictate that all passwords expire; at a high level that’s not a bad policy. The problem is the actual auditing looks at two things: PASS_MAX_DAYS in /etc/login.defs and at the value in /etc/shadow (TODO: confirm /etc/shadow vs /etc/passwd)

Meeting Technical Requirements

To meet the technical requirements, I want to continue to allow our configuration management system to configure root’s password to expire. This way we will pass

Meeting High Level Requirements

To meet the spirit of the agreement, I want to lock the root password.

While that’s not a bad policy in general, there are a number of conisderations here: 1. There are a small (minuscle) number of people with access to this system 2. All users have sudo access 3. We already block remote access via root

If the user is local but has forgotten their password, they can reboot the server using media they provide and reset the root password.

Thus the only situation where we are now leaving the person high and dry is if they have ssh keys to remotely access the box, but can’t remember their own password.

My guess is we’ll deem that an acceptable risk. If not, we can always create a backdoor account where the necessary password / keys are stored in a break-the-glass style safe.

#