This post is primarily a placeholder so I have some of the commands handy in case we ever need them again.
Approximately half of the logs that are ingested into Splunk come into our envrionment via syslog. It shouldn’t be surprising that the health checks I’ve been working on were starting to indicate that I/O was becoming a bottleneck on our syslog server, so we re-purposed a fairly new server to be a new log server to share the load.
As part of this move, we’re utilizing CentOS as our server team has standardized on RedHat and while we still run our own servers, we wanted to try and at least be a friendly environment if they were to manage the servers down the road. With that change came the introduction of SELinux and thus a couple of quick notes.
A student worker (college students that we can employee fairly cheeply) did the install, my boss did most of the initial configuration. I started getting invovled when a disk filled up and we found out that the raid array for the OS install had been created but the larger one for data (logs in this case) hadn’t.
We could throw logs under /var/log and then everything should have worked automagically, but since this is a dedicated syslog server that’s a lot of logs so we mount the drive in a different location which we’ll call /mnt/logs for the purposes of this article (its arbitrary and likely to change so a generic place holder feels best).
File System Prep
Ext3 to Ext4
The original file system for the /mnt/logs partition was formatted as ext3; while there is nothing wrong with that, most of our other partitions are using ext4 so I went ahead and converted it for consistency’s sake.
sudo tune2fs -O extents,uninit_bg,dir_index /dev/sdb sudo e2fsck -fDC0 /dev/sdb
Reserve only 5 inodes for superuser (vs 5%)
By default 5% of the filesystem is reserved for root; this is a really nice feature as if the drive “fills up” it ensures the administrator has some wiggle room to use while freeing up space. On a partition dedicated to logs, we don’t expect any files to be used by root (so this amount will likely be completely available) and as this is a large partition 5% is quite significant, so we lower this to 5 inodes.
sudo tune2fs -r 5 /dev/sdb
Find the UUID for the drive to create the /etc/fstab entry
I really like the idea of using UUID; since we’re utilizing drives in bays physically in the server’s chasis, I wouldn’t expect the order to change, but UUIDs are easy enough .
sudo blkid /dev/sdb
Change the Label:
sudo chcon -R -t var_log_t /mnt/logs
Change the Stored Label (so relabeling doesn’t break syslog)::
semanage fcontext -a -t var_log_t /mnt/logs
Run a Test Restore to Validate Above Change:
restorecon -R -v /mnt/logs
rsync -av '$(find /mnt/logs -type f -ctime -1)' /mnt/new_logs