W3tTr3y's blog

A personal technology focused blog

Forwarder's Indexer Selection Not So Random

Back in Feburary I posted about Indexer Affinity in Splunk; while we’ve been a bit distracted, we did circle back around earlier this week and I learned some surprising information.

Splunk Architecture Background

Splunk can ingest any test-based (or convertible to text-based) machine data so its used for a number of different tasks by a number of different people. A popular, if not the most popular model, involves ingesting data from a number of systems via forwarders. The forwarders send the data to indexers, a server or groups of servers which store the data and allow you to perform searches on that data. While it’s perfectly valid to only have one indexer, for larger volumes of data you scale by standing up additional indexers (a general rule of thumb I’ve been told is 100 GB/day per indexer).

When you have multiple indexers, incoming data needs to be partitioned between the indexers (or group of indexers); typically this is done by sending to an indexer for 30 seconds and then pseudo-randomly selecting a new indexer. There’s a bit more to it as there is a quarenteening process if an indexer is unreachable etc, but that’s the basic idea.

The Big Secret: Splunk MIGHT Always Send to the First Indexer Listed

While they are working on confirming this, it is the understanding of our Designated Support Engineer (DSE) and Customer Success Manager (CSM) that the forwarders will always initially connect to the first forwarder in the list. After that first connection, they will then (psuedo)randomly select an indexer to send data to.

Checking the Logs

The great things about Splunk is its a platform to analyze data, so let’s use it to analyze Splunk’s logs and confirm/deny this claim.