Unfortunately this ticket has been a real bear; luckily, my supervisor at least agrees its an issue and discovered a very interesting fact: the uneven distribution appears to be operating system specific.
Indexer Affinity OS Specific
If you look at the bottom oranged color portion representing data from forwarders running Windows, it looks fairly even across our 20 indexers. I’ll skip the purple (Linux) for a moment; the yellow for FreeBSD is semi-hard to guage due to being so uneven and it may have some affinity to it, but its a small enough volume we can safely ignore it for now. Darwin (Mac) and HPUX are basically too small to even see so we’ll ignore them also.
That leaves Linux (purple) which clearly is experiencing indexer affinity; the size of it is easily twice as much as the other indexers.
In Forwarder’s Indexer Selection Not So Random, I mention that we were given the explanation that Splunk forwarders may always send to the first indexer when they start and then the randomization kicks in; while that may still be the case, I would expect the first indexer to have a slightly higher load — nowhere near the volume of the skew we see. Also, while there might be a slight difference between the operating systems (Windows has the ___ bug, so we tend to be more liberal in our pushing of new content and restarting on the Unix side), the distribution for Windows looks really good so I think this is a good indication that something else is causing our indexer affinity issues.