Just a quote post to note that we heard back from Splunk’s support people and they pointed out SPL-69922 which affects Splunk versions prior to 5.0.5
While I don’t have a lot of details about SPL-69922, the 5.0.5 release notes:
UF forwarding data in load balancing mode is sending twice more to one specific indexer in the list. (SPL-69922)
Since I’ve written about index affinity a fair amount (2/4/2014 Splunk Indexer Affinity, 6/5/2014 Forwarder’s Indexer Selection Not So Random, 6/9/2014 Indexer Affinity Update: Forwarder OS Specific? ), there have been many discussions about is it really an issue, and despite the defect’s fix being released in September 2013, our opening a ticket in December 2013, and finally in June 2014 they point it out to us, I wanted to post our solution in hopes that it might help some one else.