Without much fanfare or notification, Splunk has changed how permissions are implemented when it comes to saved searches. Since most implementations do not limit who can create saved searches, it is quite likely they have saved searches. When upgrading to Splunk 6, there is no warning about saved searches altering how permissions to the data are implemented.
Overview of Expected Splunk Permission Behavior
Typically you create user in Splunk and then assign roles to that user. These roles include capabilities (the ability to do real-time searches or to schedule searches) and a list of data the user can access.
Unlike most systems, Splunk doesn’t have the ability to edit data (at lest not through the UI) and typically no user – not even an administrator – has delete access; instead data rolls off based on the configured retention time. Thus, giving access to the data typically means the user can search, read, and perform analytics on that data.
Pre-Splunk 6 Saved Search Permission Examples
When creating saved searches, you are really just saving the text that gets ran; the user can do post processing for different visualizations and analysis but the users permission were applied. Thus they couldn’t gain access to any data they couldn’t have just written an ad-hoc query to see. While there are permissions on the saved searches, they controlled who could utilize the search; it was more of a feature to reduce the “noise” of irrelevant searches (for that user) and improve usability and NOT a security necessity. Users sharing saved searches were not a concern as any user could only access data they already could access.
How Saved Search Permissions Work in Splunk 6
We had assumed that permissions worked the same in Splunk 6, but today I noticed that while the dashboards for our training data set were still loading, I couldn’t access the raw data.
Since this broke my understanding of permissions, I asked a co-worker to confirm that I was going crazy. While he was stunned at first, he mentioned that he had removed access to the training data last week; now that I had a lead on the change, I started tracking down why the dashboards still worked. In Splunk 6, a stored search executes with the permissions of the search’s owner. so while I couldn’t execute an ad-hoc search or utilize a dashboard with an inline search, I could access the data via a saved search created by a user who still had access.
I have received confirmation from Splunk that this is intended behavior; they compared it to the suid bit which I thought was an excellent way of concisely describing the behavior.
- Create a saved search with a user who has access to the data
- I’ll assume a saved search with named foo of
- With a user who does not have access to the index with the data:
- Run the same search by hand (e.g.
index=bar) ; no data will be returned
- run the saved search by typing
| savedsearch "foo"and this time data will be returned
While this is very exciting new functionality in terms of the possibilities, I don’t feel that there this has been clearly communicate to users nor the security implications completely thought through. Previously, when asked to audit permissions, looking at the users and roles was enough to determine who had access to what; now saved search permissions must also be considered. Similar to the assertion that ACL’s without deny cannot be properly audited, I’m concerned that this makes the evaluation complex enough that shops are going to inadvertently share data and not catch it even during an audit.
Just like setuid program open the door to unintended consequences, I’m also concerned that this opens the door to a whole host of vulnerabilities including injection attacks (similar to a SQL injection of
foo' OR 1=1 --); previously, there was no concern as the user always had access to the underlying data anyways so there was no additional access to gain — that is until this game changer.