Low Orbit Flux Logo 2 D

Splunk

Even if you don’t use it you have probably at least seen adds for Splunk. I remember back when it was relatively new. I know of its existence mainly because the adds were everywhere. You couldn’t visit a technical site without seeing an add for Splunk. This wasn’t too long before I was responsible for running a POC on our servers.

These days everybody knows about Splunk although not everybody has a full understanding of what it does.

What is Splunk

In short, Splunk is a log collection system. It does a whole lot more than that though. The main goal was to solve the problem of having logs spread across multiple different systems with no good way to correlate events from different systems. Basically, Splunk collects logs from any number of different sources and ingests them into a centralized system. The logs can then be searched allowing for the correlation of events across different systems. It also features a powerful but easy to use search language for creating custom queries. This allows you to search for anything you want and manipulate the results in all sorts of ways.

Typically data would be gathered from an agent running on a server. This would be configured to forward logs to a centralized server. Data can be taken from a large number of different source types.

Learn Splunk

Install Splunk Enterprise


tar xvfz splunk-8.0.1-6db836e2fb9e-Linux-x86_64.tgz
sudo mv splunk /opt

cd /opt/splunk/bin/

Initial startup ( accept license and create admin account ):


./splunk start --accept-license

http://docker1:8000


./splunk enable boot-start
sudo ./splunk enable boot-start

Creates legacy init script that still works on systemd based system:

/etc/init.d/splunk

You won’t see it here:


systemctl list-unit-files | grep enabled

Disable start on boot with this:


sudo ./splunk disable boot-start

Enable Splunk start on boot with systemd:


sudo ./splunk enable boot-start -systemd-managed 1

Now you will see it listed:


systemctl list-unit-files | grep enabled

You can now start and stop it like this:


sudo systemctl start Splunkd
sudo systemctl stop Splunkd

Splunk Enable Receiving

You need to enable receiving before you can actually receive data from your forwarders. This can be done from the GUI with the following steps.

Splunk Enable Receiving CLI

You can also do this from the CLI if you want.


vi /opt/splunk/etc/system/local/inputs.conf
[splunktcp://9997]
disabled = 0

Splunk Create Index

Install The Splunk Universal Forwarder


tar xvzf splunkforwarder-<…>-Linux-x86_64.tgz

rpm -i splunkforwarder-<…>-linux-2.6-x86_64.rpm
dpkg -i splunk_package_name.deb

tar xvfz splunkforwarder-8.0.1-6db836e2fb9e-Linux-x86_64.tgz
sudo mv splunkforwarder /opt

cd /opt/splunkforwarder/bin

For the first time starting, start the forwarder like this to accept the license without reading it:


cd $SPLUNK_HOME/bin
./splunk start --accept-license

Starting the forwarder:


cd $SPLUNK_HOME/bin
./splunk start

Restart after a config change:


cd $SPLUNK_HOME/bin
./splunk restart

Enable start on boot. This makes whatever changes need to be made for your system.


$SPLUNK_HOME/bin/splunk enable boot-start

Enable start on boot. ( non-root )


$SPLUNK_HOME/bin/splunk enable boot-start -user bob

Edit this and add “su”, see https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/ConfigureSplunktostartatboottime


vi /etc/init.d/splunk

Systemd


$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1

cd $SPLUNK_HOME/bin
sudo ./splunk add forward-server docker1:9997 -auth admin:password1
sudo ./splunk add monitor /var/log -sourcetype linux_logs -index my-test-index
sudo ./splunk add monitor /var/log -sourcetype journald -index my-test-index2
sudo ./splunk restart

Two versions of the default config file, don’t use these:

$SPLUNK_HOME/etc/system/default
$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

The config commands above will modify these files. They can be modified with an editor too.

$SPLUNK_HOME/etc/system/local/outputs.conf
$SPLUNK_HOME/etc/apps/search/local/inputs.conf

Splunk Minimum Free Disk Space

If you see a message like this:

The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.

You have a few options:

You can decrease the threshold like this:

sudo vi $SPLUNK_HOME/etc/system/local/server.conf


[diskUsage]
minFreeSpace = 50
pollingFrequency = 100000
pollingTimerFrequency = 10