Anthony Reinke » splunk

Lack of posts

Anthony Reinke — Sun, 28 Nov 2010 04:22:52 +0000

Sorry about the lack of the posting. I have switched jobs and the perfect storm hit us. Due to a power outage (well,multiple outages) from Lincoln Electric System (http://les.com) we lost a main switch, the firewall configuration, and dhcp configuration. I haven’t logged in to a Cisco device in many years and never in the depth I have had to in the past month.

Things I am working on:

Jumping in to Cisco head first
More with Splunk
Setting up MPLS between multiple site, and a couple of co-locations
More in to security
Guest Posters

Web (http) Certificate for Splunk

Anthony Reinke — Fri, 09 Jul 2010 16:17:13 +0000

I prefer to use a signed web certificate and not the self signed certificate. I found a couple different topics on the process, but found that most of them referred to the distributive searching certificate. Here are the step to generate the certificate and get it in to the right place for Splunk to use it.

—————————————————————–

## Generate the local key
openssl genrsa -out linux0001.key 4096

## Generate the csr
opensll req -new -key linux0001.key -out linux0001.csr

## Submit the .csr file to the CA

## Move the original certs for backup purposes
mv cert.pem cert.pem.bak
mv privkey.pem privkey.pem.bak

## Convert the binary cert to a standard cert
openssl x509 -in certnew.cer -inform DER -out cert.pem -outform PEM

## Copy the new files in the Splunk folder
cp linux0001.key /opt/splunk/share/splunk/certs/privkey.pem
cp cert.pem /opt/splunk/share/splunk/certs/cert.pem

## Restart Splunk
/opt/splunk/bin/splunk restart

Splunk Dashboards

Anthony Reinke — Mon, 03 May 2010 02:24:38 +0000

I have begun building my own dashboards in Splunk. Once I have the custom views built, I will post them up here. So far everything I have been working on is with a system’s administrator in mind because that is what I have been doing for the past 12 years (wow, thats a long time). Currently I am building a view for searching failed logins and the source of lockouts. They tie in to one another. Our technicians want to be more involved in the systems administration and hopefully this will help them respond quicker to our customers. Everything comes from Splunk being installed on all our domain controllers. From there we get all the logs in to our central logging system (Splunk). Due to the amount of data we are pushing now everyday, we might have to build a backup environment just for our Splunk data. How awesome is this!

Another good report

Anthony Reinke — Wed, 24 Mar 2010 14:54:34 +0000

This will help to track down failed logins. This could be due to someone changing their password and still are logged in to a server with the old account information. The other side is that someone could be trying to brute force an account.

Type=”Failure Audit” sourcetype=”WinEventLog:Security” | chart count by User_Name | sort – count

Daily Splunk Reports

Anthony Reinke — Wed, 24 Mar 2010 13:45:14 +0000

So I am a full convert and profit of Splunk now. I have been using it at work for around 4 months now. I have rolled it out to our domain controllers and have started rolling it to all our Windows and *nix servers. The ability to find out who did what has made my job so much easier. There was an incident where an OU was deleted in our AD. I was able to see exactly who and when did it. Normally this type of searching wasn’t possible or at least hard to get due to the size of our infrastructure. Our Event Logs roll over around once an hour. The OU was deleted 8 hours before we were contacted.

Here is a few of the reports I have scheduled to get every morning to take a look at what has happened in my environment.

User Accounts deleted:

User Accounts created:

Computer Accounts deleted:

Computer Accounts created:

Getting more intelligence on how much data splunk is eating.

Anthony Reinke — Mon, 18 Jan 2010 19:02:57 +0000

http://www.splunkninja.com/profiles/blogs/getting-more-intelligence-on

Great article from Michael Wilde on how to see how much data you are indexing from Splunk

Searching for Account Lockouts with Splunk

Anthony Reinke — Mon, 21 Dec 2009 22:03:47 +0000

This requires that the Splunk agent is getting the security event from the Domain Controller(s).

Find the username of the person
Log in to the Splunk server.
Click on the Search button.
Enter the search paramitters to find the user and select your time frame for the search:
source=”WinEventLog:Security” User_Name=”lockedUser”
Then check the “Client_Address” field. This can be found on the left column.
The client IP shows where the lockout came from.

1. Find the username of the person

2. Log in to the Splunk server.

3. Click on the Search button

4. Enter the search paramitters to find the user and select your time frame for the search:
source=”WinEventLog:Security” Type=”Failure Audit” User_Name=”lockedUser”

5. Then check the “Client_Address” field. This can be found on the left column

6. The client IP shows where the lockout came from.

Monitoring the Filesystem with Splunk

Anthony Reinke — Mon, 31 Aug 2009 19:17:06 +0000

I have used OSSEC in the past to watch the file system for changes. When I found that I can have the Splunk agent handle the monitoring itself, I was pretty excited. Since I would send my OSSEC data to Splunk anyways, it just seemed logical to have Splunk do everything.

In Windows, you need to edit the “c:program filesSplunketcsystemlocalinputs.conf” file. Of course your path could be different if you installed it in a different place. There are a lot of options and switches you can use. I went for the simplest set.

[fschange:d:temp]
recurse=true
pollPeriod=3600

This will monitor the d:temp folder and all files and folders under it. It will check the system every 3600 seconds (1 hour).

This has helped me keep track of the changes in my servers. I can see when a file was add/deleted/changed (due to the hash) and then look at who was logged in during the period that the file was changed.

Splunk article on the switches and FSCHANGE.
http://www.splunk.com/base/Documentation/4.0.3/Admin/Monitorchangestoyourfilesystem

[fschange:d:temp]
recurse=true
followLinks=false
pollPeriod=60

RegEx with Splunk for OSSEC

Anthony Reinke — Fri, 31 Jul 2009 04:12:46 +0000

Thanks to Michael Wilde for the information on RegEx in Splunk. For those like me who aren’t the best at RegEx, I will show some of the regular expressions I am using for OSSEC.

Server Name
(?i) Location:s((?P.*?))s

Windows Event User
(?i) USER: (?P[^:]*);

Server IP
(?i)^[^)]*)s+(?P[^-]*)-

Windows Events
(?i)^[^-]*-s+(?P[^.]*).

LogInUser
(?i) Name: (?Pw+)

LogInDomain
(?i) Domain: (?P[^ ]*)[ ]

******************************************************

Now, to add them…

Open your browser and login in to your Splunk server. In the Search application, type sourcetype=”ossec”

or click on “ossec” in the Sourcetypes

You should see a bunch of data from the OSSEC server. On the left of the main frame of the webpage, there should be a grey down arrow. Clicking on this I get two options. You want to select Extract Fields.

Here is where it gets fun. Splunk included a graphical RegEx builder based on examples. I ended up playing with this for a while. Once you have found the expression you like, click on the Save button.

Name your RegEx and click Save.

Restart your Splunk server.
Once restarted, on the main search page, on the left sidebar click on Pick fields.

Here you can select the fields that will be displayed on the search page.

When you get back to the search page, you will notice the new fields.

OSSEC and Splunk

Anthony Reinke — Tue, 28 Jul 2009 01:55:54 +0000

I have been playing with OSSEC and Splunk. OSSEC is a Host based Intrusion Detection System (HIDS). Splunk is a log archiving and searching system. OSSEC is open source and is multiple platform. You can run it on Linux/Unix and Windows. I am using OSSEC to forward Windows Event Logs to Splunk. Splunk makes the searching and correlation. Splunk can do WMI. This would be great since no agent would need to be installed. The problems is that if you have more than 30-50 systems, the amount time and traffic would cause issues. Using the OSSEC agent, I am able to push the event logs to the OSSEC server. From there the OSSEC server will upload to the Splunk server via Syslog.

Right now I have the servers all talking but I do need to adjust a few things. Right now Splunk sees all the hosts as the OSSEC server. I believe I just need to tweak the fields. The question is how.

Splunk
http://www.splunk.com

OSSEC
http://www.ossec.net