Posts Tagged ‘splunk’

I love Splunk. The way it simplifies my job has been completely changed due to Splunk. The number one resource I use is the Quick Reference Guide. You can print one out and then it makes it easy to just quickly grab it to complete your searches. I don’t know how many times I have used it. I have worn out two of them.

Recently I shared that I was working with Continuum ( to start a Splunk User Group in the Lincoln/Omaha area (  Since then Mike Mizener ( has found us a location and we agreed upon a first meeting day.  We will be meeting on Tuesday February 26th from 6pm to 9pm at Charlie’s on the Lake (  For this first meeting our topic will be: What’s new in Splunk 5.0.  More details coming but if you have ideas for topics or any other questions, please let me know.

I am currently working with Continuum ( to bring the Lincoln/Omaha area of Nebraska a Splunk user group. I am a big believer in the sharing of knowledge. With that I love to go on to the Splunk Answers site and review issues or questions people have and try to help them. When I was learning IT, someone took the time to answer my questions. I want to give back to the community that has taught me so much. This is where my sports life meets my geek life. I want to be that coach to help others get the most of IT. Look for more information shortly.

I recently spoke at the Splunk>Live Kansas City conference.  I was one of three speakers giving their user experiences working with Splunk.  The speakers along with two other guests also sat in a Q&A session over lunch.  I also had the chance to have dinner with a few of the people that work with Splunk.  I enjoyed speaking with the Splunk employees and other people interested in the Splunk product.  I wish I would have had someone record the talk I gave but I have included the slides below.


Sorry about the lack of the posting.  I have switched jobs and the perfect storm hit us.  Due to a power outage (well,multiple outages) from Lincoln Electric System ( we lost a main switch, the firewall configuration, and dhcp configuration.  I haven’t logged in to a Cisco device in many years and never in the depth I have had to in the past month.

Things I am working on:

  • Jumping in to Cisco head first
  • More with Splunk
  • Setting up MPLS between multiple site, and a couple of co-locations
  • More in to security
  • Guest Posters

I prefer to use a signed web certificate and not the self signed certificate.  I found a couple different topics on the process, but found that most of them referred to the distributive searching certificate.  Here are the step to generate the certificate and get it in to the right place for Splunk to use it.


## Generate the local key
openssl genrsa -out linux0001.key 4096

## Generate the csr
opensll req -new -key linux0001.key -out linux0001.csr

## Submit the .csr file to the CA

## Move the original certs for backup purposes
mv cert.pem cert.pem.bak
mv privkey.pem privkey.pem.bak

## Convert the binary cert to a standard cert
openssl x509 -in certnew.cer -inform DER -out cert.pem -outform PEM

## Copy the new files in the Splunk folder
cp linux0001.key /opt/splunk/share/splunk/certs/privkey.pem
cp cert.pem /opt/splunk/share/splunk/certs/cert.pem

## Restart Splunk
/opt/splunk/bin/splunk restart

I have begun building my own dashboards in Splunk.  Once I have the custom views built, I will post them up here.  So far everything I have been working on is with a system’s administrator in mind because that is what I have been doing for the past 12 years (wow, thats a long time).  Currently I am building a view for searching failed logins and the source of lockouts.  They tie in to one another.  Our technicians want to be more involved in the systems administration and hopefully this will help them respond quicker to our customers.  Everything comes from Splunk being installed on all our domain controllers.  From there we get all the logs in to our central logging system (Splunk).  Due to the amount of data we are pushing now everyday, we might have to build a backup environment just for our Splunk data.  How awesome is this!

This will help to track down failed logins.  This could be due to someone changing their password and still are logged in to a server with the old account information.  The other side is that someone could be trying to brute force an account.

Type=”Failure Audit” sourcetype=”WinEventLog:Security” | chart count by User_Name | sort – count

So I am a full convert and profit of Splunk now.  I have been using it at work for around 4 months now.  I have rolled it out to our domain controllers and have started rolling it to all our Windows and *nix servers.  The ability to find out who did what has made my job so much easier.  There was an incident where an OU was deleted in our AD.  I was able to see exactly who and when did it.  Normally this type of searching wasn’t possible or at least hard to get due to the size of our infrastructure.  Our Event Logs roll over around once an hour.  The OU was deleted 8 hours before we were contacted.

Here is a few of the reports I have scheduled to get every morning to take a look at what has happened in my environment.

User Accounts deleted:

EventCode=”630″ | fields Caller_User_Name, Target_Domain,  Target_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Deleted_Account | rename host as DomainController | rename Target_Domain as Users_Domain

User Accounts created:

EventCode=”624″ | fields Caller_User_Name, New_Domain, New_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Modified_Account | rename host as DomainController | rename New_Domain as New_Account_Domain

Computer Accounts deleted:

EventCode=”647″ | fields Caller_User_Name, Target_Domain, Target_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Deleted_Computer | rename host as DomainController | rename Target_Domain as Removed_Domain

Computer Accounts created:

EventCode=”645″ | fields Caller_User_Name, New_Domain, New_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename New_Account_Name as New_Computer | rename host as DomainController | rename New_Domain as Joined_Domain

Great article from Michael Wilde on how to see how much data you are indexing from Splunk