Anthony Reinke

Posts Tagged ‘splunk’

Sorry about the lack of the posting.  I have switched jobs and the perfect storm hit us.  Due to a power outage (well,multiple outages) from Lincoln Electric System (http://les.com) we lost a main switch, the firewall configuration, and dhcp configuration.  I haven’t logged in to a Cisco device in many years and never in the depth I have had to in the past month.

Things I am working on:

• Jumping in to Cisco head first
• More with Splunk
• Setting up MPLS between multiple site, and a couple of co-locations
• More in to security
• Guest Posters

I prefer to use a signed web certificate and not the self signed certificate.  I found a couple different topics on the process, but found that most of them referred to the distributive searching certificate.  Here are the step to generate the certificate and get it in to the right place for Splunk to use it.

—————————————————————–

## Generate the local key
openssl genrsa -out linux0001.key 4096

## Generate the csr
opensll req -new -key linux0001.key -out linux0001.csr

## Submit the .csr file to the CA

## Move the original certs for backup purposes
mv cert.pem cert.pem.bak
mv privkey.pem privkey.pem.bak

## Convert the binary cert to a standard cert
openssl x509 -in certnew.cer -inform DER -out cert.pem -outform PEM

## Copy the new files in the Splunk folder
cp linux0001.key /opt/splunk/share/splunk/certs/privkey.pem
cp cert.pem /opt/splunk/share/splunk/certs/cert.pem

## Restart Splunk
/opt/splunk/bin/splunk restart

I have begun building my own dashboards in Splunk.  Once I have the custom views built, I will post them up here.  So far everything I have been working on is with a system’s administrator in mind because that is what I have been doing for the past 12 years (wow, thats a long time).  Currently I am building a view for searching failed logins and the source of lockouts.  They tie in to one another.  Our technicians want to be more involved in the systems administration and hopefully this will help them respond quicker to our customers.  Everything comes from Splunk being installed on all our domain controllers.  From there we get all the logs in to our central logging system (Splunk).  Due to the amount of data we are pushing now everyday, we might have to build a backup environment just for our Splunk data.  How awesome is this!

This will help to track down failed logins.  This could be due to someone changing their password and still are logged in to a server with the old account information.  The other side is that someone could be trying to brute force an account.

Type=”Failure Audit” sourcetype=”WinEventLog:Security” | chart count by User_Name | sort – count

So I am a full convert and profit of Splunk now.  I have been using it at work for around 4 months now.  I have rolled it out to our domain controllers and have started rolling it to all our Windows and *nix servers.  The ability to find out who did what has made my job so much easier.  There was an incident where an OU was deleted in our AD.  I was able to see exactly who and when did it.  Normally this type of searching wasn’t possible or at least hard to get due to the size of our infrastructure.  Our Event Logs roll over around once an hour.  The OU was deleted 8 hours before we were contacted.

Here is a few of the reports I have scheduled to get every morning to take a look at what has happened in my environment.

User Accounts deleted:

EventCode=”630″ | fields Caller_User_Name, Target_Domain,  Target_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Deleted_Account | rename host as DomainController | rename Target_Domain as Users_Domain

User Accounts created:

EventCode=”624″ | fields Caller_User_Name, New_Domain, New_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Modified_Account | rename host as DomainController | rename New_Domain as New_Account_Domain

Computer Accounts deleted:

EventCode=”647″ | fields Caller_User_Name, Target_Domain, Target_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename Target_Account_Name as Deleted_Computer | rename host as DomainController | rename Target_Domain as Removed_Domain

Computer Accounts created:

EventCode=”645″ | fields Caller_User_Name, New_Domain, New_Account_Name, host | collect | rename Caller_User_Name as Who_Did_It | rename New_Account_Name as New_Computer | rename host as DomainController | rename New_Domain as Joined_Domain