Random post that isn't part of the 'series' as such... I did say I might!
You would think sending or monitoring file changes to Loggly via rsyslog over TLS would be the default. Sadly as I write this, it isn't and I'm guessing won't be any time soon given I've found posts older than a year saying it will be 'soon'. That rhymes with 'June' right? And if you don't say which year.....
While the good news is that Loggly do have all the information you need available in their documentation, the bad news is, its not clear, and forgets to tell you some of the needed steps you have to follow for file based monitoring.
Those of you more familiar with configuring rsyslog will already know how to do this, but if you are like me and don't create rsyslog configurations every day, you may also miss the obvious and need some help.
Lets start off with what you need:
- a Linux server to send logs from
- an application on that server creating logs
- a Loggly account
- your 'token' from your Loggly account
I'm Amazon Web Services based for this blog, so the following is all based on an EC2 instance using the Amazon Linux 64bit HVM AMI.
I won't be going into detail on how to set up your Linux server, write an application that creates logs or setting up your Loggly account - that may come later in this blog, but not today!
Loggly do have a non-automated way of installing monitoring over your main syslog events using rsyslog detailed here. This works quite well.
What doesn't work quite well is trying to get the monitoring for a file working. The requirements (rsyslog-gnutls for example) are pretty much the same, but for some reason Loggly only give you half the config needed. The part they forget is to tell you to include part of the base syslog monitoring config as well, and if you have just set up syslog monitoring, you may think its implied - it isn't.
If you follow the Loggly instructions for 'Simple File Monitoring' here, you get told to put the following in a file called /etc/rsyslog.d/21-filemonitoring-loggly.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# Input for FILE1
$InputFileName /FILE1
$InputFileTag APPNAME1
$InputFileStateFile stat-APPNAME1 #this must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# Add a tag for file events
$template LogglyFormatFile,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"file\"] %msg%\n"
# Send to Loggly then discard
if $programname == 'APPNAME1' then @@logs-01.loggly.com:514;LogglyFormatFile
if $programname == 'APPNAME1' then ~
After changing FILE1 (one line), APPNAME1 (4 lines) and TOKEN (1 line) appropriately you have just told your server to send all log changes from that file in plain text over the internet. Not always (if ever) the appropriate thing to do.
Changing the config to point at the TLS port in the second to last line is also not enough - you do need to change this line, but that is not the end of it.
For example, you change the line from:
if $programname == 'APPNAME1' then @@logs-01.loggly.com:514;LogglyFormatFile
to:
if $programname == 'APPNAME1' then @@logs-01.loggly.com:6514;LogglyFormatFile
The other part you are missing, and which is also not listed in the Loggly help article, is to also paste in the TLS configuration that was mentioned in how to set up base syslog monitoring:
#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/loggly_full.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.loggly.com
Now you should have a larger file that looks like this:
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# Input for FILE1
$InputFileName /FILE1
$InputFileTag APPNAME1
$InputFileStateFile stat-APPNAME1 #this must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/loggly_full.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.loggly.com
# Add a tag for file events
$template LogglyFormatFile,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"file\"] %msg%\n"
# Send to Loggly then discard
if $programname == 'APPNAME1' then @@logs-01.loggly.com:6514;LogglyFormatFile
if $programname == 'APPNAME1' then ~
In hind-sight, this was pretty obvious. You can be sure it was very frustrating up until I figured out those lines were also needed of course!
Side note: I truncated the last line to just be & ~ instead of the full if $programname... line. Its the same thing (repeat previous but discard) and looks much tidier in my mind.
I also don't like the 'tag=file' in Loggly, so change the tag to be the same as my application name.
If you are after filtering out some of the messages - for example I use a load-balancer in front of my application server, and don't want Loggly filled up with heartbeat messages. To stop this from happening, I put in a simple 'ignore messages with this content' request in my config and Loggly is none the wiser as a result.
My full config file then looks like this (using applicationanme as the app name, file is the full location of the log file, and I left TOKEN as TOKEN so that you don't spam my Loggly account ;)
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# Input for applicationname.log
$InputFileName /usr/applicationlocation/applicationname.log
$InputFileTag applicationname:
$InputFileStateFile stat-applicationname
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/loggly_full.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.loggly.com
# Add a tag for file events
$template LogglyFormatFile,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"applicationname\"] %msg%"
# First discard heartbeat messages
if $msg contains '/api/heartbeaturl' then ~
# Send to Loggly then discard
if $programname == 'applicationname' then @@logs-01.loggly.com:6514;LogglyFormatFile
& ~
All looks fairly straight-forward with a full example im my not so humble opinion!
Now, given this blog is going to have some Ansible related information in it, you could ask, what does that look like?
I've published the Ansible files (mostly just snippets reformed into a working set of Ansible files specific to this article) in my GitHub account in a repo called devops-all-the-things if you would like to see what that can look like.
The main points for the files in the repo are:
- This contains the full install of syslog monitoring (replace the TOKEN!) as well as an example of an application specific playbook
- Rather than grab the certificate files every time the playbook is run, I just copy over a known good copy I have already created - feel free to download the current certificate files and concatenate them together if you want to
- I use a role called 'common' to give myself the equivalence of global variables and handlers, and the file location will need to be correct for wherever you run Ansible from
- I prefix the names of my files to keep the same type of file type together
- I exclude most of the Ansible messages that go to syslog from also going to Loggly. As mentioned in the main blog series, I use Rundeck to watch my Ansible logs - Loggly is for application debugging, not Ansible debugging if you get what I mean...
- In that vein, I also have a ton of debug statements in my Ansible code
- I pass hostnames quite frequently to my playbooks as extra-vars. To be exact I usually use tags from AWS (tag_Application_appname for example) to hit the right groups of servers
- You will need to use your own TOKEN from Loggly
Don't worry if the playbooks make no sense if you are new to Ansible - that is something I will cover in this blog later on.