Idea ID: 2860475

Improve SMTP AUTH failure logging (simple enhancement!)

Status : New Idea

BRIEFLY:  When an SMTP AUTH failure is logged, please also include the client connection IP and the username being "attempted" in the same logfile line as the failure notification.

WHY:  This will make it easier for us to determine what machines (IP) are trying to hack into our SMTP servers, and what account names they are trying to compromise in order to do this.

RELATED:  This enhancement request is related to another (much more powerful) request I previously submitted: 

/cyberres/smg/i/smg_ideas/smtp-auth-failure-detection-and-blocking

However, this request should be quite easy (and faster) to accomplish.  Simply add the requested info to the log file entry (as the product already has this data on hand).

As we now have SMTP AUTH functionality, I highly recommend EVERYONE search their SMG-SMTP logs for SMTP AUTH failures to see how big a problem this is.  Literally minutes after an SMTP service becomes available on the Internet, you will see hackers starting to try and hack into your system.

If you have SMTP AUTH enabled, here is how you can see how bad it is on YOUR SERVER.

1.  Download your latest SMG-SMTP log file.

2.  Search (grep) the log file for any/all occurrences of the phrase "Authentication failed".  This will show you all the auth failures.  What it doesn't show you is what the offending IP is or what account is they were attempting to compromise...

3.  Search for the phrase "Could not authenticate connection" and then you will see all the names of the accounts hackers were trying to compromise.  (This logfile entry seems to follow immediately after the one above so it might actually be a better all-around choice for manual scanning as it also provides user names)

4.  You can then look backwards in the log file for the same thread id where it most recently logged "Processing SMTP client connection" and it will tell you the offending IP.  (You can also search forward for the threads next "Processing complete for connection" to find the IP - but ultimately, I want to slam the door on them (block their IP) BEFORE they get this far...)

So here is what a snippet from my log file looks like.  I have isolated the logfile entries only for this thread, only for this transaction, and then removed all but the signature entries I mentioned above:

 

[140674036905728] 2021-03-02 17:15:42 (SMTP) Processing SMTP client connection from 87.246.7.243 (client count 2)
...
[140674036905728] 2021-03-02 17:15:50 (SRVS)<747> Authentication failed to server 10.1.2.3: 501 Authentication failed
[140674036905728] 2021-03-02 17:15:50 (SRVS)<747> Could not authenticate connection with host 10.1.2.3 with user **PERSONAL INFORMATION REMOVED** as required
...
[140674036905728] 2021-03-02 17:15:52 (SMTP)<747> Processing complete for connection from 87.246.7.243

 

If SMG could simply add the IP to the third logfile entry above, then we could have access to all the "salient" info by searching for the string "Could not authenticate connection".  For example, something like:

[140674036905728] 2021-03-02 17:15:50 (SRVS)<747> Could not authenticate connection with host 10.2.2.2 with user **PERSONAL INFORMATION REMOVED** from 87.246.7.243 as required

Of course, the logfile entry search phrase ("Could not authenticate connection") must be unique (and it appears to be)...

I highly recommend EVERYONE look at their SMTP log files to see how widespread this type of hacking is.  For example, the current logfile I downloaded shows 659 log in hacking attempts in its short 1hr and 35 minute time span!

This is a REAL PROBLEM!

 

History:  I wrote a bash script several years ago which ran along side the GWAVA product to watch the GWVSMTP log file IN REAL TIME for these failures.  It ran under SLES, so I had lots of software utilities available to my script.  ...and I could load/install more if needed.  (I also wrote a similar script which runs on my PostFix SMTP server to watch its log file (which DOES directly include the all the salient info in a single log file entry).

Anyway, processing log files (in real time) which interleave thread entries increases the complexity of associating data posted earlier (IP Address) by a specific thread so it can be used later if/when it is needed.  This additional "tracking" might very well require tools which are not available on the SMG appliance.  (This also assumes I will be able to run user scripts/processes on the SMG appliance...)  Anyway, putting all the salient info in a single log file entry makes it much more straight forward to extract the data I need in real time.

While processing the log file in real time (tail -f), I watched for these events (SMTP failure) and then pushed an IP address up to my firewall to be blocked immediately.  Adding the IP address to the blacklist on my firewall is not a simple process, but it worked exceptionally well.  (It should be noted that locally blocking the IP using the linux "iptables" firewall is a much simpler process - but I didn't want ANY traffic from the offending IP into my network at all).

I also ran a cron job every night to give me a nightly report of the previous days offending IP's (now blocked), what user accounts they were trying to compromise (so I could notify users), and then flush old blocked IP's which I expired after a pre-defined interval (2 weeks)

As noted earlier, this is a huge problem.  No one seems to appreciate how bad this is unless they actually look for it.  The shocking truth is no product seems to do anything about it!  I have asked for this to be included in GWAVA (it wasn't) and I also asked for it to be included in my firewall (which has its own SMTP proxy) and again it wasn't.

Log File Enhancement Requests aside:  The Secure Messaging Gateway is an email security product.  If we can recognize this is a huge problem, and it should be quite simple for the device to block offending IP's, they why doesn't it do anything about it?

 

Do yourself a favor and take a look through your SMTP log file for SMTP Auth failures!

Tags:

Labels:

Other