Document Actions
23 Oct 2007
Defeating the BOTNET |
|
Sticking your finder in the dam doesn't work .. unless you're an Octopuss ...
A few weeks ago we noticed our traffic spike by about x100 as a BOTNET focussed it's attention on our main server array. This was enough to cause serious discomfort to our server's ability to render web pages, and to our ability to pay for our transit bill (!)
So, how does one defeat a BOTNET ?
Well I guess the bottom line is that it depends on the BOTNET size, but we've currently collected around 25,000 offending IP's so our method seems to work on a relatively small scale.
The first thing you need is a nice interface to the system's IPTABLES ...
I've written a bit of python code called 'blacklist' (see below) which provides a command line interface to iptables and allows you to easily block or unblock one or more IP addresses. The beauty of the system is that it sticks the results in a MySQL database, hence you can use the results on any machine on your network.
# blacklist info - show stats about collected addressed
# blacklist status - dump the contents of the blacklist
# blacklist block <cidr> <duration> - block an address range for a period
# blacklist unblock <cidr> - remove the address range from the blacklist
# blacklist start - call on system boot, load the list to the kernel
# blacklist stop - call to remove all entries from iptables
Making the list
The next trick is to actually generate the list. For this we have sql_scanner.py (below) which will try to read through all apache log files in /var/log/apache2/* and identify each hit on the server. Setting this up (and maintaining it) can be a little laborious to get going, but it's well worth it if it stops abusers from bringing you down.
The code tries to identify each entry based on client IP, URL etc and if possible, mark the entry as 'dealt with', either updating the white or black list in the process. If it finds an entry it can't automatically cope with, it will ask you. You then have the chance to black or white list the entry either by referrer, ip address or URI.
This information is all stored in the MySQL database and fed down through your IPTABLES as necessary. Once you've run through logs on one machine, the information is then available to other machines on your network, so long as they're all set up with a common SQL database.
Subsequently running sql_scanner.py on another machine will run through it's log files and automatically add any matching IP's it's the IPTABLES, or alternatively stopping and starting blackhole will reload the global ip blacklist from the database.
Anyway, it seems to be pretty effective, here's what the implementation did for us;
Database Structure
Here's the database structure used with the code:/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
CREATE DATABASE /*!32312 IF NOT EXISTS*/ `blackpages` /*!40100 DEFAULT CHARACTER SET latin1 */;
USE `blackpages`;
DROP TABLE IF EXISTS `ips`;
CREATE TABLE `ips` (
`ip` char(20) NOT NULL,
`server` char(20) character set utf8 NOT NULL,
`scope` enum('local','global') NOT NULL default 'global',
`colour` enum('black','white') NOT NULL default 'black',
`start` bigint(20) unsigned zerofill NOT NULL,
`duration` bigint(20) unsigned zerofill NOT NULL,
PRIMARY KEY (`ip`),
KEY `ips_index01` (`server`),
KEY `ips_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS `referrers`;
CREATE TABLE `referrers` (
`referrer` char(128) character set utf8 NOT NULL,
`server` char(20) character set utf8 NOT NULL,
`scope` enum('local','global') NOT NULL default 'global',
`colour` enum('black','white') NOT NULL default 'black',
`start` bigint(11) default NULL,
`duration` bigint(20) default NULL,
PRIMARY KEY (`referrer`),
KEY `referrers_index01` (`server`),
KEY `referrers_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS `uris`;
CREATE TABLE `uris` (
`uri` char(255) NOT NULL,
`server` char(20) character set utf8 NOT NULL,
`scope` enum('local','global') NOT NULL default 'global',
`colour` enum('black','white') NOT NULL default 'black',
`start` bigint(20) default NULL,
`duration` bigint(20) default NULL,
PRIMARY KEY (`uri`),
KEY `uris_index01` (`server`),
KEY `uris_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
Blacklist
This is the source code for the blacklist module, store it in /sbin. Don't forget to change the database settings to match your environment!
#!/usr/bin/python
import sys
import time
import os
import MySQLdb
import MySQLdb.cursors
#
##############################################################################
#
# connect to the database
#
server = MySQLdb.connect(host="mysql",user="blacklist",passwd="blackpass1234",db="blackpages",cursorclass=MySQLdb.cursors.DictCursor)
cursor = server.cursor()
#
##############################################################################
#
# read the hostname (better python method?)
#
wc=os.popen("hostname")
hostname=wc.readline().split("\n")[0]
wc.close()
#
#
##############################################################################
#
data={}
def do_status():
""" print a list of blocked ips """
now = time.time()
print "%-20s\t%-43s\t%8s\t%8s" % ("ip", "blocking since", "duration", "remains")
cursor.execute("SELECT ip,server,scope,colour,start,duration FROM ips ORDER BY ip")
rows = cursor.fetchall()
for row in rows:
start_time = time.strftime("%a %b %d %H:%M:%S %Y",time.localtime(row['start']))
time_left = row['start']+row['duration']-now
print "%-20s\t%-43s\t%8d\t%8d" % (row['ip'],start_time,row['duration'],time_left)
def do_info():
""" print out sql info """
cursor.execute("SELECT colour,COUNT(ip) as count FROM ips GROUP BY colour")
rows = cursor.fetchall()
for row in rows:
print "IP Database [%s] count is %s" % (row['colour'],row['count'])
cursor.execute("SELECT colour,COUNT(referrer) as count FROM referrers GROUP BY colour")
rows = cursor.fetchall()
for row in rows:
print "Referrer Database [%s] count is %s" % (row['colour'],row['count'])
cursor.execute("SELECT colour,COUNT(uri) as count FROM uris GROUP BY colour")
rows = cursor.fetchall()
for row in rows:
print "URI Database [%s] count is %s" % (row['colour'],row['count'])
def do_block(cidr,duration,quiet):
""" block off an ip or ip range """
cursor.execute("SELECT ip FROM ips WHERE ip = '%s'" % cidr)
rows = cursor.fetchall()
if len(rows):
print "IP address (%s) is already blocked" % cidr
return
print "Block (%s) for (%s)" % (cidr,duration)
stdin,stdout = os.popen4("/sbin/iptables -I BLACKLIST -s %s -j DROP" % cidr)
line = stdout.readline()
stdin.close()
stdout.close()
if len(line):
print "Bad IP address: (%s)" % line
return
try:
duration = long(duration)
except:
print "Bad duration (%s)" % duration
return
cursor.execute("BEGIN")
cursor.fetchall()
cursor.execute("INSERT INTO ips (ip,server,scope,colour,start,duration) VALUES ('%s','%s','global','black','%s','%s')" % (cidr,hostname,time.time(),duration))
cursor.fetchall()
cursor.execute("COMMIT")
cursor.fetchall()
if quiet: sys.exit(0)
def do_unblock(cidr):
""" unblock an ip or ip range """
cursor.execute("SELECT ip FROM ips WHERE ip = '%s'" % cidr)
rows = cursor.fetchall()
if not len(rows):
print "IP address (%s) is *NOT* blocked" % cidr
return
print "Unblock (%s)" % (cidr)
stdin,stdout = os.popen4("/sbin/iptables -D BLACKLIST -s %s -j DROP" % cidr)
line = stdout.readline()
stdin.close()
stdout.close()
if len(line):
print "Bad IP address: (%s)" % line
return
cursor.execute("BEGIN")
cursor.fetchall()
cursor.execute("DELETE FROM ips WHERE ip = '%s'" % cidr)
cursor.fetchall()
cursor.execute("COMMIT")
cursor.fetchall()
def do_clear():
""" clear the table """
os.system("/sbin/iptables -F BLACKLIST")
os.system("/sbin/iptables -A BLACKLIST -j RETURN")
def blockip(ip):
""" block an ip address """
stdin,stdout = os.popen4("/sbin/iptables -I BLACKLIST -s %s -j DROP" % ip)
line = stdout.readline()
stdin.close()
stdout.close()
return line
def do_start():
""" load the table from the database """
do_clear()
cursor.execute("SELECT ip FROM ips WHERE colour = 'black'")
rows = cursor.fetchall()
count=1
max = len(rows)
for row in rows:
print ("Loading %d/%d\r" % (count,max)),
sys.stdout.flush()
count += 1
line = blockip(row['ip'])
if len(line):
time.sleep(1)
line = blockip(row['ip'])
if len(line): print "Bad table entry (%s): %s" % (row['ip'],line)
print "\nFirewall reloaded (%s) entries." % len(rows)
if len(sys.argv)<2:
print "Usage: blacklist status | info | start | stop | block <ip> <duration> | unblock <ip>"
sys.exit(1)
def heading():
""" print the heading """
print ""
print "Looking up BlackPages on host ** %s **" % hostname
print ""
if sys.argv[1] == "status":
heading()
do_status()
elif sys.argv[1] == "info":
heading()
do_info()
elif sys.argv[1] == "start":
heading()
do_start()
elif sys.argv[1] == "stop":
heading()
do_clear()
elif sys.argv[1] == "unblock":
if len(sys.argv)<3:
print "Usage: blacklist unblock <cidr>"
else:
heading()
do_unblock(sys.argv[2])
elif sys.argv[1].lower() == "block":
if len(sys.argv)<4:
print "Usage: blacklist block <cidr> <duration>"
else:
if sys.argv[1] == "block":
heading()
quiet = False
else:
quiet = True
do_block(sys.argv[2],sys.argv[3],quiet)
else:
print "No such command (%s)" % sys.argv[1]
print ""
sys.exit(0)
sql_scanner.py
Finally here's the code to scan through your apache logs, bear in mind that you mush use the combined log format otherwise the URL parser won't work. Also note that you may need to install additional python modules, see the first few lines of code for import statements.
#!/usr/bin/python
import string
import sys
import os
import time
import pyparsing
import MySQLdb
import MySQLdb.cursors
#
print ""
##############################################################################
#
# Global Variables
#
progress = 0
last_printed = 0
exceptions = []
#
##############################################################################
#
# read the hostname (better python method?)
#
wc=os.popen("hostname")
hostname=wc.readline().split("\n")[0]
wc.close()
#
##############################################################################
#
# connect to the database
#
server = MySQLdb.connect(host="mysql",user="blacklist",passwd="blackpass1234",db="blackpages",cursorclass=MySQLdb.cursors.DictCursor)
cursor = server.cursor()
#
##############################################################################
#
# handle the optional [--batch] command line argument
#
batch=False
if len(sys.argv)>1 and (sys.argv[1] == "--batch"): batch = True
#
##############################################################################
#
# getCmdFields - used to split a request to components
#
def getCmdFields( s, l, t ):
splits = t[0].strip('"').split()
if len(splits)>0: t["method"] = splits[0]
else: t["method"]="GET"
if len(splits)>1: t["requestURI"] = splits[1]
else: t["requestURI"] = "unknown.com"
if len(splits)>2: t["protocolVersion"] = splits[2]
else: t["protocolVersion"] = "HTTP/1.0"
#
##############################################################################
#
# set up the pyparser
#
integer = pyparsing.Word( pyparsing.nums )
ipAddress = pyparsing.delimitedList( integer, ".", combine=True )
timeZoneOffset = pyparsing.Word("+-",pyparsing.nums)
month = pyparsing.Word(string.uppercase, string.lowercase, exact=3)
serverDateTime = pyparsing.Group( pyparsing.Suppress("[") +
pyparsing.Combine( integer + "/" + month + "/" + integer + ":" + integer + ":" + integer + ":" + integer ) +
timeZoneOffset + pyparsing.Suppress("]") )
logLineBNF = ( ipAddress.setResultsName("ipAddr") + pyparsing.Suppress("-") +
("-" | pyparsing.Word( pyparsing.alphas+pyparsing.nums+"@._" )).setResultsName("auth") +
serverDateTime.setResultsName("timestamp") +
pyparsing.dblQuotedString.setResultsName("cmd").setParseAction(getCmdFields) +
(integer | "-").setResultsName("statusCode") +
(integer | "-").setResultsName("numBytesSent") +
pyparsing.dblQuotedString.setResultsName("referrer").setParseAction(pyparsing.removeQuotes) +
pyparsing.dblQuotedString.setResultsName("clientSfw").setParseAction(pyparsing.removeQuotes) )
#
##############################################################################
#
# load - read a batch
#
def load(sql,key):
"""load a list from a table"""
list=[]
cursor.execute(sql)
rows = cursor.fetchall()
for row in rows:
list.append(row[key])
return list
#
##############################################################################
#
# save - write a batch
#
def save(table,field,colour,list):
"""save a list back to a table"""
if not len(list): return
print "Updating table (%s) with (%s) entries of colour (%s)" % (table,len(list),colour)
if (table=="ips") and (colour=="black"):
for x in list:
os.system("/sbin/blacklist BLOCK %s 999999999" % x)
else:
cursor.execute("BEGIN")
cursor.fetchall()
for x in list:
sql=("INSERT INTO %s (%s,server,scope,colour,start,duration) VALUES ('%s','"+hostname+"','global','%s','%s','999999999')") % (table,field,x,colour,time.time())
cursor.execute(sql)
cursor.fetchall()
cursor.execute("COMMIT")
cursor.fetchall()
#
##############################################################################
#
# preload information from the database
#
ip_blacklist = load("SELECT ip FROM ips WHERE colour = 'black'","ip")
ip_whitelist = load("SELECT ip FROM ips WHERE colour = 'white'","ip")
ref_blacklist = load("SELECT referrer FROM referrers WHERE colour = 'black'","referrer")
ref_whitelist = load("SELECT referrer FROM referrers WHERE colour = 'white'","referrer")
uri_blacklist = load("SELECT uri FROM uris WHERE colour = 'black'","uri")
uri_whitelist = load("SELECT uri FROM uris WHERE colour = 'white'","uri")
ip_blacklist_local = []
ip_whitelist_local = []
ref_blacklist_local = []
ref_whitelist_local = []
uri_blacklist_local = []
uri_whitelist_local = []
#
##############################################################################
#
# merge the blacklist data
#
#io = open("/etc/blacklist.dat")
#while True:
# line = io.readline()
# if not line: break
# line = line.split("\n")[0].split(" ")[0]
# if not line in ip_blacklist: ip_blacklist_local.append(line)
#
#io.close()
#print "[Merging %d entries from /etc/blacklist.dat]" % len(ip_blacklist_local)
#
##############################################################################
#
# print_progress - track our process through the log file
#
def print_progress(p,l):
global last_printed
global batch
if batch: return
if progress>(last_printed+99):
print "[%d of %d]\r" % (p,l),
last_printed = progress
stdout.flush()
return
#
##############################################################################
#
# scan - main routine to itterate through a log file
#
def scan(filename):
"""scan a file for log entries"""
filename="/var/log/apache2/%s" % filename
wc=os.popen('wc -l %s' % filename,"r")
lines=int(wc.readline().split(" ")[0])
wc.close()
print "Scanning '%s' for %d lines" % (filename,lines)
logfile = open(filename,"r")
progress=0
while True:
line = logfile.readline().split("\n")[0]
progress = progress + 1
print_progress(progress,lines)
if not line: break
if line[:3] == "::1": continue
try:
fields = logLineBNF.parseString(line)
except:
print "** Exception: "+line
ip = fields['ipAddr']
if (ip in ip_whitelist) or (ip in ip_whitelist_local): continue
if (ip in ip_blacklist) or (ip in ip_blacklist_local): continue
referrer = ""
if 'referrer' in fields: referrer = fields['referrer'][:255]
#print "Full:",referrer
ref=referrer
if referrer[:7] == "http://": ref=ref[7:]
ref=ref.split("/")[0]
#print "Short:",ref
#if len(referrer):
# if (referrer in ref_whitelist) or (referrer in ref_whitelist_local): continue
# if (referrer in ref_blacklist) or (referrer in ref_blacklist_local): continue
uri = fields['requestURI']
if (uri[:7]=="http://") or (uri[:8]=="https://") or (fields['method']=="CONNECT"):
ip_blacklist_local.append(ip)
continue
if (uri in uri_whitelist) or (uri in uri_whitelist_local):
ip_whitelist_local.append(ip)
continue
if (uri in uri_blacklist) or (uri in uri_blacklist_local):
ip_blacklist_local.append(ip)
continue
if (ref in ref_whitelist) or (ref in ref_whitelist_local):
ip_whitelist_local.append(ip)
continue
if (ref in ref_blacklist) or (ref in ref_blacklist_local):
ip_blacklist_local.append(ip)
continue
if batch: continue
#if len(referrer)>1:
# default="Blacklist Referrer"
#elif len(ip)>1:
# default="Blacklist URI"
#else:
# default="Blacklist IP"
print "+---------------------------------------------------------------------+"
print "| * * * * Manual clarification required for unknown log entry * * * * |"
print "| |"
if len(referrer):
print "| Referrer ..... %-52s |" % referrer
print "| Ref Domain ... %-52s |" % ref
print "| Address ...... %-52s |" % ip
print "| URI .......... %-52s |" % uri[:55]
print "+---------------------------------------------------------------------+"
while True:
print "[q]=Quit [b]=BlackList [w]=WhiteList [return]=WhitelistIP (q,b,w,return) > ",
choice = sys.stdin.readline().split("\n")[0].lower()
if choice == 'q': break
if choice == '':
ip_whitelist_local.append(ip)
break
# if default == "Blacklist Referrer": ref_blacklist_local.append(referrer)
# if default == "Blacklist URI": uri_blacklist_local.append(uri)
if (choice <> 'w') and (choice <> 'b'): continue
while True:
print "[r]=By Referrer [i]=By IP [u]=By URI > ",
action = sys.stdin.readline().split("\n")[0].lower()
if action == "r":
if choice == "w":
ref_whitelist_local.append(ref)
else:
ref_blacklist_local.append(ref)
break
if action == "i":
if choice == "w":
ip_whitelist_local.append(ip)
else:
ip_blacklist_local.append(ip)
break
if action == "u":
if choice == "w":
uri_whitelist_local.append(uri)
else:
uri_blacklist_local.append(uri)
break
break
logfile.close()
#
##############################################################################
#
# main routine
#
print ("Scanning Log files on host ** %s **" % hostname),
if batch:
print " (batch mode)"
else: print ""
for f in os.listdir("/var/log/apache2/"):
if f[-4:]==".log":
if f[-9:]=="error.log": continue
scan(f)
#
print ""
#
# save all our changes
#
save("ips" ,"ip" ,"black",ip_blacklist_local)
save("ips" ,"ip" ,"white",ip_whitelist_local)
save("referrers","referrer" ,"black",ref_blacklist_local)
save("referrers","referrer" ,"white",ref_whitelist_local)
save("uris" ,"uri" ,"black",uri_blacklist_local)
save("uris" ,"uri" ,"white",uri_whitelist_local)
#
# Quit here!
#
print ""
Have Fun!
- The URL to Trackback this entry is:
- http://hairy.trollstomper.org.uk/the-trolls-blog/defeating-the-botnet/tbping