Apache request-based throttling

Ok, theoretically my last post about mod_rpaf was supposed to lead to mod_qos working. It did, in the most technical way… it just made it instantly obvious that mod_qos was not the solution I was looking for! mod_qos performs qos on a URI but applies it to all connecting clients, not just offenders. It’s best used for resource limiting… not in API throttling to put a stop to abuse, which is my intent.

I grudgingly turned to mod_security. I’ve known all along that mod_security would be the best tool to help me reach my goal; however, mod_security is the least user-friendly piece of software that I’ve ever used, with a highly esoteric language and odd processing rules. Forced to sit down and make it work, however, I’ve come up with a few rules that may help others who wish to perform request-based throttling.

SecAction "phase:2,pass,nolog,initcol:IP=%{REQUEST_HEADERS.X-Forwarded-For}"
SecAction "phase:2,nolog,setvar:IP.hitcount=+1,deprecatevar:IP.hitcount=1/1"
SecRule IP:hitcount "@gt 3" "phase:2,pause:3000,nolog,allow,msg:'API abuser, throttling'"

First, I initialize a collection called “IP”, based on the X-Forwarded-For header. Because I’m using mod_rpaf, I could technically use the remote address, but “just in case” I opted for the X-Forwarded-For, since that’s much more important to me. It also prevents the load balancer from getting blocked… ever.

Second line is where I do the IP increment — and decrement. As you can see, for every hit from that IP I increment the IP.hitcount variable by 1; the ‘deprecatevar:IP.hitcount=1/1’ tells the variable to decrement the count by one per second. If the user makes one hit per second, they will never hit the limit. If they make 2 hits per second, the net gain will be 1 one first second, 2 the next, 3 the next, etc.

The last line, of course, is where we do our test. If the hitcount is greater than 3, I’m allowing the request to go through, but adding a 3000ms pause — 3 seconds.

I configured these rules within my VirtualHost definition, and used Location tags to specify the URIs that require throttling. It works like a champ. In each of the rules, I’ve specified ‘nolog’, as it’s pretty spammy, though you’ll want to change that to ‘log’ for testing. Because I’m disabling mod_security’s spammy logging, I’m timing requests with a custom log format:

LogFormat "%h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" %D" combined-time
CustomLog "/var/log/httpd/access_log" combined-time

The %D at the end of the LogFormat spits out the total time taken by Apache to fulfill the request in microseconds, which will include the artificial delay. With this CustomLog definition, you can now easily visualize throttled requests:

tail -f access_log |awk '($NF > 3000000)'

Leave a Reply