Browse Tag: rpm

mod_rpaf and Amazon ELB

Amazon’s ELB service is nice — magical load balancers that just work, sitting in front of your servers, that you can update and modify on a whim. Of course, because it’s a load balancer (a distributed load balancer infrastructure, to be more precise), Apache and other applications sitting behind it see all the incoming traffic as coming from the load balancer — ie, $REMOTE_ADDR is 10.251.74.17 instead of the end client’s public IP.

This is normal behavior when sitting behind a load balancer, and it’s also normal behavior for the load balancer to encapsulate the original client IP in an X-Forwarded-For header. Using Apache, we can, for example, modify LogFormat definitions to account for this, logging %{X-Forwarded-For}i to log the end user’s IP.

Where this falls short, however, is when you want to *do* things with the originating IP beyond logging. The real-world scenario I ran into was using mod_qos to do rate-limiting based on URIs within Apache — mod_qos tests against the remote IP, not the X-Forwarded-For, so using the module as is, I’m unable to apply any QoS rules against anything beyond the load balancer… which of course defeats the purpose.

Luckily, I’m not the only person to have ever run into this issue. The Apache module mod_rpaf is explicitly designed to address this type of situation by translating the X-Forwarded-For header into the remote address as Apache expects, so that other modules can properly run against the originating IP — not the load balancer.

ELB makes implementation of mod_rpaf much more difficult that it should be, however. ELB is architected as a large network of load balancers, such that incoming outside requests bounce around a bit within the ELB infrastructure before being passed to your instance. Each “bounce” adds an additional IP to X-Forwarded-For, essentially chaining proxies. Additionally, there are hundreds of internal IPs within ELB that would need to be accounted for to use mod_rpaf as is, as you must specify the proxy IPs to strip.

So I patched up mod_rpaf to work with ELB. I’ve been running it for a day or so in dev and it appears to be working as expected, passing the original client value to mod_qos (and mod_qos testing and working against that), but of course if you run into issues, please let me know (because your issues will probably show up in my environment as well).

Here is the patch:

--- mod_rpaf-2.0.c	2008-01-01 03:05:40.000000000 +0000
+++ mod_rpaf-2.0.c~	2011-08-25 20:04:39.000000000 +0000
@@ -136,13 +136,25 @@
 }
 
 static int is_in_array(const char *remote_ip, apr_array_header_t *proxy_ips) {
-    int i;
+   /* int i;
     char **list = (char**)proxy_ips->elts;
     for (i = 0; i < proxy_ips->nelts; i++) {
         if (strcmp(remote_ip, list[i]) == 0)
             return 1;
     }
     return 0;
+    */
+    return 1;
+}
+
+static char* last_not_in_array(apr_array_header_t *forwarded_for,
+			       apr_array_header_t *proxy_ips) {
+    int i;
+    for (i = (forwarded_for->nelts)-1; i > 0; i--) {
+	if (!is_in_array(((char **)forwarded_for->elts)[i], proxy_ips))
+	    break;
+    }
+    return ((char **)forwarded_for->elts)[i];
 }
 
 static apr_status_t rpaf_cleanup(void *data) {
@@ -161,7 +173,7 @@
     if (!cfg->enable)
         return DECLINED;
 
-    if (is_in_array(r->connection->remote_ip, cfg->proxy_ips) == 1) {
+    /* if (is_in_array(r->connection->remote_ip, cfg->proxy_ips) == 1) { */
         /* check if cfg->headername is set and if it is use
            that instead of X-Forwarded-For by default */
         if (cfg->headername && (fwdvalue = apr_table_get(r->headers_in, cfg->headername))) {
@@ -183,7 +195,8 @@
             rcr->old_ip = apr_pstrdup(r->connection->pool, r->connection->remote_ip);
             rcr->r = r;
             apr_pool_cleanup_register(r->pool, (void *)rcr, rpaf_cleanup, apr_pool_cleanup_null);
-            r->connection->remote_ip = apr_pstrdup(r->connection->pool, ((char **)arr->elts)[((arr->nelts)-1)]);
+            /* r->connection->remote_ip = apr_pstrdup(r->connection->pool, ((char **)arr->elts)[((arr->nelts)-1)]); */
+            r->connection->remote_ip = apr_pstrdup(r->connection->pool, last_not_in_array(arr, cfg->proxy_ips));
             r->connection->remote_addr->sa.sin.sin_addr.s_addr = apr_inet_addr(r->connection->remote_ip);
             if (cfg->sethostname) {
                 const char *hostvalue;
@@ -201,7 +214,7 @@
             }
 
         }
-    }
+    /* } */
     return DECLINED;
 }

Or, if you’d prefer ez-mode, I rolled some RPMs of mod_rpaf that include this patch:

mod_rpaf-0.6-0.7.i386.rpm
mod_rpaf-0.6-0.7.x86_64.rpm

And, for completeness, mod_rpaf.conf:

LoadModule rpaf_module        modules/mod_rpaf-2.0.so

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 10.
RPAFheader X-Forwarded-For

Segmentation faults with up2date/rpm

Had a nasty one tonight that I had to turn over to Paul. up2date was segfaulting on a RHEL3 server, along with some rpm queries, such as `rpm -qa’. I was able to narrow it down to a specific package, though that did not help at all. The rpm database had some severe corruption.

I removed the __db.00* files from /var/lib/rpm and rebuilt the database with rpm –rebuilddb; however, this did not resolve the issue. I attempted to recreate the database, by doing the following:

[code lang=”bash”]rpm -qa > rpm-list ; rpm –initdb ; cat rpm-list | while read line ; do rpm –nodeps –noscripts –notriggers –excludepath / $line ; done[/code]

However, the `rpm -qa’ hung, as noted earlier. Tough spot.

I had to move on, but Paul dove in. After a few hours of fitful hacking, he declared himself the winner — he had solved it. What did he do?

He removed all of the files in /var/lib/rpm except for Packages, then ran `rpm –rebuilddb’. The __db.00* files are just lock files, and while removing them can help, rpm transactions and queries still read all the other files and databases, thus rereading the corruption. Removing all the other databases (except the base Packages database) and then running a –rebuilddb operation actually rebuilds the databases… and the corruption cleared.

I also found this neat snippet to see if anything has a lock on the rpm databases:

[code lang=”bash”]cd /var/lib/rpm
/usr/lib/rpm/rpmdb_stat -Cl
[/code]

Silly RPM tricks

Find all non-Red Hat-supplied packages:
[code lang=”bash”]rpm -qa –qf ‘%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n’ | grep -v ‘Red Hat, Inc\.’ | sort[/code]

Handy for diagnosing issues where things seem a little “off”.