tech.superhappykittymeow.com

You can easily log how much memory each request for a PHP page takes by modifying the LogFormat:

Note the latter definition, which includes %{mod_php_memory_usage}n — this will print out the amount of memory, in bytes, required to execute the script as requested. Big help in finding memory leaks. To use, just change the log definition to the newly-created "combined-php" format:

Do note that this may (probably will) break Apache log parsers that are expecting the standard combined format. If using for troubleshooting, I recommend logging to an alternate location so as to not screw up log statistics.

Credit to Brad Ison for this find

There's no built-in way in Urchin to re-run the processing job for all domains (such as after fixing a problem). This can, however, be done on the command line with a while loop:

I found a great article on Secure Computing: Sec-C that includes some excellent, simple Apache configurations and RewriteRules to blog various annoyances and compromises. A wonderful example is this bit, designed to stop POST requests that have no referrer set. There's no reason for anyone to be trying to post arbitrary data to a script to not have a referrer, as that would indicate a direct hit — which is bad juju.

Lots of other fascinating security and forensics insights on the Sec-C blog as well!

Install a desktop environment and the VNC server:

yum groupinstall "GNOME Desktop Environment"
yum  install vnc-server

Change to the user who will be owning the session and run `vncserver' to set up their password and create the default files

su kale -
vncserver

Edit the xstartup file for that user to point to GNOME:

vi /home/kale/.vnc/xstartup

# Uncomment the following two lines for normal desktop:
 unset SESSION_MANAGER
 exec /etc/X11/xinit/xinitrc

Kill that VNC session

killall Xvnc

Edit vncserver's config file

vi /etc/sysconfig/vncservers

VNCSERVERS="2:kale"
VNCSERVERARGS[2]="-geometry 800x600 -nohttpd"

Start VNC

service vncserver start

And connect! Since it's specified in /etc/sysconfig/vncservers that kale's session is on display 2 (the 2:kale bit), the port for this connection is 5902. Connect to this port and enter in the password you specified earlier and voila!

I'm fond of WHOIS data for getting an idea who's visiting a site, though most WHOIS servers return data that's full of disclaimers and irrelevant data. Rather, I much prefer Team Cymru's batch WHOIS lookup server, whois.cymru.com.

First, extract your IPs:

Now send them to Cymru for processing:

Review whois.out at your leisure for detailed IP information. It's well-formatted, allowing for easily scripting against:

91      | 128.113.197.128  | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
91      | 128.113.247.58   | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
9121    | 88.232.9.77      | 88.232.0.0/17       | TR | ripencc  | 2005-10-27 | TTNET TTnet Autonomous System
9       | 128.2.161.88     | 128.2.0.0/16        | US | arin     | 1984-04-17 | CMU-ROUTER - Carnegie Mellon University
9136    | 91.186.50.28     | 91.186.32.0/19      | DE | ripencc  | 2006-11-07 | WOBCOM WOBCOM GmbH - www.wobcom.de
9143    | 212.203.31.1     | 212.203.0.0/19      | NL | ripencc  | 2000-08-08 | ZIGGO Ziggo - tv, internet, telefoon

Creates a much easier-to-read view of the output of "show table status":

Name                    Rows   Avg_row_length   Update_time
 wp_users                1      140              2009-08-08 04:13:07
 wp_links                9      106              2009-10-16 12:57:32
 wp_comments             14     464              2009-11-28 16:09:43
 wp_usermeta             15     166              2009-11-29 06:41:19
 wp_term_taxonomy        53     40               2009-11-20 14:06:21
 wp_postmeta             141    46               2009-11-29 06:44:05
 wp_options              172    4624             2009-11-29 06:40:59
 wp_term_relationships   357    21               2009-11-21 02:35:42 

Tonight, I found a clever open-source project entitled Z-Push. This small collection of PHP sits in a web directory and responds to ActiveSync queries — the protocol used for Exchange. It then checks and delivers email.

This is useful because of the limitations of some smartphones — such as the iPhone — wherein Exchange-hosted mail is delivered instantly, while standard POP3 or IMAP mail accounts suffer a long polling delay.

On the server side, configuration is fairly simple:

1. wget http://download.berlios.de/z-push/z-push-1.3RC.tar.gz
2. tar xzvf z-push-1.3RC.tar.gz
3. mv z-push /var/www/html
4. yum install php-imap
5. chown apache:apache /var/www/html/z-push/state
6. vi /var/www/html/z-push/config.php and configure the following:

$BACKEND_PROVIDER = “BackendIMAP”;
define(’IMAP_SERVER’, ‘localhost’);
define(’IMAP_PORT’, 143);
define(’IMAP_OPTIONS’, ‘/notls/norsh’);

7. Add the following Alias to an Apache SSL VirtualHost:

Alias /Microsoft-Server-ActiveSync /var/www/html/z-push/index.php

8. Restart Apache

On your phone, simply create a new Exchange-type account that points to your server as if it was an Exchange server. Send a test mail and marvel at how fast it appears on your phone! Tested on iPhone and Motorola Droid with excellent success.

I hacked up a Greasemonkey script tonight to address my very real problem with Youtube: Flash.

I use a Macbook Air and dearly love it, though I'm rather cognizant of the fact that it's powered by a dying hamster. Playing a video in Youtube will inevitably stutter or cause my lap to catch fire, both undesirable. There are also advertisements and Flash sucks up memory and CPU like crazy.

I attempted to address a different issue before arriving at this solution, namely, Youtube's new throttling algorithm by. I realized, however, that I'm simply not smart enough to figure that puzzle out, though I learned quite a bit about HTML5 and its <video> tag along the way.

I'd like to point out first that I'm not a JavaScript coder — I've actually never used it until tonight, so please don't judge too harshly. This Greasemonkey script is a mishmash of about 3 other Greasemonkey scripts and some regexes I used in a PHP script.

Install Youtube Flash-No-Mo

Do note that while the script works like a champ on Safari and Chrome (and likely any other webkit-based browser), Firefox 3.5 does not have a built-in handler for mp4 content. You may need to install an additional plugin (mplayer maybe?) to get things to work right on Firefox.

This ugly one-liner comes to us courtesy Chuck. Plesk calculates bandwidth statistics by literally reading the raw log files and performing math based on the byte totals noted in the log entries. This beast will run against the Plesk maillogs and give you a pretty summary of mail bandwidth:

While Apache's mod_rewrite is fairly powerful, it does have some limitations (mostly to not break the HTTP spec). While you can specify any number of esoteric conditions, it's not possible to rewrite anything but the URI. If you want to redirect sillydomaintwo.com to two.sillydomain.com/blog, you'll have to use an intermediary processor like PHP.

Setting this code as the index.php for sillydomaintwo.com will suffice, as it will perform a 301 redirect to the target location.

In Apache, the simplest way to support this sort of configuration is to have the master sillydomain.com as its own VirtualHost with a ServerAlias of *.sillydomain.com. In another VirtualHost, set the ServerName as * with the above index.php in its DocumentRoot sillydomaintwo.com won't match the first, but will match the * (catch-all) VirtualHost, run the index.php, then redirect to the first VirtualHost:

Configuring WebDAV in Apache is simple, but it's even easier to configure and manage with Plesk!

1. Create a Protected Directory
Log into Plesk and select the domain that is to receive the DAV repository. Click on "Protected Directories" and create a new one – name it as the DAV share will be named, for they are one and the same.

2. Configure WebDAV Users
Add users who should have access to this DAV repo.

3. Edit vhost.conf and Reconfigure Plesk
On the server, edit the domain's vhost.conf and enter the following:

Regenerate Apache's configuration and you're golden:

4. Test
You can easily test DAV configuration by using a DAV client such as `cadaver'.

Success! You can manage access to the DAV share through the Plesk interface.

Symptom: seemingly random PHP scripts are causing Apache to segfault.

Looking deeper: all the PHP scripts that are causing segfaults make database queries (specifically, MySQL).

Look even closer: the following line is in your Apache configuration:

Solution: comment that line out of your Apache configuration and restart Apache.

Why: If the PHP code is run through Apache, you've essentially got one process making the SQL queries (if your PHP code makes it so). However, while your code made the connection and is expecting responses and whatnot, Apache, with mod_auth_mysql loaded, is ready and willing to make and take database connections. When a connection that returns a response is made from your PHP code, Apache will attempt to accept the response and handle it itself, instead of passing it to PHP. Since Apache is not expecting the data it's getting, it has no error handling code for this situation and simply segfaults.

Disable mod_auth_mysql by commenting it out and everything will work without issue.

Don't work together.

I think it's a bug in mod_rewrite, to be honest, though more of a "not thinking these two modules would ever be used together" kind of oversight, rather than a full bug.

Essentially, if you are using MultiViews to make for pretty URLs (say, http://www.foo.com/bar, where 'bar' doesn't exist, but instead loads the content from bar.php), and you attempt to implement RewriteRules to modify the URL, you will see erratic results.

If, for example, you have a RewriteRule as follows:

which, essentially, takes all non-WWW requests and makes them www.foo.com, you will find that MultiView URLs will be redirected to their real resources if the URL matches a rule. For example,

http://foo.com/bar

will become

http://www.foo.com/bar.php

after going through the MultiView filter and the RewriteRules. This is due to the way the rules work — essentially, the request will be parsed through mod_rewrite to find a match. If no match against the URL, the MultiView is processed to get the real resource which is then presented to the end user. If a match is made, however, mod_rewrite has mod_negotiation process the MultiView to find the real resource so it can properly do the rewrite — it is never changed back, however, to the pretty MultiView URL. If your goal is pretty URLs without any effort expended, relying on MultiView, you will find that RewriteRules are your nemesis.

There are a few routes available to get around this odd behavior, but my favorite (and easiest to implement) is to move the RewriteRule logic to the site code. It's much harder to implement MultiView-esque functionality than it is to re-implement RewriteRules.

To implement the above RewriteRule, redirecting non-www to www, simply add an auto_prepend_file to your .htaccess in lieu of the RewriteRule as such:

This file contains simply:

With this code prepended to every PHP script (assuming your site is written in PHP, of course), all non-www requests will be redirected to www — *after* the MultiView is processed and not interfering with its inner workings.

via Paul (lovepig.org):

This one-liner is quite effective when tossed into a file and run as a cronjob once per minute. Any IP with more than 100 concurrent connections — which, quite honestly, is far more than any one IP should ever have on a standard webserver — will be blocked via iptables. This script as a cronjob is extremely effective dealing with small-to-midsize DDoSes (too much traffic for Apache/whatever service to handle, but not saturating the pipe).

Sophos's Linux antivirus product is an interesting beast, but I'll reserve opinion. We offer a web interface wherein the end-user may review alerts, though some also wish an email alert. This can be configured through savwebd, the web GUI provided with the Sophos antivirus client, or configured on the command line:

From this post on Jared's tech blog:

Now you can test it by sending a SIGSEGV to a random apache child process:

You can then get a backtrace using gdb:

Brilliant -- thanks Jared, I fought Apache for an hour to enable CoreDumps before putting my fist through the monitor!

Grab fuse, fuse-ntfs-3g and dkms-fuse from Dag's repo:

Install:

Mount:

If you've got a SCSI card, SCA backplane, and a chassis that supports hotswappable drives, you can easily add a new drive without a reboot. Simply insert the drive and

Note the host, bus (channel), ID and LUN of the existing drive(s).

where "h b i l" is the host, bus, id and lun of the new drive. If it's going on the same adapter as the rest of the drive(s) listed already, use the same host, bus and lun — the id will be incremented by one:

cat /proc/scsi/scsi and you should see the new device:

fdisk and make a filesystem as usual!

…for a list of domains, without digging through the database!

'domains', of course, is a text file with a list of domains hosted on the server. Can be populated in whatever way you need. Easily plugged into other Plesk utilities (such as changing Plesk FTP passwords).

To combine two (or more) text files as individual columns in the same file, such as:

file1:

file2:

into:

rather than using an ugly combination of sed and awk, you can use the `paste' command:

Red5 is an open source streaming flash media server. It's a java-based application that is surprisingly easy to install and well-documented as such, though the documentation fails when it comes to usage — such as streaming live video.

Install:

Download from the Red5 page: http://osflash.org/red5/080rc1 . Caveat: I've never really gotten this page to work. I've mirrored it on my slice:

Java 1.5: http://slice.superhappykittymeow.com/~kale/red5-0.8.0-java5.tar.gz
Java 1.6: http://slice.superhappykittymeow.com/~kale/red5-0.8.0.tar.gz

Untar:

Run:

That's it for the install. Congratulations! Red5 is running and accessible at http://your-ip:5080 .

Go there and follow the instructions. Namely, click on where it states clearly to "Click here to install demos". Red5 ships with a number of demos that are inappropriately named but you should probably install anyway if you want to do cool things like stream live video.

"oflaDemo" is the key mis-named application that need be installed. Select and click "install".

Thereafter, visit the Publisher utility: http://your-ip:5080/demos/publisher.html

You may need to change the server settings to point to your server (hint: 127.0.0.1 won't work — use the public IP). Create a stream with your input source (webcam, screencast, etc) and assign a stream name, and hit publish. Hurray, now that video is being broadcast over rtmp!

You can connect to this RTMP source using a Flash viewer like Flowplayer, setting the RTMP source as rtmp://your.ip/oflaDemo, with the clip URL as the name you assigned as the name in the publisher app.

Simple!

Edit /etc/rc.conf:

Restart networking:

The PHP eval() function is an odd beast. I've encountered this a few times before, wherein a site will display just fine, but Google won't index it, and if you review the headers for the page (using curl or telnet), you'll see that instead of the expected 200 OK response, you'll get a "HTTP/1.0 500 Internal Server Error" — bad juju for all of your hard SEO work.

Taking a look at the site's error_log may reveal a PHP error — often a syntax error or some such. However, the line of code to which it refers is just an eval() statement for an included file.

If the eval'd code has an error, the code running the eval statement will work just fine; however, because of the error in the eval'd code, the response code header will return a 500. This, interestingly enough, will only occur if display_errors is set to off.

The ideal solution is to fix the code that is being eval'd.

The hackish way, which I prefer because I'm lazy, is to do something terrible such as:

The response code will now be 200 and Google will be happy; however, the problem isn't *really* solved…

**NOTE** The following only works with FTP daemons that log full paths in xferlog — ie, not vsftpd with its default configuration. Works like a charm on Plesk, fails terribly on non-Plesk. For non-Plesk, please scroll to the bottom of this post.

I made an earlier post about this subject, but there are too many holes in the script provided. Rather, I've found this simple awk recipe to do the trick quite well.

Note that the output it prints is not definitive, but it certainly gives you something to start with. Now, roll a grep:

**You will need to review this output to find the actual string and distinguish between legitimate iframes and the baddies.** The following sed will usually take care of about 80% of them:

Of course, there are also JavaScript-obfuscated redirects to clean up:

This will catch *most* of them. Unfortunately with the JS ones, you need to develop a regex to match with sed on a per-exploit basis — and there are tons. Look over the results in eval.out and craft up a sed that is tailored enough for the JS exploits — that won't affect legit code. I usually end up with something like this:

But of course, use your brain and — most importantly — *always* test the sed using the -e switch on one of the infected files first to ensure it works before running it with -i against the whole list! These cleanups are a good way to fine-tune your practical regex skills. Remember not to be too broad — or too specific!

If the server does not have Plesk or is doing chrooting, such that xferlog shows relative paths rather than absolute, we'll skip the xferlog bit and just look at our docroots for recently modified files.

To be quite honest, these aren't "ftp-modified" files, but you can drag'n'drop with the rest of my sniplets here. The entire purpose of generating these file lists is to narrow down the sheer amount of files we have to look through to make it more manageable, as opposed to grepping through everything in the server's DocumentRoots.

The above sniplets are the fastest ways I've developed to deal with this stuff — you'll spend most of your time reviewing the output and generating regexes with which to clean them up. Absolutely remember to change the FTP passwords for at least the FTP users exploited, and have the end user scan all computers that may have connected to the server via FTP for viruses and trojans.

I haven't the faintest how to deal with the new google analytics-esque variant yet… I hope that doesn't become more popular

This will print the whole line except for the first field (because you've set it to null).

If you've opened a file in vi for which you don't have write privileges, you can save the file as the superuser (if your user is in the sudoers list) by running the following vi command:

After doing so, vi will detect that the file has changed and ask if you want to reload it.

Add an iptables rule such as:

This will make all outgoing mail leave over the secondary IP, 65.61.180.228, as opposed to the primary IP address of the server (the standard behavior of MTAs).

You can use the `sort' utility to sort not only on the first field, but also an arbitrary column. Take this output from ps:

This can be sorted by the SZ column, column 9, as such:

The -k switch tells sort to sort based on a key, which we specify as 9n (column 9, numeric). Much easier to review the output.

I often see viruses spread through usually-legit sites — see more in my related post about malware one-liners. Finding which files are infected is usually a pain, though Paul hacked me up this script to identify virus-like behavior and pinpoint infected files:

Usage:

Download virus-parse.pl to server and execute against the xferlog.

This will output a list of files suspected of containing malicious iframes or javascript, from which you can clean up.

Runs in the foreground a simple, single-threaded web server on port 8000 as the current user. Logging is to stdout/stderr, and a ctrl-c will stop the server. Great for temporarily sharing a directory.

http://rugmonster.org/2009/03/building-mod_fastcgi-on-rhel5/

Great for command-line logging into sites to pull content for whatever reason.

Of course, you'll have to look at the source for the target location's login page to see what variables it wants. I use it to grab a single Cacti-generated graph that is normally password protected, but I want to include a single graph on another site, so I cron'd a script to run a line similar to the above to log in and save it locally.

from command-line-fu

Spot DDoS's and the like quickly:

Thanks Geoff!

If you add an entry to crontab that is an interval, such as */3 (every 3 minutes), you can verify that it runs at the specified interval with a bit of awk:

This essentially checks to see that the minute field of the timestamp is divisible by three — the interval. It'll also run at 00 after the hour, not divisible by three, but expected.

Cron can "run late" at times due to high load situations, so if there are any irregularities in your intervals, you may wish to investigate deeper, looking for expensive processes that are chewing up precious cron time.

1. Man pages exist and should be used. Seriously, everything's there, from application docs to syscall docs to syntax and formatting of log files.

2. Don't reinvent the wheel. 99% of problems you're experiencing or ever will experience, somebody's already gone through it and figured it out. Google is your friend.

3. If you don't know what something's doing, or why it's not working, strace it!

4. Logs exist for a reason. Read them.

5. Applications crash, servers don't. If your server crashes, it's either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups. Always.

7. Always mount NFS mounts with the 'intr' option. Having to reboot because of a network blip is uncool.

8. Learn to use `grep', `sed' and `awk'. Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage. 100% memory usage does not mean you don't have any more available for new applications. You can run out of inodes before you run out of disk space.

10. TCP wrappers suck. If you've been hacking at an issue for over 3 hours, look to your TCP wrappers. /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.

Don't do it.

While there's no technical file limit on a per-directory basis with ext3 (there is on a filesystem level — see inodes), there is *significant* performance degradation as you add more and more files to a single directory. I think the most I've seen without any user-noticeable sluggishness was about 300,000. Note that this is well beyond the point where you can't `ls' anymore and you have to resort to `find' and `xargs'. This should be your first warning sign.

Approaching 5 million files in one directory, things start to get weird. Creating new files in that directory generates significant load, though resource usage is low. However, statting a specific file (as opposed to a lookup with a glob) is decently fast. As long as you know what you want, it'll work acceptably fast.

The more files you add, the slower lookup-based operations (new file creation, for example) will go — we're talking seconds and tens of seconds here, not a few milliseconds more. As long as you give it an exact filename, though, it will be of an acceptable speed.

The filesystem option dir_indexes will help, though not hugely once you start getting into millions of files. Compared to no dir_indexing, it's faster, but it doesn't make it magically work. Converting to ext2 is a terrible idea and should not be considered — journals are good things and well worth the extremely slight (comparatively) performance hit endured.

The real solution, however, is to not put that many files into a single directory. Subdirectories are always a good idea (though keep in mind the subdirectory limit — 32k subdirs per dir!). Heck, most code can almost trivially be modified to pull content from a hash from the filename, such as /var/www/images/y/o/yourmom.jpg and /var/www/images/y/i/yipee.jpg. When designing an application, one should be mindful of the limitations of the underlying OS (and in this case, the filesystem being used).

Moving a Plesk server behind a firewall is always a pain, since the IPs are associated with domains within the Plesk database. I used to hack the database every time I had to update IPs, but doing this for 50 IPs is… not so good.

Luckily, I stumbled up on this Parallels knowledge base article, which introduces reconfigurator.pl — it reads a mapping of IPs and updates system interfaces as well as all the internal Plesky goodness.

Most modern distros are quite smart and will recognize the variety of USB devices plugged in automagically. Today, for whatever reason, RHEL5 refused to do so, and I had to do it the hard way.

First, make sure you've got USB modules loaded:

With the last one, wait a few moments, then run `dmesg'. You should see some useful information:

If you're lucky, you can simple do an `fstab -l' and see the drive and it's partitions at the stated point (sda). I wasn't so lucky, as this server didn't have device nodes for sda. These, however, are easily created:

Now you should be able to mount it:

Ever wish you knew what Apache was working on at any given moment, but kicking yourself because you forgot to enable a server-status directive? This snippet will help you diagnose timeouts and long-running scripts (for bad coders like myself):

Had a nasty one tonight that I had to turn over to Paul. up2date was segfaulting on a RHEL3 server, along with some rpm queries, such as `rpm -qa'. I was able to narrow it down to a specific package, though that did not help at all. The rpm database had some severe corruption.

I removed the __db.00* files from /var/lib/rpm and rebuilt the database with rpm --rebuilddb; however, this did not resolve the issue. I attempted to recreate the database, by doing the following:

However, the `rpm -qa' hung, as noted earlier. Tough spot.

I had to move on, but Paul dove in. After a few hours of fitful hacking, he declared himself the winner — he had solved it. What did he do?

He removed all of the files in /var/lib/rpm except for Packages, then ran `rpm --rebuilddb'. The __db.00* files are just lock files, and while removing them can help, rpm transactions and queries still read all the other files and databases, thus rereading the corruption. Removing all the other databases (except the base Packages database) and then running a --rebuilddb operation actually rebuilds the databases… and the corruption cleared.

I also found this neat snippet to see if anything has a lock on the rpm databases:

On Debian, bash is patched with an interesting new function: command_not_found_handle. This intercepts exit code 127 ("command not found") and allows you to do neat things. Debian uses it to pass it through the apt database, letting you know if a command you tried to invoke is not available, but can be found in the apt repos, and how to install it. Pretty spiffy.

This, of course, can be modified. Where I work, we use numbers to identify servers. I have a script that grabs login credentials from our internal systems and auto-logs me into servers based on their number. For example, I'd run `connect 12345′ to connect to server 12345.

By adding the following to my .bashrc:

And creating the following script, with regex in place to only care about numbers, placed in /home/kale/bin/command-not-found:

(where `connect' is the path to my connect script, previously written)

This now allows me to do this awesome deal:

If you didn't catch it, I don't need to specify a command — just the argument. As there's no application in my $PATH named `12345′, it falls through to the command_not_found_handle function, which then launches my connect script.

Who needs commands? I just saved hundreds of wasted seconds per night on typing "connect"!

ftp://ftp.cisco.com/pub/mibs/oid/

Go forth and monitor!

Surprisingly hard to find on HP's site if you just need a copy of hpacucli.

HP Insight Diagnostics Online Edition for Linux

Set bash's internal field seperator to an enter instead of a space, so grep (or whatever) doesn't freak out:

More and more client workstations are being infected with keyloggers and trojans. In addition to stealing your WoW username and password (oh noes, my purpz!), they also have been stealing FTP logins.

This has manifested itself in the linux server world by seemingly legit users logging in over FTP, downloading a file, then uploading it a few seconds later with 100-ish bytes appended. A look at xferlog reveals this behavior, usually against a regex of pages (index.*, default.*, etc), and the connecting IP will often be foreign. A look at the secure log will reveal that the password was not brute-forced; rather, it was known.

The real solution is to change all passwords and force the end user to reformat their computer, since they're infected and do not realize it. Alas, this is not quite practical (though if someone could invent a remote formatter, I'll give you $10 for it). Rather, advise the end user of the situation and suggest reformatting — or, at the very least, using a collection of anti-spyware, anti-virus, and anti-everything software on their workstation. Change the affected user's password.

To clean up the leftover malicious code that was appended, find out the exact string (usually a `tail index.php' will reveal it) — it's often a <JavaScript> line or an <iframe>. Copy the string completely and we'll just sed it out:

sed's not very good at recursing, but luckily, grep is. Make a list of files that match:

And then feed it to sed:

It should also be noted that this user logged in as the FTP user with no failed password attempts — they knew the password. This situation most often occurs when a client workstation that has access to this FTP account is compromised with a virus, spyware, trojan horse or keylogger that transmits the login credentials to a third party attacker. I strongly recommend running anti-virus and anti-spyware software on all client workstations that have access to this account.