Archive for the ‘howto’ Category

SOLVED: Macbook Air kernel_task slowness

Saturday, July 17th, 2010

I love my Macbook Air more than I've loved any laptop before (my first experience with a 12″ iBook in 2005 was a thing of beauty, but pales in comparison to my relationship with my Air). However, its CPU throttling to prevent heat drives me batty due to its aggressiveness — play a Flash video, for example, for a few minutes and it will start stuttering as the CPU temperature rises. Let it continue and the entire computer will slow to a crawl. If you open Activity Monitor or look at top, you'll see a process owned by root called 'kernel_task' using 150% CPU usage or so. Kill the Flash video and the kernel_task will slowly scale back and things will return to normal.

What's going on here is an interesting approach to temperature management. As temperature rises due to load on the CPU, the kernel runs some low-cost operations over and over — think 'gettimeofday()'-style functions. Since the kernel has top priority, system CPU usage spikes while userland CPU usage is forced down, lowering the actual activity that the CPU is doing and thus lowering the temperature. A decent idea, I guess, but in practice it's way too aggressive.

Luckily, doing a bit of digging in /System/Library/Extensions, I came across an extension called 'AppleIntelPenrynProfile.kext' that, looking at the Info.plist, ties into power management and performance monitoring — the IOClass is 'AppleIntelPenrynPerformanceMonitor' and the IOProviderClass 'AppleACPICPU'.

This kernel extension is loaded on boot into userland, but interestingly, if you boot into safe mode (hold shift during boot), it is not loaded — and the kernel_task CPU spikes don't occur, even under heat-generating load. You can verify this by running `kextstat', which lists all loaded kernel extensions — run it in Terminal while booted normally and you should see an extension called 'com.apple.driver.AppleIntelPenrynProfile' loaded. Boot into safe mode by holding down shift before the chime and run `kextstat' again — no com.apple.driver.AppleIntelPenrynProfile, and no kernel_task CPU spike when generating heat (play a Flash video).

So… why load that module at all?

Back in normal OSX, launch Terminal and run the following:

cd /System/Library/Extensions/AppleProfileFamily.kext/Contents/PlugIns
sudo kextunload AppleIntelPenrynProfile.kext

Verify it's unloaded:

kextstat | grep Penryn

This should return no output if the module was successfully unloaded. Now, go play a Flash video and enjoy a less-crippled Air!

* Disclaimer: I take NO responsibility if you brick your Mac, it catches on fire, never boots again, or otherwise break. I've had no problems and it's been working quite well, but your experience/hardware/whatever may be different.

** Disclaimer 2: I highly recommend running SMCFanControl and pushing your fans to max when running heat-intensive operations. I do not recommend running your Air at 80ºC or hotter for extended periods — the kernel will no longer discourage this activity by slowing things down. It's unlikely that you will fry your CPU due to extensive hot use, as the CPU's thermal shutdown is lower than its point of combustion, but this doesn't mean you should push that threshold.

*** Disclaimer 3: The Penryn profile is for the Rev 2 Macbook Airs. Rev 1 is Merom, with the extension called AppleIntelMeromProfile.kext.

ImageMagick, PDFs, and third-party fonts

Friday, July 2nd, 2010

If you use ImageMagick to convert PDFs, you'll know it's as simple as

convert file.pdf file.jpg

However, if you're using a third-party non-GhostScript-sanctioned font, this won't work terribly well and fail with a rather cryptic GhostScript error such as:

ERROR: /invalidfileaccess in --file--
Operand stack:
--dict:5/5(L)-- F2 10.0 --dict:6/6(L)-- --dict:6/6(L)-- STSongStd-Light-Acro-UniGB-UCS2-H --dict:10/12(ro)(G)-- --nostringval-- --dict:7/7(L)-- --dict:7/7(L)-- Adobe-GB1 CIDFont Adobe-GB1 Adobe-GB1 --nostringval-- (/usr/share/fonts/chinese/TrueType/uming.ttf) (r)
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop --nostringval-- --nostringval-- 2 1 6 --nostringval-- %for_pos_int_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- 1 %stopped_push --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- false 1 %stopped_push --nostringval-- %loop_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- %loop_continue --nostringval-- 12 9 %oparray_pop --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval--
Dictionary stack:
--dict:1127/1686(ro)(G)-- --dict:0/20(G)-- --dict:107/200(L)-- --dict:107/200(L)-- --dict:104/127(ro)(G)-- --dict:241/347(ro)(G)-- --dict:20/24(L)-- --dict:4/6(L)-- --dict:24/31(L)-- --dict:38/50(ro)(G)--
Current allocation mode is local
Last OS error: 2
ESP Ghostscript 815.02: Unrecoverable error, exit code 1
convert: Postscript delegate failed `/home/kale/poop.pdf'.
convert: missing an image filename `/home/kale/poop.pnm'.

In this instance, the PDF poop.pdf contains a Chinese font which Ghostscript knows about but won't let me use since I installed it after-the-fact from RPM (chinese-fonts.noarch). GhostScript by default is run from ImageMagick with the -dSAFER flag which is a mildly paranoid flag that prevents GS from using files outside of its root (/usr/share/ghostscript, usually). To work around this error, rather than dealing with GS's confusing and arcane font directory configuration, I recommend simply amending ImageMagick's delegates definition for the PDF filetype:

/usr/lib64/ImageMagick-6.2.8/config/delegates.xml:

<delegate decode="pdf" encode="ps" mode="bi" command='"gs" -q -dBATCH -dSAFER -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE="pswrite" -sOutputFile="%o" -f"%i"' />

to

<delegate decode="pdf" encode="ps" mode="bi" command='"gs" -q -dBATCH -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE="pswrite" -sOutputFile="%o" -f"%i"' />

(remove the -dSAFER flag)

Henceforth, `convert poop.pdf poop.jpg' will work with third-party fonts without issue.

Enforcing Secure Passwords in Horde

Sunday, June 13th, 2010

A coworker, Alex, discovered that Horde, in conjunction with Plesk, allows users to change their passwords arbitrarily — but doesn't enforce any sort of password policy, allowing such passwords as "test" or even "" (null). This, obviously, is a huge security risk as mail compromises can lead to fairly terrible things.

From his article:

If you (or a client you are representing) want to set horde to do the typical “strict password” enforcement, look for the file:

horde/passwd/backends.php

And read the bit about password policy. An example policy that can be set in this file that would require 1 capital, 1 lowercase, 1 special character and 1 number, with a minimum password size of 8, would look like:

‘password policy’ => array(
‘minLength’ => 8,
‘maxLength’ => 64,
‘maxSpace’ => 0,
‘minUpper’ => 1,
‘minLower’ => 1,
‘minNumeric’ => 1,
‘minSymbols’ => 1
),

Stop snmpd from spamming syslog

Friday, April 30th, 2010

Dell's OpenManage tools come with a MIB to allow for the system to query against and set up traps. However, by default, snmpd will not allow arbitrary smux peers, and your logs will be spammed with the following:

Apr 29 19:19:37 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39622
Apr 29 19:19:37 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:40 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39693
Apr 29 19:19:40 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:43 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39790
Apr 29 19:19:43 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager

Add the following to you /etc/snmpd/snmpd.conf:

smuxpeer .1.3.6.1.4.1.674.10892.1

and restart snmpd.

Run Urchin on-demand for all profiles at once

Saturday, January 2nd, 2010

There's no built-in way in Urchin to re-run the processing job for all domains (such as after fixing a problem). This can, however, be done on the command line with a while loop:

ls -alh ../usr/local/urchin/data/reports/ |awk '{print $NF}' |while read line ; do /usr/local/urchin/bin/urchin -p"$line" ; done

HOWTO: Install VNC server on RHEL5

Sunday, December 20th, 2009

Install a desktop environment and the VNC server:

yum groupinstall "GNOME Desktop Environment"
yum  install vnc-server

Change to the user who will be owning the session and run `vncserver' to set up their password and create the default files

su kale -
vncserver

Edit the xstartup file for that user to point to GNOME:

vi /home/kale/.vnc/xstartup

# Uncomment the following two lines for normal desktop:
 unset SESSION_MANAGER
 exec /etc/X11/xinit/xinitrc

Kill that VNC session

killall Xvnc

Edit vncserver's config file

vi /etc/sysconfig/vncservers

VNCSERVERS="2:kale"
VNCSERVERARGS[2]="-geometry 800x600 -nohttpd"

Start VNC

service vncserver start

And connect! Since it's specified in /etc/sysconfig/vncservers that kale's session is on display 2 (the 2:kale bit), the port for this connection is 5902. Connect to this port and enter in the password you specified earlier and voila!

WHOIS visiting your site?

Monday, November 30th, 2009

I'm fond of WHOIS data for getting an idea who's visiting a site, though most WHOIS servers return data that's full of disclaimers and irrelevant data. Rather, I much prefer Team Cymru's batch WHOIS lookup server, whois.cymru.com.

First, extract your IPs:

F=ips.out ; echo  "begin">>$F ; echo "verbose">>$F ; awk '{print $1}' tech-access_log |sort |uniq>>$F ; echo "end" >>$F

Now send them to Cymru for processing:

nc whois.cymru.com 43 < $F | sort > whois.out

Review whois.out at your leisure for detailed IP information. It's well-formatted, allowing for easily scripting against:

91      | 128.113.197.128  | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
91      | 128.113.247.58   | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
9121    | 88.232.9.77      | 88.232.0.0/17       | TR | ripencc  | 2005-10-27 | TTNET TTnet Autonomous System
9       | 128.2.161.88     | 128.2.0.0/16        | US | arin     | 1984-04-17 | CMU-ROUTER - Carnegie Mellon University
9136    | 91.186.50.28     | 91.186.32.0/19      | DE | ripencc  | 2006-11-07 | WOBCOM WOBCOM GmbH - www.wobcom.de
9143    | 212.203.31.1     | 212.203.0.0/19      | NL | ripencc  | 2000-08-08 | ZIGGO Ziggo - tv, internet, telefoon

Push email on iPhone and other smartphones… without Exchange

Friday, November 20th, 2009

Tonight, I found a clever open-source project entitled Z-Push. This small collection of PHP sits in a web directory and responds to ActiveSync queries — the protocol used for Exchange. It then checks and delivers email.

This is useful because of the limitations of some smartphones — such as the iPhone — wherein Exchange-hosted mail is delivered instantly, while standard POP3 or IMAP mail accounts suffer a long polling delay.

On the server side, configuration is fairly simple:

  1. wget http://download.berlios.de/z-push/z-push-1.3RC.tar.gz
  2. tar xzvf z-push-1.3RC.tar.gz
  3. mv z-push /var/www/html
  4. yum install php-imap
  5. chown apache:apache /var/www/html/z-push/state
  6. vi /var/www/html/z-push/config.php and configure the following:
  7. $BACKEND_PROVIDER = “BackendIMAP”;
    define(’IMAP_SERVER’, ‘localhost’);
    define(’IMAP_PORT’, 143);
    define(’IMAP_OPTIONS’, ‘/notls/norsh’);

  8. Add the following Alias to an Apache SSL VirtualHost:
  9. Alias /Microsoft-Server-ActiveSync /var/www/html/z-push/index.php

  10. Restart Apache

On your phone, simply create a new Exchange-type account that points to your server as if it was an Exchange server. Send a test mail and marvel at how fast it appears on your phone! Tested on iPhone and Motorola Droid with excellent success.

Rewriting URLs based on domain name

Thursday, October 15th, 2009

While Apache's mod_rewrite is fairly powerful, it does have some limitations (mostly to not break the HTTP spec). While you can specify any number of esoteric conditions, it's not possible to rewrite anything but the URI. If you want to redirect sillydomaintwo.com to two.sillydomain.com/blog, you'll have to use an intermediary processor like PHP.

<?php
$pattern = "/^(?:([^\.]+)\.)?sillydomain(.*)\.com/";
$rep = "$2″;
$sub = preg_replace($pattern, $rep, $_SERVER["HTTP_HOST"]);
header("Location: http://$sub.sillydomain.com/blog");
?>

Setting this code as the index.php for sillydomaintwo.com will suffice, as it will perform a 301 redirect to the target location.

In Apache, the simplest way to support this sort of configuration is to have the master sillydomain.com as its own VirtualHost with a ServerAlias of *.sillydomain.com. In another VirtualHost, set the ServerName as * with the above index.php in its DocumentRoot sillydomaintwo.com won't match the first, but will match the * (catch-all) VirtualHost, run the index.php, then redirect to the first VirtualHost:

<VirtualHost *:80>
ServerName sillydomain.com
ServerAlias *.sillydomain.com
DocumentRoot /var/www/sillydomain.com
</VirtualHost>

<VirtualHost *:80>
ServerName *
DocumentRoot /var/www/sillydomain-redirect
#in this directory is the index.php with the redirection as described above
</VirtualHost>

Enable WebDAV with Plesk

Friday, September 25th, 2009

Configuring WebDAV in Apache is simple, but it's even easier to configure and manage with Plesk!

1. Create a Protected Directory
Log into Plesk and select the domain that is to receive the DAV repository. Click on "Protected Directories" and create a new one – name it as the DAV share will be named, for they are one and the same.

2. Configure WebDAV Users
Add users who should have access to this DAV repo.

3. Edit vhost.conf and Reconfigure Plesk
On the server, edit the domain's vhost.conf and enter the following:

<Directory "/var/www/vhosts/domain.com/httpdocs/DAVdir">
DAV on
AllowOverride None
</Directory>

Regenerate Apache's configuration and you're golden:

/usr/local/psa/admin/bin/websrvmng -av

4. Test
You can easily test DAV configuration by using a DAV client such as `cadaver'.

[kale@superhappykittymeow ~]$ cadaver http://www.domain.com/DAVdir
Authentication required for on server `domain.com':
Username: kale
Password:
dav:/DAVDir/> ls
Listing collection `/DAVDir/': collection is empty.

Success! You can manage access to the DAV share through the Plesk interface.

Apache MultiViews and RewriteRules

Sunday, August 30th, 2009

Don't work together.

I think it's a bug in mod_rewrite, to be honest, though more of a "not thinking these two modules would ever be used together" kind of oversight, rather than a full bug.

Essentially, if you are using MultiViews to make for pretty URLs (say, http://www.foo.com/bar, where 'bar' doesn't exist, but instead loads the content from bar.php), and you attempt to implement RewriteRules to modify the URL, you will see erratic results.

If, for example, you have a RewriteRule as follows:

RewriteCond %{HTTP_HOST} !^www\.foo\.com
RewriteRule (.*) http://www.foo.com/$1 [R=301,L]

which, essentially, takes all non-WWW requests and makes them www.foo.com, you will find that MultiView URLs will be redirected to their real resources if the URL matches a rule. For example,

http://foo.com/bar

will become

http://www.foo.com/bar.php

after going through the MultiView filter and the RewriteRules. This is due to the way the rules work — essentially, the request will be parsed through mod_rewrite to find a match. If no match against the URL, the MultiView is processed to get the real resource which is then presented to the end user. If a match is made, however, mod_rewrite has mod_negotiation process the MultiView to find the real resource so it can properly do the rewrite — it is never changed back, however, to the pretty MultiView URL. If your goal is pretty URLs without any effort expended, relying on MultiView, you will find that RewriteRules are your nemesis.

There are a few routes available to get around this odd behavior, but my favorite (and easiest to implement) is to move the RewriteRule logic to the site code. It's much harder to implement MultiView-esque functionality than it is to re-implement RewriteRules.

To implement the above RewriteRule, redirecting non-www to www, simply add an auto_prepend_file to your .htaccess in lieu of the RewriteRule as such:

php_value auto_prepend_file "/var/www/html/prepend.php"

This file contains simply:

<?
if ( !( preg_match('/^www/', $_SERVER['SERVER_NAME']) ) ) {
    header("Location: http://www.$_SERVER[SERVER_NAME]$_SERVER[REQUEST_URI]");
    }
?>

With this code prepended to every PHP script (assuming your site is written in PHP, of course), all non-www requests will be redirected to www — *after* the MultiView is processed and not interfering with its inner workings.

Email alerts on new virus with Sophos

Friday, August 7th, 2009

Sophos's Linux antivirus product is an interesting beast, but I'll reserve opinion. We offer a web interface wherein the end-user may review alerts, though some also wish an email alert. This can be configured through savwebd, the web GUI provided with the Sophos antivirus client, or configured on the command line:

cd /opt/sophos-av/bin
./savconfig -v  # review current configuration settings
./savconfig set Email email@address.com  # recipient
./savconfig set EmailNotifier true
./savconfig set EmailDemandSummaryIfThreat true
./savconfig set EmailServer localhost
./savconfig set SendThreatEmail true
./savconfig set ThreatMessage "A virus has been detected and blocked.  Please contact your support team for more information."

Enable core dumps with apache, RHEL5

Thursday, July 23rd, 2009

From this post on Jared's tech blog:

echo "ulimit -c unlimited >/dev/null 2>&1″ >> /etc/profile
echo "DAEMON_COREFILE_LIMIT='unlimited'" >> /etc/sysconfig/init
echo 1 > /proc/sys/fs/suid_dumpable
echo "core.%p" > /proc/sys/kernel/core_pattern
echo "CoreDumpDirectory /var/apache-core-dumps" > \
/etc/httpd/conf.d/core_dumps.conf
mkdir /var/apache-core-dumps
chown apache: /var/apache-core-dumps
source /etc/profile
/etc/init.d/httpd restart

Now you can test it by sending a SIGSEGV to a random apache child process:

tail -f /var/log/httpd/error_log | grep -i seg &
ps auxwww |grep httpd (pick a random pid not owned by root)
kill -11 2014
[Mon Jul 06 21:05:39 2009] [notice] child pid 2014 exit signal
Segmentation fault (11), possible coredump in /var/apache-core-dumps
cd /var/apache-core-dumps
ls
core.2014

You can then get a backtrace using gdb:

gdb /usr/sbin/httpd core.2014
(gdb) > bt full

Brilliant -- thanks Jared, I fought Apache for an hour to enable CoreDumps before putting my fist through the monitor!

Mount NTFS drive in RHEL5

Saturday, June 27th, 2009

Grab fuse, fuse-ntfs-3g and dkms-fuse from Dag's repo:

wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/fuse-2.7.3-1.el5.rf.x86_64.rpm
wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/fuse-ntfs-3g-1.2310-1.el5.rf.x86_64.rpm
wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/dkms-fuse-2.7.2-1.nodist.rf.noarch.rpm

Install:

rpm -Uvh fuse-2.7.3-1.el5.rf.x86_64.rpm fuse-ntfs-3g-1.2310-1.el5.rf.x86_64.rpm dkms-fuse-2.7.2-1.nodist.rf.noarch.rpm

Mount:

mount.ntfs-3g /dev/sdc1 /mnt/usb/ -o force

Add a new hotswap SCSI drive

Saturday, June 20th, 2009

If you've got a SCSI card, SCA backplane, and a chassis that supports hotswappable drives, you can easily add a new drive without a reboot. Simply insert the drive and

cat /proc/scsi/scsi

Note the host, bus (channel), ID and LUN of the existing drive(s).

echo "scsi add-single-device h b i l" > /proc/scsi/scsi

where "h b i l" is the host, bus, id and lun of the new drive. If it's going on the same adapter as the rest of the drive(s) listed already, use the same host, bus and lun — the id will be incremented by one:

echo "scsi add-single-device 1 0 1 0″ > /proc/scsi/scsi

cat /proc/scsi/scsi and you should see the new device:

[root@foo scsi]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: HITACHI Model: HUS103073FL3800 Rev: SA1B
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: SEAGATE Model: ST3146707LC Rev: 0005
Type: Direct-Access ANSI SCSI revision: 03

fdisk and make a filesystem as usual!

Red5 Installation and Usage

Monday, June 15th, 2009

Red5 is an open source streaming flash media server. It's a java-based application that is surprisingly easy to install and well-documented as such, though the documentation fails when it comes to usage — such as streaming live video.

Install:

Download from the Red5 page: http://osflash.org/red5/080rc1 . Caveat: I've never really gotten this page to work. I've mirrored it on my slice:

Java 1.5: http://slice.superhappykittymeow.com/~kale/red5-0.8.0-java5.tar.gz
Java 1.6: http://slice.superhappykittymeow.com/~kale/red5-0.8.0.tar.gz

Untar:

tar xzvf red5-0.8.0-java5.tar.gz

Run:

./red5.sh

That's it for the install. Congratulations! Red5 is running and accessible at http://your-ip:5080 .

Go there and follow the instructions. Namely, click on where it states clearly to "Click here to install demos". Red5 ships with a number of demos that are inappropriately named but you should probably install anyway if you want to do cool things like stream live video.

"oflaDemo" is the key mis-named application that need be installed. Select and click "install".

Thereafter, visit the Publisher utility: http://your-ip:5080/demos/publisher.html

You may need to change the server settings to point to your server (hint: 127.0.0.1 won't work — use the public IP). Create a stream with your input source (webcam, screencast, etc) and assign a stream name, and hit publish. Hurray, now that video is being broadcast over rtmp!

You can connect to this RTMP source using a Flash viewer like Flowplayer, setting the RTMP source as rtmp://your.ip/oflaDemo, with the clip URL as the name you assigned as the name in the publisher app.

Simple!

Adding IP aliases in FreeBSD

Saturday, June 6th, 2009

Edit /etc/rc.conf:

ifconfig_fxp0="inet 1.2.3.4 netmask 255.255.255.192″
ifconfig_fxp0_alias0="1.2.3.5 netmask 255.255.255.255″
ifconfig_fxp0_alias1="1.2.3.6 netmask 255.255.255.255″
ifconfig_fxp0_alias2="1.2.3.7 netmask 255.255.255.255″

Restart networking:

/etc/rc.d/netif restart && /etc/rc.d/routing restart

Better way to scan for -- and clean up -- virus activity

Saturday, April 25th, 2009

**NOTE** The following only works with FTP daemons that log full paths in xferlog — ie, not vsftpd with its default configuration. Works like a charm on Plesk, fails terribly on non-Plesk. For non-Plesk, please scroll to the bottom of this post.

I made an earlier post about this subject, but there are too many holes in the script provided. Rather, I've found this simple awk recipe to do the trick quite well.

awk '$12 != prev {print $9; prev=$12}' xferlog | egrep "\.php|\.htm|\.shtm|\.js" | sort |uniq > ftp_modified.out

Note that the output it prints is not definitive, but it certainly gives you something to start with. Now, roll a grep:

cat ftp_modified.out |while read line; do grep -H iframe $line >> iframe.out ; done

**You will need to review this output to find the actual string and distinguish between legitimate iframes and the baddies.** The following sed will usually take care of about 80% of them:

cat iframe.out | awk -F\: '{print $1}' | while read line ; do sed -i 's/<iframe src=.*\/in\.cgi\?.*<\/iframe>//g' $line ; done

Of course, there are also JavaScript-obfuscated redirects to clean up:

cat ftp_modified.out | while read line; do grep -H eval $line >> eval.out ; done

This will catch *most* of them. Unfortunately with the JS ones, you need to develop a regex to match with sed on a per-exploit basis — and there are tons. Look over the results in eval.out and craft up a sed that is tailored enough for the JS exploits — that won't affect legit code. I usually end up with something like this:

cat eval.out | awk -F\: '{print $1}' | while read line ; do sed -i 's/function.*String.fromCharCode.*document\.write.*));//g' $line ; done

But of course, use your brain and — most importantly — *always* test the sed using the -e switch on one of the infected files first to ensure it works before running it with -i against the whole list! These cleanups are a good way to fine-tune your practical regex skills. Remember not to be too broad — or too specific!

If the server does not have Plesk or is doing chrooting, such that xferlog shows relative paths rather than absolute, we'll skip the xferlog bit and just look at our docroots for recently modified files.

grep DocumentRoot /etc/httpd/conf/httpd.conf |awk '{print $2}' > docroots.out
cat docroots.out |while read line ; do find $line -mtime -180 | egrep "\.php|\.htm|\.shtm|\.js" | sort |uniq > ftp_modified.out

To be quite honest, these aren't "ftp-modified" files, but you can drag'n'drop with the rest of my sniplets here. The entire purpose of generating these file lists is to narrow down the sheer amount of files we have to look through to make it more manageable, as opposed to grepping through everything in the server's DocumentRoots.

The above sniplets are the fastest ways I've developed to deal with this stuff — you'll spend most of your time reviewing the output and generating regexes with which to clean them up. Absolutely remember to change the FTP passwords for at least the FTP users exploited, and have the end user scan all computers that may have connected to the server via FTP for viruses and trojans.

I haven't the faintest how to deal with the new google analytics-esque variant yet… I hope that doesn't become more popular :(

Save a file in vi as the superuser

Thursday, April 23rd, 2009

If you've opened a file in vi for which you don't have write privileges, you can save the file as the superuser (if your user is in the sudoers list) by running the following vi command:

:w !sudo tee %

After doing so, vi will detect that the file has changed and ask if you want to reload it.

Force outgoing mail to come from a specific IP

Thursday, April 23rd, 2009

Add an iptables rule such as:

iptables -t nat -A POSTROUTING -p tcp -s ! 127.0.0.1 --dport 25 -j SNAT --to-source 65.61.180.228

This will make all outgoing mail leave over the secondary IP, 65.61.180.228, as opposed to the primary IP address of the server (the standard behavior of MTAs).

Scan FTP xferlog for virus-like activity

Saturday, April 11th, 2009

I often see viruses spread through usually-legit sites — see more in my related post about malware one-liners. Finding which files are infected is usually a pain, though Paul hacked me up this script to identify virus-like behavior and pinpoint infected files:

#!/usr/bin/perl

use strict;
use warnings;

my $file = $ARGV[0];

my @iArray;
my @oArray;

open LOGFILE, "<", $file or die "Can't open my $file: $!";
while (<LOGFILE>) {
        my @line = split(/ /);
        if ($line[11] eq "o") {
            push(@oArray, $line[8]);
        }
        if ($line[11] eq "i") {
            push(@iArray, $line[8]);
        }
}
foreach (@oArray) {
    my $entry = $_;
    foreach (@iArray) {
        my $entry2 = $_;
        if ($entry2 eq $entry) {
            print "$entry\n";
            last;
        }
    }
}

Usage:

Download virus-parse.pl to server and execute against the xferlog.

wget http://tech.superhappykittymeow.com/src/virus-parse.pl
chmod +x virus-parse.pl
./virus-parse.pl /var/log/xferlog

This will output a list of files suspected of containing malicious iframes or javascript, from which you can clean up.

Extend bash functionality

Saturday, April 11th, 2009
# make bash autocomplete with up arrow
bind '"\e[A":history-search-backward'
bind '"\e[B":history-search-forward'

# make tab cycle through commands instead of listing
bind '"\t":menu-complete'

The 10 Golden Rules for Troubleshooting Linux

Tuesday, March 17th, 2009

1. Man pages exist and should be used. Seriously, everything's there, from application docs to syscall docs to syntax and formatting of log files.

2. Don't reinvent the wheel. 99% of problems you're experiencing or ever will experience, somebody's already gone through it and figured it out. Google is your friend.

3. If you don't know what something's doing, or why it's not working, strace it!

4. Logs exist for a reason. Read them.

5. Applications crash, servers don't. If your server crashes, it's either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups. Always.

7. Always mount NFS mounts with the 'intr' option. Having to reboot because of a network blip is uncool.

8. Learn to use `grep', `sed' and `awk'. Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage. 100% memory usage does not mean you don't have any more available for new applications. You can run out of inodes before you run out of disk space.

10. TCP wrappers suck. If you've been hacking at an issue for over 3 hours, look to your TCP wrappers. /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.

Manually mounting a USB drive in Linux

Friday, March 13th, 2009

Most modern distros are quite smart and will recognize the variety of USB devices plugged in automagically. Today, for whatever reason, RHEL5 refused to do so, and I had to do it the hard way.

First, make sure you've got USB modules loaded:

modprobe uhci_hcd
modprobe ohci_hcd
modprobe ehci_hcd
modprobe usb-storage

With the last one, wait a few moments, then run `dmesg'. You should see some useful information:

Initializing USB Mass Storage driver…
scsi1 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: waiting for device to settle before scanning
usb 1-3.3: reset high speed USB device using ehci_hcd and address 8
Vendor: Seagate Model: FreeAgent Go Rev: 102D
Type: Direct-Access ANSI SCSI revision: 04
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
sda: sda1
sd 1:0:0:0: Attached scsi disk sda
sd 1:0:0:0: Attached scsi generic sg0 type 0
usb-storage: device scan complete

If you're lucky, you can simple do an `fstab -l' and see the drive and it's partitions at the stated point (sda). I wasn't so lucky, as this server didn't have device nodes for sda. These, however, are easily created:

/dev/MAKEDEV sda

Now you should be able to mount it:

mount /dev/sda1 /mnt/usb

Intercepting "command not found" in bash

Friday, March 6th, 2009

On Debian, bash is patched with an interesting new function: command_not_found_handle. This intercepts exit code 127 ("command not found") and allows you to do neat things. Debian uses it to pass it through the apt database, letting you know if a command you tried to invoke is not available, but can be found in the apt repos, and how to install it. Pretty spiffy.

This, of course, can be modified. Where I work, we use numbers to identify servers. I have a script that grabs login credentials from our internal systems and auto-logs me into servers based on their number. For example, I'd run `connect 12345′ to connect to server 12345.

By adding the following to my .bashrc:

function command_not_found_handle {
    /home/kale/bin/command-not-found $1
}

And creating the following script, with regex in place to only care about numbers, placed in /home/kale/bin/command-not-found:

#!/bin/bash

MYBOOL=`echo $1 | awk '$1 ~ /^[0-9]+-*[0-9]*$/ {print $0}' | wc -l | awk '{print $1}'`

if [ "$MYBOOL" == "1″ ]; then
        /home/kale/bin/connect $1
else
        exit 127
fi

(where `connect' is the path to my connect script, previously written)

This now allows me to do this awesome deal:

kale@bastion:~$ 12345
Connecting to server 12345
root@12345:~#
 

If you didn't catch it, I don't need to specify a command — just the argument. As there's no application in my $PATH named `12345′, it falls through to the command_not_found_handle function, which then launches my connect script.

Who needs commands? I just saved hundreds of wasted seconds per night on typing "connect"!

Removing 1-liner malware code in webpages

Wednesday, March 4th, 2009

More and more client workstations are being infected with keyloggers and trojans. In addition to stealing your WoW username and password (oh noes, my purpz!), they also have been stealing FTP logins.

This has manifested itself in the linux server world by seemingly legit users logging in over FTP, downloading a file, then uploading it a few seconds later with 100-ish bytes appended. A look at xferlog reveals this behavior, usually against a regex of pages (index.*, default.*, etc), and the connecting IP will often be foreign. A look at the secure log will reveal that the password was not brute-forced; rather, it was known.

The real solution is to change all passwords and force the end user to reformat their computer, since they're infected and do not realize it. Alas, this is not quite practical (though if someone could invent a remote formatter, I'll give you $10 for it). Rather, advise the end user of the situation and suggest reformatting — or, at the very least, using a collection of anti-spyware, anti-virus, and anti-everything software on their workstation. Change the affected user's password.

To clean up the leftover malicious code that was appended, find out the exact string (usually a `tail index.php' will reveal it) — it's often a <JavaScript> line or an <iframe>. Copy the string completely and we'll just sed it out:

sed -i "s#<script src='http:\/\/b\.adserv\.cn\/E\/J\.JS'>##g" *

sed's not very good at recursing, but luckily, grep is. Make a list of files that match:

grep -R "b.adserv.cn" * |awk -F\: '{print $1}' > filelist

And then feed it to sed:

cat filelist |while read line ; do sed -i"s#<script src='http:\/\/b\.adserv\.cn\/E\/J\.JS'>##g" $line ; done

It should also be noted that this user logged in as the FTP user with no failed password attempts — they knew the password. This situation most often occurs when a client workstation that has access to this FTP account is compromised with a virus, spyware, trojan horse or keylogger that transmits the login credentials to a third party attacker. I strongly recommend running anti-virus and anti-spyware software on all client workstations that have access to this account.

Adding a new drive on an HP RAID controller

Wednesday, March 4th, 2009

Launch `hpacucli', the RAID interface and run the following:

controller slot=0 pd show all

All drives will be shown — it's obvious which ones are not configured as a logical disk.

controller slot=0 create type=ld drives=2:2

We've only added one drive, and the controller is smart enough to know that making a logical disk out of one drive will be a RAID-0, thus no RAID level is specified. If using multiple drives, list them with commas (drives=2:2,2:3,2:4) and add a "raid" command:

controller slot=0 create type=ld drives=2:2,2:3,2:4 raid=?

Using the ? will tell you what RAID levels are available for the drives selected. Specify the RAID level with "raid=5″.

controller slot=0 ld show all

Will show your new logical disk composed of the previously unused drives. You may need to run `partprobe' on the server to enable it to see the new logical disk, but thereafter you are free to partition away and do what must be done.

Mount an HFS+ drive in linux

Wednesday, March 4th, 2009

Modern kernels have a module for HFS+, but Apple used BSD-style partitioning, which linux's fdisk can't handle.

Try pdisk, an almost-10-year-old program that still builds well on modern linuxes. Download, untar, and make.

[root@web1 pdisk]# ./pdisk /dev/sdb
Edit /dev/sdb --
Command (? for help): p

Partition map (with 512 byte blocks) on '/dev/sdb'
#: type name length base ( size )
1: Apple_partition_map Apple 63 @ 1
2: Apple_Free 262144 @ 64 (128.0M)
3: Apple_HFS Apple_HFS_Untitled_1 2930035632 @ 262208 ( 1.4T)
4: Apple_Free 16 @ 2930297840

Device block size=512, Number of Blocks=2930297856 (1.4T)
DeviceType=0×0, DeviceId=0×0

[root@web1 pdisk]# mount -t hfsplus /dev/sdb3 /mnt/usbdrive

[root@web1 pdisk]# ls /mnt/usbdrive/
Directory1 Directory2

Install OpenManage on RHEL servers

Wednesday, March 4th, 2009
wget -q -O -- http://linux.dell.com/repo/hardware/bootstrap.cgi | bash
up2date -i srvadmin-all
service dataeng start

Add a system user as a Webmin user

Wednesday, March 4th, 2009

Edit the user file:

vi /etc/webmin/miniserv.users

Enter the system user's name, followed by :x :

kale:x

Edit /etc/webmin/webmin.acl to give access to this new user

Restart Webmin