tech.superhappykittymeow.com

I love my Macbook Air more than I've loved any laptop before (my first experience with a 12″ iBook in 2005 was a thing of beauty, but pales in comparison to my relationship with my Air). However, its CPU throttling to prevent heat drives me batty due to its aggressiveness — play a Flash video, for example, for a few minutes and it will start stuttering as the CPU temperature rises. Let it continue and the entire computer will slow to a crawl. If you open Activity Monitor or look at top, you'll see a process owned by root called 'kernel_task' using 150% CPU usage or so. Kill the Flash video and the kernel_task will slowly scale back and things will return to normal.

What's going on here is an interesting approach to temperature management. As temperature rises due to load on the CPU, the kernel runs some low-cost operations over and over — think 'gettimeofday()'-style functions. Since the kernel has top priority, system CPU usage spikes while userland CPU usage is forced down, lowering the actual activity that the CPU is doing and thus lowering the temperature. A decent idea, I guess, but in practice it's way too aggressive.

Luckily, doing a bit of digging in /System/Library/Extensions, I came across an extension called 'AppleIntelPenrynProfile.kext' that, looking at the Info.plist, ties into power management and performance monitoring — the IOClass is 'AppleIntelPenrynPerformanceMonitor' and the IOProviderClass 'AppleACPICPU'.

This kernel extension is loaded on boot into userland, but interestingly, if you boot into safe mode (hold shift during boot), it is not loaded — and the kernel_task CPU spikes don't occur, even under heat-generating load. You can verify this by running `kextstat', which lists all loaded kernel extensions — run it in Terminal while booted normally and you should see an extension called 'com.apple.driver.AppleIntelPenrynProfile' loaded. Boot into safe mode by holding down shift before the chime and run `kextstat' again — no com.apple.driver.AppleIntelPenrynProfile, and no kernel_task CPU spike when generating heat (play a Flash video).

So… why load that module at all?

Back in normal OSX, launch Terminal and run the following:

cd /System/Library/Extensions/AppleProfileFamily.kext/Contents/PlugIns
sudo kextunload AppleIntelPenrynProfile.kext

Verify it's unloaded:

kextstat | grep Penryn

This should return no output if the module was successfully unloaded. Now, go play a Flash video and enjoy a less-crippled Air!

* Disclaimer: I take NO responsibility if you brick your Mac, it catches on fire, never boots again, or otherwise break. I've had no problems and it's been working quite well, but your experience/hardware/whatever may be different.

** Disclaimer 2: I highly recommend running SMCFanControl and pushing your fans to max when running heat-intensive operations. I do not recommend running your Air at 80ºC or hotter for extended periods — the kernel will no longer discourage this activity by slowing things down. It's unlikely that you will fry your CPU due to extensive hot use, as the CPU's thermal shutdown is lower than its point of combustion, but this doesn't mean you should push that threshold.

*** Disclaimer 3: The Penryn profile is for the Rev 2 Macbook Airs. Rev 1 is Merom, with the extension called AppleIntelMeromProfile.kext.

If you use ImageMagick to convert PDFs, you'll know it's as simple as

However, if you're using a third-party non-GhostScript-sanctioned font, this won't work terribly well and fail with a rather cryptic GhostScript error such as:

ERROR: /invalidfileaccess in --file--
Operand stack:
--dict:5/5(L)-- F2 10.0 --dict:6/6(L)-- --dict:6/6(L)-- STSongStd-Light-Acro-UniGB-UCS2-H --dict:10/12(ro)(G)-- --nostringval-- --dict:7/7(L)-- --dict:7/7(L)-- Adobe-GB1 CIDFont Adobe-GB1 Adobe-GB1 --nostringval-- (/usr/share/fonts/chinese/TrueType/uming.ttf) (r)
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop --nostringval-- --nostringval-- 2 1 6 --nostringval-- %for_pos_int_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- 1 %stopped_push --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- false 1 %stopped_push --nostringval-- %loop_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval-- %array_continue --nostringval-- --nostringval-- --nostringval-- --nostringval-- %loop_continue --nostringval-- 12 9 %oparray_pop --nostringval-- --nostringval-- --nostringval-- --nostringval-- --nostringval--
Dictionary stack:
--dict:1127/1686(ro)(G)-- --dict:0/20(G)-- --dict:107/200(L)-- --dict:107/200(L)-- --dict:104/127(ro)(G)-- --dict:241/347(ro)(G)-- --dict:20/24(L)-- --dict:4/6(L)-- --dict:24/31(L)-- --dict:38/50(ro)(G)--
Current allocation mode is local
Last OS error: 2
ESP Ghostscript 815.02: Unrecoverable error, exit code 1
convert: Postscript delegate failed `/home/kale/poop.pdf'.
convert: missing an image filename `/home/kale/poop.pnm'.

In this instance, the PDF poop.pdf contains a Chinese font which Ghostscript knows about but won't let me use since I installed it after-the-fact from RPM (chinese-fonts.noarch). GhostScript by default is run from ImageMagick with the -dSAFER flag which is a mildly paranoid flag that prevents GS from using files outside of its root (/usr/share/ghostscript, usually). To work around this error, rather than dealing with GS's confusing and arcane font directory configuration, I recommend simply amending ImageMagick's delegates definition for the PDF filetype:

/usr/lib64/ImageMagick-6.2.8/config/delegates.xml:

<delegate decode="pdf" encode="ps" mode="bi" command='"gs" -q -dBATCH -dSAFER -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE="pswrite" -sOutputFile="%o" -f"%i"' />

to

<delegate decode="pdf" encode="ps" mode="bi" command='"gs" -q -dBATCH -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE="pswrite" -sOutputFile="%o" -f"%i"' />

(remove the -dSAFER flag)

Henceforth, `convert poop.pdf poop.jpg' will work with third-party fonts without issue.

I'm bad at counting, so when I'm using awk to print specific fields, I end up with greasy fingerprints on my screen as I manually count out each field. Thanks to my colleague James, here's a script that counts for you!

Works with STDIN as is, assuming default field separator (space):

[kale@superhappykittymeow log]# tail -n 1 xferlog |awk 'NR == 1 { for (i=1;i<=NF;i++) {printf i " "} print ""} {print}' | column -t
1    2    3   4         5     6  7          8    9                          10  11  12  13  14    15   16  17  18
Sat  Jun  19  13:19:25  2010  1  127.0.0.1  220  /var/www/poop/wp-rss2.php  b   _   i   r   root  ftp  0   *   c

Or, if you're lazy like myself, encapsulate it in an alias:

[kale@superhappykittymeow log]# tail -n 1 xferlog | count
1    2    3   4         5     6  7          8    9                          10  11  12  13  14    15   16  17  18
Sat  Jun  19  13:19:25  2010  1  127.0.0.1  220  /var/www/poop/wp-rss2.php  b   _   i   r   root  ftp  0   *   c

A coworker, Alex, discovered that Horde, in conjunction with Plesk, allows users to change their passwords arbitrarily — but doesn't enforce any sort of password policy, allowing such passwords as "test" or even "" (null). This, obviously, is a huge security risk as mail compromises can lead to fairly terrible things.

From his article:

If you (or a client you are representing) want to set horde to do the typical “strict password” enforcement, look for the file:

horde/passwd/backends.php

And read the bit about password policy. An example policy that can be set in this file that would require 1 capital, 1 lowercase, 1 special character and 1 number, with a minimum password size of 8, would look like:

‘password policy’ => array(
‘minLength’ => 8,
‘maxLength’ => 64,
‘maxSpace’ => 0,
‘minUpper’ => 1,
‘minLower’ => 1,
‘minNumeric’ => 1,
‘minSymbols’ => 1
),

Can't fork but need to see what's going on? Hint: a box that can't fork can often `exec'.

Here are a pair of slick bash functions that can be lifesavers in dire situations:

`ls':

`cat':

I received a lovely stack trace trying to run a python script that makes an external call using curl. The meat of it:

File "/Library/Python/2.6/site-packages/pyechonest/util.py", line 52, in call
f = urllib.urlopen(url)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 87, in urlopen
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 203, in open
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 310, in open_http
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 1555, in proxy_bypass
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 1425, in proxy_bypass_macosx_sysconf

This is a bug due to how python on OSX parses the values in the proxy bypass box in the system's Proxy configuration. If you have an explicit IP, it will bomb out. Remove the explicitly specified IP or change it to a name value to get around this bug. Netblocks (such as "10/8″) won't cause it to bomb — just a definite IP like 1.2.3.4.

Installing EV from CPAN, I received this message:

Checking if your kit is complete…
Looks good
Warning: prerequisite common::sense 0 not found.

:/

Dell's OpenManage tools come with a MIB to allow for the system to query against and set up traps. However, by default, snmpd will not allow arbitrary smux peers, and your logs will be spammed with the following:

Apr 29 19:19:37 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39622
Apr 29 19:19:37 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:40 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39693
Apr 29 19:19:40 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:43 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39790
Apr 29 19:19:43 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager

Add the following to you /etc/snmpd/snmpd.conf:

smuxpeer .1.3.6.1.4.1.674.10892.1

and restart snmpd.

The iPad's built-in movie player identifies itself as the useragent "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)" and explicitly specifies a range to download, allowing for chunked streaming. Here are the headers from both sides of a recent transaction on my server:

GET /music/lt-fundies.mp4 HTTP/1.1
Host: growl.superhappykittymeow.com
Range: bytes=110691320-147521535
Connection: close
User-Agent: AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)
Accept: */*
Accept-Encoding: identity

HTTP/1.1 206 Partial Content
Date: Sat, 24 Apr 2010 09:28:22 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Wed, 17 Sep 2008 18:35:48 GMT
ETag: \"4ad19f-998329f-4571bbcac8900\"
Accept-Ranges: bytes
Content-Length: 36830216
Content-Range: bytes 110691320-147521535/160969375
Connection: close
Content-Type: text/plain; charset=UTF-8

Fun fact -- a dropped packet results in awesome skipping on the client! …it also spams your access_logs with each request:

66.216.90.167 -- - [24/Apr/2010:09:33:59 +0000] "GET /music/lt-fundies.mp4 HTTP/1.1″ 206 7012352 "-" "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)"
66.216.90.167 -- - [24/Apr/2010:09:34:00 +0000] "GET /music/lt-fundies.mp4 HTTP/1.1″ 206 26657760 "-" "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)"
66.216.90.167 -- - [24/Apr/2010:09:34:02 +0000] "GET /music/lt-fundies.mp4 HTTP/1.1″ 206 2558 "-" "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)"
66.216.90.167 -- - [24/Apr/2010:09:34:02 +0000] "GET /music/lt-fundies.mp4 HTTP/1.1″ 206 6912055 "-" "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)"
66.216.90.167 -- - [24/Apr/2010:09:34:03 +0000] "GET /music/lt-fundies.mp4 HTTP/1.1″ 206 26500344 "-" "AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)"

If your SNMP config works, try this:

You can easily log how much memory each request for a PHP page takes by modifying the LogFormat:

Note the latter definition, which includes %{mod_php_memory_usage}n — this will print out the amount of memory, in bytes, required to execute the script as requested. Big help in finding memory leaks. To use, just change the log definition to the newly-created "combined-php" format:

Do note that this may (probably will) break Apache log parsers that are expecting the standard combined format. If using for troubleshooting, I recommend logging to an alternate location so as to not screw up log statistics.

Credit to Brad Ison for this find

There's no built-in way in Urchin to re-run the processing job for all domains (such as after fixing a problem). This can, however, be done on the command line with a while loop:

I found a great article on Secure Computing: Sec-C that includes some excellent, simple Apache configurations and RewriteRules to blog various annoyances and compromises. A wonderful example is this bit, designed to stop POST requests that have no referrer set. There's no reason for anyone to be trying to post arbitrary data to a script to not have a referrer, as that would indicate a direct hit — which is bad juju.

Lots of other fascinating security and forensics insights on the Sec-C blog as well!

Install a desktop environment and the VNC server:

yum groupinstall "GNOME Desktop Environment"
yum  install vnc-server

Change to the user who will be owning the session and run `vncserver' to set up their password and create the default files

su kale -
vncserver

Edit the xstartup file for that user to point to GNOME:

vi /home/kale/.vnc/xstartup

# Uncomment the following two lines for normal desktop:
 unset SESSION_MANAGER
 exec /etc/X11/xinit/xinitrc

Kill that VNC session

killall Xvnc

Edit vncserver's config file

vi /etc/sysconfig/vncservers

VNCSERVERS="2:kale"
VNCSERVERARGS[2]="-geometry 800x600 -nohttpd"

Start VNC

service vncserver start

And connect! Since it's specified in /etc/sysconfig/vncservers that kale's session is on display 2 (the 2:kale bit), the port for this connection is 5902. Connect to this port and enter in the password you specified earlier and voila!

I'm fond of WHOIS data for getting an idea who's visiting a site, though most WHOIS servers return data that's full of disclaimers and irrelevant data. Rather, I much prefer Team Cymru's batch WHOIS lookup server, whois.cymru.com.

First, extract your IPs:

Now send them to Cymru for processing:

Review whois.out at your leisure for detailed IP information. It's well-formatted, allowing for easily scripting against:

91      | 128.113.197.128  | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
91      | 128.113.247.58   | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
9121    | 88.232.9.77      | 88.232.0.0/17       | TR | ripencc  | 2005-10-27 | TTNET TTnet Autonomous System
9       | 128.2.161.88     | 128.2.0.0/16        | US | arin     | 1984-04-17 | CMU-ROUTER - Carnegie Mellon University
9136    | 91.186.50.28     | 91.186.32.0/19      | DE | ripencc  | 2006-11-07 | WOBCOM WOBCOM GmbH - www.wobcom.de
9143    | 212.203.31.1     | 212.203.0.0/19      | NL | ripencc  | 2000-08-08 | ZIGGO Ziggo - tv, internet, telefoon

Creates a much easier-to-read view of the output of "show table status":

Name                    Rows   Avg_row_length   Update_time
 wp_users                1      140              2009-08-08 04:13:07
 wp_links                9      106              2009-10-16 12:57:32
 wp_comments             14     464              2009-11-28 16:09:43
 wp_usermeta             15     166              2009-11-29 06:41:19
 wp_term_taxonomy        53     40               2009-11-20 14:06:21
 wp_postmeta             141    46               2009-11-29 06:44:05
 wp_options              172    4624             2009-11-29 06:40:59
 wp_term_relationships   357    21               2009-11-21 02:35:42 

Tonight, I found a clever open-source project entitled Z-Push. This small collection of PHP sits in a web directory and responds to ActiveSync queries — the protocol used for Exchange. It then checks and delivers email.

This is useful because of the limitations of some smartphones — such as the iPhone — wherein Exchange-hosted mail is delivered instantly, while standard POP3 or IMAP mail accounts suffer a long polling delay.

On the server side, configuration is fairly simple:

1. wget http://download.berlios.de/z-push/z-push-1.3RC.tar.gz
2. tar xzvf z-push-1.3RC.tar.gz
3. mv z-push /var/www/html
4. yum install php-imap
5. chown apache:apache /var/www/html/z-push/state
6. vi /var/www/html/z-push/config.php and configure the following:

$BACKEND_PROVIDER = “BackendIMAP”;
define(’IMAP_SERVER’, ‘localhost’);
define(’IMAP_PORT’, 143);
define(’IMAP_OPTIONS’, ‘/notls/norsh’);

7. Add the following Alias to an Apache SSL VirtualHost:

Alias /Microsoft-Server-ActiveSync /var/www/html/z-push/index.php

8. Restart Apache

On your phone, simply create a new Exchange-type account that points to your server as if it was an Exchange server. Send a test mail and marvel at how fast it appears on your phone! Tested on iPhone and Motorola Droid with excellent success.

I hacked up a Greasemonkey script tonight to address my very real problem with Youtube: Flash.

I use a Macbook Air and dearly love it, though I'm rather cognizant of the fact that it's powered by a dying hamster. Playing a video in Youtube will inevitably stutter or cause my lap to catch fire, both undesirable. There are also advertisements and Flash sucks up memory and CPU like crazy.

I attempted to address a different issue before arriving at this solution, namely, Youtube's new throttling algorithm by. I realized, however, that I'm simply not smart enough to figure that puzzle out, though I learned quite a bit about HTML5 and its <video> tag along the way.

I'd like to point out first that I'm not a JavaScript coder — I've actually never used it until tonight, so please don't judge too harshly. This Greasemonkey script is a mishmash of about 3 other Greasemonkey scripts and some regexes I used in a PHP script.

Install Youtube Flash-No-Mo

Do note that while the script works like a champ on Safari and Chrome (and likely any other webkit-based browser), Firefox 3.5 does not have a built-in handler for mp4 content. You may need to install an additional plugin (mplayer maybe?) to get things to work right on Firefox.

This ugly one-liner comes to us courtesy Chuck. Plesk calculates bandwidth statistics by literally reading the raw log files and performing math based on the byte totals noted in the log entries. This beast will run against the Plesk maillogs and give you a pretty summary of mail bandwidth:

While Apache's mod_rewrite is fairly powerful, it does have some limitations (mostly to not break the HTTP spec). While you can specify any number of esoteric conditions, it's not possible to rewrite anything but the URI. If you want to redirect sillydomaintwo.com to two.sillydomain.com/blog, you'll have to use an intermediary processor like PHP.

Setting this code as the index.php for sillydomaintwo.com will suffice, as it will perform a 301 redirect to the target location.

In Apache, the simplest way to support this sort of configuration is to have the master sillydomain.com as its own VirtualHost with a ServerAlias of *.sillydomain.com. In another VirtualHost, set the ServerName as * with the above index.php in its DocumentRoot sillydomaintwo.com won't match the first, but will match the * (catch-all) VirtualHost, run the index.php, then redirect to the first VirtualHost:

Configuring WebDAV in Apache is simple, but it's even easier to configure and manage with Plesk!

1. Create a Protected Directory
Log into Plesk and select the domain that is to receive the DAV repository. Click on "Protected Directories" and create a new one – name it as the DAV share will be named, for they are one and the same.

2. Configure WebDAV Users
Add users who should have access to this DAV repo.

3. Edit vhost.conf and Reconfigure Plesk
On the server, edit the domain's vhost.conf and enter the following:

Regenerate Apache's configuration and you're golden:

4. Test
You can easily test DAV configuration by using a DAV client such as `cadaver'.

Success! You can manage access to the DAV share through the Plesk interface.

Symptom: seemingly random PHP scripts are causing Apache to segfault.

Looking deeper: all the PHP scripts that are causing segfaults make database queries (specifically, MySQL).

Look even closer: the following line is in your Apache configuration:

Solution: comment that line out of your Apache configuration and restart Apache.

Why: If the PHP code is run through Apache, you've essentially got one process making the SQL queries (if your PHP code makes it so). However, while your code made the connection and is expecting responses and whatnot, Apache, with mod_auth_mysql loaded, is ready and willing to make and take database connections. When a connection that returns a response is made from your PHP code, Apache will attempt to accept the response and handle it itself, instead of passing it to PHP. Since Apache is not expecting the data it's getting, it has no error handling code for this situation and simply segfaults.

Disable mod_auth_mysql by commenting it out and everything will work without issue.

Don't work together.

I think it's a bug in mod_rewrite, to be honest, though more of a "not thinking these two modules would ever be used together" kind of oversight, rather than a full bug.

Essentially, if you are using MultiViews to make for pretty URLs (say, http://www.foo.com/bar, where 'bar' doesn't exist, but instead loads the content from bar.php), and you attempt to implement RewriteRules to modify the URL, you will see erratic results.

If, for example, you have a RewriteRule as follows:

which, essentially, takes all non-WWW requests and makes them www.foo.com, you will find that MultiView URLs will be redirected to their real resources if the URL matches a rule. For example,

http://foo.com/bar

will become

http://www.foo.com/bar.php

after going through the MultiView filter and the RewriteRules. This is due to the way the rules work — essentially, the request will be parsed through mod_rewrite to find a match. If no match against the URL, the MultiView is processed to get the real resource which is then presented to the end user. If a match is made, however, mod_rewrite has mod_negotiation process the MultiView to find the real resource so it can properly do the rewrite — it is never changed back, however, to the pretty MultiView URL. If your goal is pretty URLs without any effort expended, relying on MultiView, you will find that RewriteRules are your nemesis.

There are a few routes available to get around this odd behavior, but my favorite (and easiest to implement) is to move the RewriteRule logic to the site code. It's much harder to implement MultiView-esque functionality than it is to re-implement RewriteRules.

To implement the above RewriteRule, redirecting non-www to www, simply add an auto_prepend_file to your .htaccess in lieu of the RewriteRule as such:

This file contains simply:

With this code prepended to every PHP script (assuming your site is written in PHP, of course), all non-www requests will be redirected to www — *after* the MultiView is processed and not interfering with its inner workings.

via Paul (lovepig.org):

This one-liner is quite effective when tossed into a file and run as a cronjob once per minute. Any IP with more than 100 concurrent connections — which, quite honestly, is far more than any one IP should ever have on a standard webserver — will be blocked via iptables. This script as a cronjob is extremely effective dealing with small-to-midsize DDoSes (too much traffic for Apache/whatever service to handle, but not saturating the pipe).

Sophos's Linux antivirus product is an interesting beast, but I'll reserve opinion. We offer a web interface wherein the end-user may review alerts, though some also wish an email alert. This can be configured through savwebd, the web GUI provided with the Sophos antivirus client, or configured on the command line:

From this post on Jared's tech blog:

Now you can test it by sending a SIGSEGV to a random apache child process:

You can then get a backtrace using gdb:

Brilliant -- thanks Jared, I fought Apache for an hour to enable CoreDumps before putting my fist through the monitor!

Grab fuse, fuse-ntfs-3g and dkms-fuse from Dag's repo:

Install:

Mount:

If you've got a SCSI card, SCA backplane, and a chassis that supports hotswappable drives, you can easily add a new drive without a reboot. Simply insert the drive and

Note the host, bus (channel), ID and LUN of the existing drive(s).

where "h b i l" is the host, bus, id and lun of the new drive. If it's going on the same adapter as the rest of the drive(s) listed already, use the same host, bus and lun — the id will be incremented by one:

cat /proc/scsi/scsi and you should see the new device:

fdisk and make a filesystem as usual!

…for a list of domains, without digging through the database!

'domains', of course, is a text file with a list of domains hosted on the server. Can be populated in whatever way you need. Easily plugged into other Plesk utilities (such as changing Plesk FTP passwords).

To combine two (or more) text files as individual columns in the same file, such as:

file1:

file2:

into:

rather than using an ugly combination of sed and awk, you can use the `paste' command:

Red5 is an open source streaming flash media server. It's a java-based application that is surprisingly easy to install and well-documented as such, though the documentation fails when it comes to usage — such as streaming live video.

Install:

Download from the Red5 page: http://osflash.org/red5/080rc1 . Caveat: I've never really gotten this page to work. I've mirrored it on my slice:

Java 1.5: http://slice.superhappykittymeow.com/~kale/red5-0.8.0-java5.tar.gz
Java 1.6: http://slice.superhappykittymeow.com/~kale/red5-0.8.0.tar.gz

Untar:

Run:

That's it for the install. Congratulations! Red5 is running and accessible at http://your-ip:5080 .

Go there and follow the instructions. Namely, click on where it states clearly to "Click here to install demos". Red5 ships with a number of demos that are inappropriately named but you should probably install anyway if you want to do cool things like stream live video.

"oflaDemo" is the key mis-named application that need be installed. Select and click "install".

Thereafter, visit the Publisher utility: http://your-ip:5080/demos/publisher.html

You may need to change the server settings to point to your server (hint: 127.0.0.1 won't work — use the public IP). Create a stream with your input source (webcam, screencast, etc) and assign a stream name, and hit publish. Hurray, now that video is being broadcast over rtmp!

You can connect to this RTMP source using a Flash viewer like Flowplayer, setting the RTMP source as rtmp://your.ip/oflaDemo, with the clip URL as the name you assigned as the name in the publisher app.

Simple!

Edit /etc/rc.conf:

Restart networking:

The PHP eval() function is an odd beast. I've encountered this a few times before, wherein a site will display just fine, but Google won't index it, and if you review the headers for the page (using curl or telnet), you'll see that instead of the expected 200 OK response, you'll get a "HTTP/1.0 500 Internal Server Error" — bad juju for all of your hard SEO work.

Taking a look at the site's error_log may reveal a PHP error — often a syntax error or some such. However, the line of code to which it refers is just an eval() statement for an included file.

If the eval'd code has an error, the code running the eval statement will work just fine; however, because of the error in the eval'd code, the response code header will return a 500. This, interestingly enough, will only occur if display_errors is set to off.

The ideal solution is to fix the code that is being eval'd.

The hackish way, which I prefer because I'm lazy, is to do something terrible such as:

The response code will now be 200 and Google will be happy; however, the problem isn't *really* solved…

**NOTE** The following only works with FTP daemons that log full paths in xferlog — ie, not vsftpd with its default configuration. Works like a charm on Plesk, fails terribly on non-Plesk. For non-Plesk, please scroll to the bottom of this post.

I made an earlier post about this subject, but there are too many holes in the script provided. Rather, I've found this simple awk recipe to do the trick quite well.

Note that the output it prints is not definitive, but it certainly gives you something to start with. Now, roll a grep:

**You will need to review this output to find the actual string and distinguish between legitimate iframes and the baddies.** The following sed will usually take care of about 80% of them:

Of course, there are also JavaScript-obfuscated redirects to clean up:

This will catch *most* of them. Unfortunately with the JS ones, you need to develop a regex to match with sed on a per-exploit basis — and there are tons. Look over the results in eval.out and craft up a sed that is tailored enough for the JS exploits — that won't affect legit code. I usually end up with something like this:

But of course, use your brain and — most importantly — *always* test the sed using the -e switch on one of the infected files first to ensure it works before running it with -i against the whole list! These cleanups are a good way to fine-tune your practical regex skills. Remember not to be too broad — or too specific!

If the server does not have Plesk or is doing chrooting, such that xferlog shows relative paths rather than absolute, we'll skip the xferlog bit and just look at our docroots for recently modified files.

To be quite honest, these aren't "ftp-modified" files, but you can drag'n'drop with the rest of my sniplets here. The entire purpose of generating these file lists is to narrow down the sheer amount of files we have to look through to make it more manageable, as opposed to grepping through everything in the server's DocumentRoots.

The above sniplets are the fastest ways I've developed to deal with this stuff — you'll spend most of your time reviewing the output and generating regexes with which to clean them up. Absolutely remember to change the FTP passwords for at least the FTP users exploited, and have the end user scan all computers that may have connected to the server via FTP for viruses and trojans.

I haven't the faintest how to deal with the new google analytics-esque variant yet… I hope that doesn't become more popular

This will print the whole line except for the first field (because you've set it to null).

If you've opened a file in vi for which you don't have write privileges, you can save the file as the superuser (if your user is in the sudoers list) by running the following vi command:

After doing so, vi will detect that the file has changed and ask if you want to reload it.

Add an iptables rule such as:

This will make all outgoing mail leave over the secondary IP, 65.61.180.228, as opposed to the primary IP address of the server (the standard behavior of MTAs).

You can use the `sort' utility to sort not only on the first field, but also an arbitrary column. Take this output from ps:

This can be sorted by the SZ column, column 9, as such:

The -k switch tells sort to sort based on a key, which we specify as 9n (column 9, numeric). Much easier to review the output.

I often see viruses spread through usually-legit sites — see more in my related post about malware one-liners. Finding which files are infected is usually a pain, though Paul hacked me up this script to identify virus-like behavior and pinpoint infected files:

Usage:

Download virus-parse.pl to server and execute against the xferlog.

This will output a list of files suspected of containing malicious iframes or javascript, from which you can clean up.

Runs in the foreground a simple, single-threaded web server on port 8000 as the current user. Logging is to stdout/stderr, and a ctrl-c will stop the server. Great for temporarily sharing a directory.

http://rugmonster.org/2009/03/building-mod_fastcgi-on-rhel5/

Great for command-line logging into sites to pull content for whatever reason.

Of course, you'll have to look at the source for the target location's login page to see what variables it wants. I use it to grab a single Cacti-generated graph that is normally password protected, but I want to include a single graph on another site, so I cron'd a script to run a line similar to the above to log in and save it locally.

from command-line-fu

Spot DDoS's and the like quickly:

Thanks Geoff!

If you add an entry to crontab that is an interval, such as */3 (every 3 minutes), you can verify that it runs at the specified interval with a bit of awk:

This essentially checks to see that the minute field of the timestamp is divisible by three — the interval. It'll also run at 00 after the hour, not divisible by three, but expected.

Cron can "run late" at times due to high load situations, so if there are any irregularities in your intervals, you may wish to investigate deeper, looking for expensive processes that are chewing up precious cron time.

1. Man pages exist and should be used. Seriously, everything's there, from application docs to syscall docs to syntax and formatting of log files.

2. Don't reinvent the wheel. 99% of problems you're experiencing or ever will experience, somebody's already gone through it and figured it out. Google is your friend.

3. If you don't know what something's doing, or why it's not working, strace it!

4. Logs exist for a reason. Read them.

5. Applications crash, servers don't. If your server crashes, it's either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups. Always.

7. Always mount NFS mounts with the 'intr' option. Having to reboot because of a network blip is uncool.

8. Learn to use `grep', `sed' and `awk'. Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage. 100% memory usage does not mean you don't have any more available for new applications. You can run out of inodes before you run out of disk space.

10. TCP wrappers suck. If you've been hacking at an issue for over 3 hours, look to your TCP wrappers. /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.

Don't do it.

While there's no technical file limit on a per-directory basis with ext3 (there is on a filesystem level — see inodes), there is *significant* performance degradation as you add more and more files to a single directory. I think the most I've seen without any user-noticeable sluggishness was about 300,000. Note that this is well beyond the point where you can't `ls' anymore and you have to resort to `find' and `xargs'. This should be your first warning sign.

Approaching 5 million files in one directory, things start to get weird. Creating new files in that directory generates significant load, though resource usage is low. However, statting a specific file (as opposed to a lookup with a glob) is decently fast. As long as you know what you want, it'll work acceptably fast.

The more files you add, the slower lookup-based operations (new file creation, for example) will go — we're talking seconds and tens of seconds here, not a few milliseconds more. As long as you give it an exact filename, though, it will be of an acceptable speed.

The filesystem option dir_indexes will help, though not hugely once you start getting into millions of files. Compared to no dir_indexing, it's faster, but it doesn't make it magically work. Converting to ext2 is a terrible idea and should not be considered — journals are good things and well worth the extremely slight (comparatively) performance hit endured.

The real solution, however, is to not put that many files into a single directory. Subdirectories are always a good idea (though keep in mind the subdirectory limit — 32k subdirs per dir!). Heck, most code can almost trivially be modified to pull content from a hash from the filename, such as /var/www/images/y/o/yourmom.jpg and /var/www/images/y/i/yipee.jpg. When designing an application, one should be mindful of the limitations of the underlying OS (and in this case, the filesystem being used).