SOLVED: Macbook Air kernel_task slowness

I love my Macbook Air more than I’ve loved any laptop before (my first experience with a 12″ iBook in 2005 was a thing of beauty, but pales in comparison to my relationship with my Air). However, its CPU throttling to prevent heat drives me batty due to its aggressiveness — play a Flash video, for example, for a few minutes and it will start stuttering as the CPU temperature rises. Let it continue and the entire computer will slow to a crawl. If you open Activity Monitor or look at top, you’ll see a process owned by root called ‘kernel_task’ using 150% CPU usage or so. Kill the Flash video and the kernel_task will slowly scale back and things will return to normal.

What’s going on here is an interesting approach to temperature management. As temperature rises due to load on the CPU, the kernel runs some low-cost operations over and over — think ‘gettimeofday()’-style functions. Since the kernel has top priority, system CPU usage spikes while userland CPU usage is forced down, lowering the actual activity that the CPU is doing and thus lowering the temperature. A decent idea, I guess, but in practice it’s way too aggressive.

Luckily, doing a bit of digging in /System/Library/Extensions, I came across an extension called ‘AppleIntelPenrynProfile.kext’ that, looking at the Info.plist, ties into power management and performance monitoring — the IOClass is ‘AppleIntelPenrynPerformanceMonitor’ and the IOProviderClass ‘AppleACPICPU’.

This kernel extension is loaded on boot into userland, but interestingly, if you boot into safe mode (hold shift during boot), it is not loaded — and the kernel_task CPU spikes don’t occur, even under heat-generating load. You can verify this by running `kextstat’, which lists all loaded kernel extensions — run it in Terminal while booted normally and you should see an extension called ‘com.apple.driver.AppleIntelPenrynProfile’ loaded. Boot into safe mode by holding down shift before the chime and run `kextstat’ again — no com.apple.driver.AppleIntelPenrynProfile, and no kernel_task CPU spike when generating heat (play a Flash video).

So… why load that module at all?

Back in normal OSX, launch Terminal and run the following:

cd /System/Library/Extensions/AppleProfileFamily.kext/Contents/PlugIns
sudo kextunload AppleIntelPenrynProfile.kext

Verify it’s unloaded:

kextstat | grep Penryn

This should return no output if the module was successfully unloaded. Now, go play a Flash video and enjoy a less-crippled Air!

* Disclaimer: I take NO responsibility if you brick your Mac, it catches on fire, never boots again, or otherwise break. I’ve had no problems and it’s been working quite well, but your experience/hardware/whatever may be different.

** Disclaimer 2: I highly recommend running SMCFanControl and pushing your fans to max when running heat-intensive operations. I do not recommend running your Air at 80ºC or hotter for extended periods — the kernel will no longer discourage this activity by slowing things down. It’s unlikely that you will fry your CPU due to extensive hot use, as the CPU’s thermal shutdown is lower than its point of combustion, but this doesn’t mean you should push that threshold.

*** Disclaimer 3: The Penryn profile is for the Rev 2 Macbook Airs. Rev 1 is Merom, with the extension called AppleIntelMeromProfile.kext.

ImageMagick, PDFs, and third-party fonts

If you use ImageMagick to convert PDFs, you’ll know it’s as simple as

convert file.pdf file.jpg

However, if you’re using a third-party non-GhostScript-sanctioned font, this won’t work terribly well and fail with a rather cryptic GhostScript error such as:

ERROR: /invalidfileaccess in –file–
Operand stack:
–dict:5/5(L)– F2 10.0 –dict:6/6(L)– –dict:6/6(L)– STSongStd-Light-Acro-UniGB-UCS2-H –dict:10/12(ro)(G)– –nostringval– –dict:7/7(L)– –dict:7/7(L)– Adobe-GB1 CIDFont Adobe-GB1 Adobe-GB1 –nostringval– (/usr/share/fonts/chinese/TrueType/uming.ttf) (r)
Execution stack:
%interp_exit .runexec2 –nostringval– –nostringval– –nostringval– 2 %stopped_push –nostringval– –nostringval– –nostringval– false 1 %stopped_push 1 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop –nostringval– –nostringval– 2 1 6 –nostringval– %for_pos_int_continue –nostringval– –nostringval– –nostringval– –nostringval– 1 %stopped_push –nostringval– –nostringval– –nostringval– %array_continue –nostringval– false 1 %stopped_push –nostringval– %loop_continue –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval– –nostringval– –nostringval– –nostringval– %loop_continue –nostringval– 12 9 %oparray_pop –nostringval– –nostringval– –nostringval– –nostringval– –nostringval–
Dictionary stack:
–dict:1127/1686(ro)(G)– –dict:0/20(G)– –dict:107/200(L)– –dict:107/200(L)– –dict:104/127(ro)(G)– –dict:241/347(ro)(G)– –dict:20/24(L)– –dict:4/6(L)– –dict:24/31(L)– –dict:38/50(ro)(G)–
Current allocation mode is local
Last OS error: 2
ESP Ghostscript 815.02: Unrecoverable error, exit code 1
convert: Postscript delegate failed `/home/kale/poop.pdf’.
convert: missing an image filename `/home/kale/poop.pnm’.

In this instance, the PDF poop.pdf contains a Chinese font which Ghostscript knows about but won’t let me use since I installed it after-the-fact from RPM (chinese-fonts.noarch). GhostScript by default is run from ImageMagick with the -dSAFER flag which is a mildly paranoid flag that prevents GS from using files outside of its root (/usr/share/ghostscript, usually). To work around this error, rather than dealing with GS’s confusing and arcane font directory configuration, I recommend simply amending ImageMagick’s delegates definition for the PDF filetype:

/usr/lib64/ImageMagick-6.2.8/config/delegates.xml:

<delegate decode=”pdf” encode=”ps” mode=”bi” command='”gs” -q -dBATCH -dSAFER -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE=”pswrite” -sOutputFile=”%o” -f”%i”‘ />

to

<delegate decode=”pdf” encode=”ps” mode=”bi” command='”gs” -q -dBATCH -dMaxBitmap=500000000 -dNOPAUSE -dAlignToPixels=0 -dEPSCrop -sDEVICE=”pswrite” -sOutputFile=”%o” -f”%i”‘ />

(remove the -dSAFER flag)

Henceforth, `convert poop.pdf poop.jpg’ will work with third-party fonts without issue.

Enumerate columns for awk

I’m bad at counting, so when I’m using awk to print specific fields, I end up with greasy fingerprints on my screen as I manually count out each field. Thanks to my colleague James, here’s a script that counts for you!

awk 'NR == 1 { for (i=1;i<=NF;i++) {printf i " "} print ""} {print}' | column -t

Works with STDIN as is, assuming default field separator (space):

[kale@superhappykittymeow log]# tail -n 1 xferlog |awk 'NR == 1 { for (i=1;i<=NF;i++) {printf i " "} print ""} {print}' | column -t
1    2    3   4         5     6  7          8    9                          10  11  12  13  14    15   16  17  18
Sat  Jun  19  13:19:25  2010  1  127.0.0.1  220  /var/www/poop/wp-rss2.php  b   _   i   r   root  ftp  0   *   c

Or, if you're lazy like myself, encapsulate it in an alias:

alias count='awk 'NR == 1 { for (i=1;i<=NF;i++) {printf i " "} print ""} {print}' | column -t'

[kale@superhappykittymeow log]# tail -n 1 xferlog | count
1    2    3   4         5     6  7          8    9                          10  11  12  13  14    15   16  17  18
Sat  Jun  19  13:19:25  2010  1  127.0.0.1  220  /var/www/poop/wp-rss2.php  b   _   i   r   root  ftp  0   *   c

Enforcing Secure Passwords in Horde

A coworker, Alex, discovered that Horde, in conjunction with Plesk, allows users to change their passwords arbitrarily — but doesn’t enforce any sort of password policy, allowing such passwords as “test” or even “” (null). This, obviously, is a huge security risk as mail compromises can lead to fairly terrible things.

From his article:

If you (or a client you are representing) want to set horde to do the typical “strict password” enforcement, look for the file:

horde/passwd/backends.php

And read the bit about password policy. An example policy that can be set in this file that would require 1 capital, 1 lowercase, 1 special character and 1 number, with a minimum password size of 8, would look like:

‘password policy’ => array(
‘minLength’ => 8,
‘maxLength’ => 64,
‘maxSpace’ => 0,
‘minUpper’ => 1,
‘minLower’ => 1,
‘minNumeric’ => 1,
‘minSymbols’ => 1
),

Can’t fork?

Can’t fork but need to see what’s going on? Hint: a box that can’t fork can often `exec’.

Here are a pair of slick bash functions that can be lifesavers in dire situations:

`ls’:

$ myls() { while [ $# -ne 0 ] ; do echo "$1" ; shift ; done ; }
$ myls /etc/s*
/etc/services
/etc/shells
/etc/syslog.conf

`cat’:

$ mycat() { while IFS="" read l ; do echo "$l" ; done < $1 ; }
$ mycat /etc/shells

Python on OSX proxy bug with curl

I received a lovely stack trace trying to run a python script that makes an external call using curl. The meat of it:

File “/Library/Python/2.6/site-packages/pyechonest/util.py”, line 52, in call
f = urllib.urlopen(url)
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py”, line 87, in urlopen
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py”, line 203, in open
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py”, line 310, in open_http
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py”, line 1555, in proxy_bypass
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py”, line 1425, in proxy_bypass_macosx_sysconf

This is a bug due to how python on OSX parses the values in the proxy bypass box in the system’s Proxy configuration. If you have an explicit IP, it will bomb out. Remove the explicitly specified IP or change it to a name value to get around this bug. Netblocks (such as “10/8”) won’t cause it to bomb — just a definite IP like 1.2.3.4.

Oh, perl…

Installing EV from CPAN, I received this message:

Checking if your kit is complete…
Looks good
Warning: prerequisite common::sense 0 not found.

:/

Stop snmpd from spamming syslog

Dell’s OpenManage tools come with a MIB to allow for the system to query against and set up traps. However, by default, snmpd will not allow arbitrary smux peers, and your logs will be spammed with the following:

Apr 29 19:19:37 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39622
Apr 29 19:19:37 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:40 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39693
Apr 29 19:19:40 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
Apr 29 19:19:43 server snmpd[4321]: [smux_accept] accepted fd 9 from 127.0.0.1:39790
Apr 29 19:19:43 server snmpd[4321]: refused smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager

Add the following to you /etc/snmpd/snmpd.conf:

smuxpeer .1.3.6.1.4.1.674.10892.1

and restart snmpd.

Notes on iPad’s movie player

The iPad’s built-in movie player identifies itself as the useragent “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)” and explicitly specifies a range to download, allowing for chunked streaming. Here are the headers from both sides of a recent transaction on my server:

GET /music/lt-fundies.mp4 HTTP/1.1
Host: growl.superhappykittymeow.com
Range: bytes=110691320-147521535
Connection: close
User-Agent: AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)
Accept: */*
Accept-Encoding: identity

HTTP/1.1 206 Partial Content
Date: Sat, 24 Apr 2010 09:28:22 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Wed, 17 Sep 2008 18:35:48 GMT
ETag: \”4ad19f-998329f-4571bbcac8900\”
Accept-Ranges: bytes
Content-Length: 36830216
Content-Range: bytes 110691320-147521535/160969375
Connection: close
Content-Type: text/plain; charset=UTF-8

Fun fact – a dropped packet results in awesome skipping on the client! …it also spams your access_logs with each request:

66.216.90.167 – – [24/Apr/2010:09:33:59 +0000] “GET /music/lt-fundies.mp4 HTTP/1.1” 206 7012352 “-” “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)”
66.216.90.167 – – [24/Apr/2010:09:34:00 +0000] “GET /music/lt-fundies.mp4 HTTP/1.1” 206 26657760 “-” “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)”
66.216.90.167 – – [24/Apr/2010:09:34:02 +0000] “GET /music/lt-fundies.mp4 HTTP/1.1” 206 2558 “-” “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)”
66.216.90.167 – – [24/Apr/2010:09:34:02 +0000] “GET /music/lt-fundies.mp4 HTTP/1.1” 206 6912055 “-” “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)”
66.216.90.167 – – [24/Apr/2010:09:34:03 +0000] “GET /music/lt-fundies.mp4 HTTP/1.1” 206 26500344 “-” “AppleCoreMedia/1.0.0.7B367 (iPad; U; CPU OS 3_2 like Mac OS X)”

SNMP RAID monitoring with LSI SATA RAID controllers

wget http://ftp.us.dell.com/ide/lin_apps_a04.tar.gz
tar xzvf lin_apps_a04.tar.gz
rpm -Uvh percsnmp-4.06-1.i386.rpm
cp /etc/percsnmp/release/perc.mib /usr/share/snmp/mibs

If your SNMP config works, try this:

snmpwalk -m +RAID-Adapter-MIB -v1 -c public localhost .1.3.6.1.4.1.3582

Log PHP memory usage per-request

You can easily log how much memory each request for a PHP page takes by modifying the LogFormat:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b %{mod_php_memory_usage}n \"%{Referer}i\" \"%{User-Agent}i\"" combined-php

Note the latter definition, which includes %{mod_php_memory_usage}n — this will print out the amount of memory, in bytes, required to execute the script as requested. Big help in finding memory leaks. To use, just change the log definition to the newly-created “combined-php” format:

CustomLog logs/fever-access_log combined-php

Do note that this may (probably will) break Apache log parsers that are expecting the standard combined format. If using for troubleshooting, I recommend logging to an alternate location so as to not screw up log statistics.

Credit to Brad Ison for this find

Run Urchin on-demand for all profiles at once

There’s no built-in way in Urchin to re-run the processing job for all domains (such as after fixing a problem). This can, however, be done on the command line with a while loop:

[code lang=”bash”]ls -alh ../usr/local/urchin/data/reports/ |awk ‘{print $NF}’ |while read line ; do /usr/local/urchin/bin/urchin -p”$line” ; done[/code]

Block POSTs from blank referrers

I found a great article on Secure Computing: Sec-C that includes some excellent, simple Apache configurations and RewriteRules to blog various annoyances and compromises. A wonderful example is this bit, designed to stop POST requests that have no referrer set. There’s no reason for anyone to be trying to post arbitrary data to a script to not have a referrer, as that would indicate a direct hit — which is bad juju.

# Identify if a Referer is used
SetEnvIf Referer "^$" no_referer=1
<Limit POST>
Order Allow,Deny
Allow from all
Deny from env=no_referer
</Limit>

Lots of other fascinating security and forensics insights on the Sec-C blog as well!

HOWTO: Install VNC server on RHEL5

Install a desktop environment and the VNC server:

yum groupinstall "GNOME Desktop Environment"
yum  install vnc-server

Change to the user who will be owning the session and run `vncserver’ to set up their password and create the default files

su kale -
vncserver

Edit the xstartup file for that user to point to GNOME:

vi /home/kale/.vnc/xstartup

# Uncomment the following two lines for normal desktop:
 unset SESSION_MANAGER
 exec /etc/X11/xinit/xinitrc

Kill that VNC session

killall Xvnc

Edit vncserver’s config file

vi /etc/sysconfig/vncservers

VNCSERVERS="2:kale"
VNCSERVERARGS[2]="-geometry 800x600 -nohttpd"

Start VNC

service vncserver start

And connect! Since it’s specified in /etc/sysconfig/vncservers that kale’s session is on display 2 (the 2:kale bit), the port for this connection is 5902. Connect to this port and enter in the password you specified earlier and voila!

WHOIS visiting your site?

I’m fond of WHOIS data for getting an idea who’s visiting a site, though most WHOIS servers return data that’s full of disclaimers and irrelevant data. Rather, I much prefer Team Cymru’s batch WHOIS lookup server, whois.cymru.com.

First, extract your IPs:
[code lang=”bash”]F=ips.out ; echo “begin”>>$F ; echo “verbose”>>$F ; awk ‘{print $1}’ tech-access_log |sort |uniq>>$F ; echo “end” >>$F[/code]

Now send them to Cymru for processing:
[code lang=”bash”]nc whois.cymru.com 43 < $F | sort > whois.out[/code]

Review whois.out at your leisure for detailed IP information. It’s well-formatted, allowing for easily scripting against:

91      | 128.113.197.128  | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
91      | 128.113.247.58   | 128.113.0.0/16      | US | arin     | 1986-02-27 | RPI-AS - Rensselaer Polytechnic Institute
9121    | 88.232.9.77      | 88.232.0.0/17       | TR | ripencc  | 2005-10-27 | TTNET TTnet Autonomous System
9       | 128.2.161.88     | 128.2.0.0/16        | US | arin     | 1984-04-17 | CMU-ROUTER - Carnegie Mellon University
9136    | 91.186.50.28     | 91.186.32.0/19      | DE | ripencc  | 2006-11-07 | WOBCOM WOBCOM GmbH - www.wobcom.de
9143    | 212.203.31.1     | 212.203.0.0/19      | NL | ripencc  | 2000-08-08 | ZIGGO Ziggo - tv, internet, telefoon