Browse Category: howto

The 10 Golden Rules for Troubleshooting Linux

1. Man pages exist and should be used. Seriously, everything’s there, from application docs to syscall docs to syntax and formatting of log files.

2. Don’t reinvent the wheel. 99% of problems you’re experiencing or ever will experience, somebody’s already gone through it and figured it out. Google is your friend.

3. If you don’t know what something’s doing, or why it’s not working, strace it!

4. Logs exist for a reason. Read them.

5. Applications crash, servers don’t. If your server crashes, it’s either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups. Always.

7. Always mount NFS mounts with the ‘intr’ option. Having to reboot because of a network blip is uncool.

8. Learn to use `grep’, `sed’ and `awk’. Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage. 100% memory usage does not mean you don’t have any more available for new applications. You can run out of inodes before you run out of disk space.

10. TCP wrappers suck. If you’ve been hacking at an issue for over 3 hours, look to your TCP wrappers. /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.

Manually mounting a USB drive in Linux

Most modern distros are quite smart and will recognize the variety of USB devices plugged in automagically. Today, for whatever reason, RHEL5 refused to do so, and I had to do it the hard way.

First, make sure you’ve got USB modules loaded:

[code lang=”bash”]modprobe uhci_hcd
modprobe ohci_hcd
modprobe ehci_hcd
modprobe usb-storage[/code]

With the last one, wait a few moments, then run `dmesg’. You should see some useful information:

Initializing USB Mass Storage driver...
scsi1 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: waiting for device to settle before scanning
usb 1-3.3: reset high speed USB device using ehci_hcd and address 8
Vendor: Seagate Model: FreeAgent Go Rev: 102D
Type: Direct-Access ANSI SCSI revision: 04
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
sda: sda1
sd 1:0:0:0: Attached scsi disk sda
sd 1:0:0:0: Attached scsi generic sg0 type 0
usb-storage: device scan complete

If you’re lucky, you can simple do an `fstab -l’ and see the drive and it’s partitions at the stated point (sda). I wasn’t so lucky, as this server didn’t have device nodes for sda. These, however, are easily created:

[code lang=”bash”]/dev/MAKEDEV sda[/code]

Now you should be able to mount it:

[code lang=”bash”]mount /dev/sda1 /mnt/usb[/code]

Intercepting “command not found” in bash

On Debian, bash is patched with an interesting new function: command_not_found_handle. This intercepts exit code 127 (“command not found”) and allows you to do neat things. Debian uses it to pass it through the apt database, letting you know if a command you tried to invoke is not available, but can be found in the apt repos, and how to install it. Pretty spiffy.

This, of course, can be modified. Where I work, we use numbers to identify servers. I have a script that grabs login credentials from our internal systems and auto-logs me into servers based on their number. For example, I’d run `connect 12345′ to connect to server 12345.

By adding the following to my .bashrc:

[code lang=”bash”]function command_not_found_handle {
/home/kale/bin/command-not-found $1

And creating the following script, with regex in place to only care about numbers, placed in /home/kale/bin/command-not-found:

[code lang=”bash”]#!/bin/bash

MYBOOL=`echo $1 | awk ‘$1 ~ /^[0-9]+-*[0-9]*$/ {print $0}’ | wc -l | awk ‘{print $1}’`

if [ “$MYBOOL” == “1” ]; then
/home/kale/bin/connect $1
exit 127

(where `connect’ is the path to my connect script, previously written)

This now allows me to do this awesome deal:

[code lang=”bash”]kale@bastion:~$ 12345
Connecting to server 12345…

If you didn’t catch it, I don’t need to specify a command — just the argument. As there’s no application in my $PATH named `12345′, it falls through to the command_not_found_handle function, which then launches my connect script.

Who needs commands? I just saved hundreds of wasted seconds per night on typing “connect”!

Removing 1-liner malware code in webpages

More and more client workstations are being infected with keyloggers and trojans. In addition to stealing your WoW username and password (oh noes, my purpz!), they also have been stealing FTP logins.

This has manifested itself in the linux server world by seemingly legit users logging in over FTP, downloading a file, then uploading it a few seconds later with 100-ish bytes appended. A look at xferlog reveals this behavior, usually against a regex of pages (index.*, default.*, etc), and the connecting IP will often be foreign. A look at the secure log will reveal that the password was not brute-forced; rather, it was known.

The real solution is to change all passwords and force the end user to reformat their computer, since they’re infected and do not realize it. Alas, this is not quite practical (though if someone could invent a remote formatter, I’ll give you $10 for it). Rather, advise the end user of the situation and suggest reformatting — or, at the very least, using a collection of anti-spyware, anti-virus, and anti-everything software on their workstation. Change the affected user’s password.

To clean up the leftover malicious code that was appended, find out the exact string (usually a `tail index.php’ will reveal it) — it’s often a <JavaScript> line or an <iframe>. Copy the string completely and we’ll just sed it out:

[code lang=”bash”]sed -i “s#