Browse Tag: linux

Mount NTFS drive in RHEL5

Grab fuse, fuse-ntfs-3g and dkms-fuse from Dag’s repo:

[code lang=”bash”]wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/fuse-2.7.3-1.el5.rf.x86_64.rpm
wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/fuse-ntfs-3g-1.2310-1.el5.rf.x86_64.rpm
wget http://www.mirrorservice.org/sites/apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/dkms-fuse-2.7.2-1.nodist.rf.noarch.rpm[/code]

Install:

[code lang=”bash”]rpm -Uvh fuse-2.7.3-1.el5.rf.x86_64.rpm fuse-ntfs-3g-1.2310-1.el5.rf.x86_64.rpm dkms-fuse-2.7.2-1.nodist.rf.noarch.rpm[/code]

Mount:

[code lang=”bash”]mount.ntfs-3g /dev/sdc1 /mnt/usb/ -o force[/code]

Add a new hotswap SCSI drive

If you’ve got a SCSI card, SCA backplane, and a chassis that supports hotswappable drives, you can easily add a new drive without a reboot. Simply insert the drive and

[code lang=”bash”]cat /proc/scsi/scsi[/code]

Note the host, bus (channel), ID and LUN of the existing drive(s).

[code lang=”bash”]echo “scsi add-single-device h b i l” > /proc/scsi/scsi[/code]

where “h b i l” is the host, bus, id and lun of the new drive. If it’s going on the same adapter as the rest of the drive(s) listed already, use the same host, bus and lun — the id will be incremented by one:

[code lang=”bash”]echo “scsi add-single-device 1 0 1 0” > /proc/scsi/scsi[/code]

cat /proc/scsi/scsi and you should see the new device:

[code][root@foo scsi]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: HITACHI Model: HUS103073FL3800 Rev: SA1B
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: SEAGATE Model: ST3146707LC Rev: 0005
Type: Direct-Access ANSI SCSI revision: 03[/code]

fdisk and make a filesystem as usual!

Combining text files as columns

To combine two (or more) text files as individual columns in the same file, such as:

file1:

[code]foo
foo1
foo2
foo3[/code]

file2:

[code]foobar
foobar1
foobar2
foobar3[/code]

into:

[code]foo foobar
foo1 foobar1
foo2 foobar2
foo3 foobar3[/code]

rather than using an ugly combination of sed and awk, you can use the `paste’ command:

[code lang=”bash”]paste file1 file2[/code]

Save a file in vi as the superuser

If you’ve opened a file in vi for which you don’t have write privileges, you can save the file as the superuser (if your user is in the sudoers list) by running the following vi command:

[code]:w !sudo tee %[/code]

After doing so, vi will detect that the file has changed and ask if you want to reload it.

Force outgoing mail to come from a specific IP

Add an iptables rule such as:

[code lang=”bash”]iptables -t nat -A POSTROUTING -p tcp -s ! 127.0.0.1 –dport 25 -j SNAT –to-source 65.61.180.228[/code]

This will make all outgoing mail leave over the secondary IP, 65.61.180.228, as opposed to the primary IP address of the server (the standard behavior of MTAs).

Sort based on a column

You can use the `sort’ utility to sort not only on the first field, but also an arbitrary column. Take this output from ps:

[code lang=”bash”][kale@superhappykittymeow ~]$ ps ax -ly |grep httpd
R 502 10027 8387 0 78 0 668 979 – pts/1 0:00 grep httpd
S 502 10247 28321 0 76 0 16924 9961 semtim ? 1:58 /usr/sbin/httpd
S 0 28321 1 0 78 0 11564 7645 – ? 0:00 /usr/sbin/httpd
S 502 28327 28321 0 75 0 19152 10412 semtim ? 1:39 /usr/sbin/httpd
S 502 28328 28321 0 84 0 16628 9903 semtim ? 1:56 /usr/sbin/httpd
S 502 28331 28321 0 75 0 17108 9962 semtim ? 1:45 /usr/sbin/httpd
S 502 28332 28321 0 75 0 19152 10446 semtim ? 1:54 /usr/sbin/httpd
S 502 28333 28321 0 75 0 15692 9624 semtim ? 1:54 /usr/sbin/httpd
S 502 28334 28321 0 78 0 17476 10107 semtim ? 2:01 /usr/sbin/httpd
S 502 28335 28321 0 75 0 17460 10237 semtim ? 1:57 /usr/sbin/httpd
S 502 28336 28321 0 75 0 16836 9897 – ? 1:54 /usr/sbin/httpd
S 502 30058 28321 0 75 0 15248 9622 semtim ? 0:33 /usr/sbin/httpd
[/code]

This can be sorted by the SZ column, column 9, as such:

[code lang=”bash”][kale@superhappykittymeow ~]$ ps ax -ly |grep httpd |sort -nr -k 9n
R 502 10045 8387 0 78 0 668 979 – pts/1 0:00 grep httpd
S 0 28321 1 0 78 0 11564 7645 – ? 0:00 /usr/sbin/httpd
S 502 30058 28321 0 75 0 15248 9622 semtim ? 0:33 /usr/sbin/httpd
S 502 28333 28321 0 75 0 15692 9624 semtim ? 1:54 /usr/sbin/httpd
S 502 28336 28321 0 75 0 16836 9897 – ? 1:54 /usr/sbin/httpd
S 502 28328 28321 0 84 0 16628 9903 semtim ? 1:56 /usr/sbin/httpd
S 502 10247 28321 0 76 0 16924 9961 semtim ? 1:58 /usr/sbin/httpd
S 502 28331 28321 0 75 0 17108 9962 semtim ? 1:45 /usr/sbin/httpd
S 502 28334 28321 0 78 0 17476 10107 semtim ? 2:01 /usr/sbin/httpd
S 502 28335 28321 0 75 0 17460 10237 semtim ? 1:57 /usr/sbin/httpd
S 502 28327 28321 0 75 0 19152 10412 semtim ? 1:39 /usr/sbin/httpd
S 502 28332 28321 0 75 0 19152 10446 semtim ? 1:54 /usr/sbin/httpd
[/code]

The -k switch tells sort to sort based on a key, which we specify as 9n (column 9, numeric). Much easier to review the output.

Curl with postdata and cookies

Great for command-line logging into sites to pull content for whatever reason.

[code lang=”bash”]curl -c cookies.txt -d “username=username&password=password&action=login” -o /home/kale/outputfile.txt “http://www.domain.com/authenticated_page.php?foo=bar”[/code]

Of course, you’ll have to look at the source for the target location’s login page to see what variables it wants. I use it to grab a single Cacti-generated graph that is normally password protected, but I want to include a single graph on another site, so I cron’d a script to run a line similar to the above to log in and save it locally.

Make sure your crons run on time

If you add an entry to crontab that is an interval, such as */3 (every 3 minutes), you can verify that it runs at the specified interval with a bit of awk:

[code lang=”bash”]cat /var/log/cron |grep cron-script |awk -F\: ‘{if ($2/3 == 0) print $0}’ |grep -v “:00:”[/code]

This essentially checks to see that the minute field of the timestamp is divisible by three — the interval. It’ll also run at 00 after the hour, not divisible by three, but expected.

Cron can “run late” at times due to high load situations, so if there are any irregularities in your intervals, you may wish to investigate deeper, looking for expensive processes that are chewing up precious cron time.

The 10 Golden Rules for Troubleshooting Linux

1. Man pages exist and should be used. Seriously, everything’s there, from application docs to syscall docs to syntax and formatting of log files.

2. Don’t reinvent the wheel. 99% of problems you’re experiencing or ever will experience, somebody’s already gone through it and figured it out. Google is your friend.

3. If you don’t know what something’s doing, or why it’s not working, strace it!

4. Logs exist for a reason. Read them.

5. Applications crash, servers don’t. If your server crashes, it’s either bad hardware or a kernel bug (fairly rare on popular distros).

6. Always make backups. Always.

7. Always mount NFS mounts with the ‘intr’ option. Having to reboot because of a network blip is uncool.

8. Learn to use `grep’, `sed’ and `awk’. Learning to manipulate text is surprisingly important for a text-based interface.

9. Load average does not mean CPU usage. 100% memory usage does not mean you don’t have any more available for new applications. You can run out of inodes before you run out of disk space.

10. TCP wrappers suck. If you’ve been hacking at an issue for over 3 hours, look to your TCP wrappers. /etc/hosts, /etc/hosts.allow and /etc/hosts.deny will hold the answer.

Manually mounting a USB drive in Linux

Most modern distros are quite smart and will recognize the variety of USB devices plugged in automagically. Today, for whatever reason, RHEL5 refused to do so, and I had to do it the hard way.

First, make sure you’ve got USB modules loaded:

[code lang=”bash”]modprobe uhci_hcd
modprobe ohci_hcd
modprobe ehci_hcd
modprobe usb-storage[/code]

With the last one, wait a few moments, then run `dmesg’. You should see some useful information:

Initializing USB Mass Storage driver...
scsi1 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: waiting for device to settle before scanning
usb 1-3.3: reset high speed USB device using ehci_hcd and address 8
Vendor: Seagate Model: FreeAgent Go Rev: 102D
Type: Direct-Access ANSI SCSI revision: 04
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 1c 00 00 00
sda: assuming drive cache: write through
sda: sda1
sd 1:0:0:0: Attached scsi disk sda
sd 1:0:0:0: Attached scsi generic sg0 type 0
usb-storage: device scan complete

If you’re lucky, you can simple do an `fstab -l’ and see the drive and it’s partitions at the stated point (sda). I wasn’t so lucky, as this server didn’t have device nodes for sda. These, however, are easily created:

[code lang=”bash”]/dev/MAKEDEV sda[/code]

Now you should be able to mount it:

[code lang=”bash”]mount /dev/sda1 /mnt/usb[/code]

Segmentation faults with up2date/rpm

Had a nasty one tonight that I had to turn over to Paul. up2date was segfaulting on a RHEL3 server, along with some rpm queries, such as `rpm -qa’. I was able to narrow it down to a specific package, though that did not help at all. The rpm database had some severe corruption.

I removed the __db.00* files from /var/lib/rpm and rebuilt the database with rpm –rebuilddb; however, this did not resolve the issue. I attempted to recreate the database, by doing the following:

[code lang=”bash”]rpm -qa > rpm-list ; rpm –initdb ; cat rpm-list | while read line ; do rpm –nodeps –noscripts –notriggers –excludepath / $line ; done[/code]

However, the `rpm -qa’ hung, as noted earlier. Tough spot.

I had to move on, but Paul dove in. After a few hours of fitful hacking, he declared himself the winner — he had solved it. What did he do?

He removed all of the files in /var/lib/rpm except for Packages, then ran `rpm –rebuilddb’. The __db.00* files are just lock files, and while removing them can help, rpm transactions and queries still read all the other files and databases, thus rereading the corruption. Removing all the other databases (except the base Packages database) and then running a –rebuilddb operation actually rebuilds the databases… and the corruption cleared.

I also found this neat snippet to see if anything has a lock on the rpm databases:

[code lang=”bash”]cd /var/lib/rpm
/usr/lib/rpm/rpmdb_stat -Cl
[/code]

Intercepting “command not found” in bash

On Debian, bash is patched with an interesting new function: command_not_found_handle. This intercepts exit code 127 (“command not found”) and allows you to do neat things. Debian uses it to pass it through the apt database, letting you know if a command you tried to invoke is not available, but can be found in the apt repos, and how to install it. Pretty spiffy.

This, of course, can be modified. Where I work, we use numbers to identify servers. I have a script that grabs login credentials from our internal systems and auto-logs me into servers based on their number. For example, I’d run `connect 12345′ to connect to server 12345.

By adding the following to my .bashrc:

[code lang=”bash”]function command_not_found_handle {
/home/kale/bin/command-not-found $1
}
[/code]

And creating the following script, with regex in place to only care about numbers, placed in /home/kale/bin/command-not-found:

[code lang=”bash”]#!/bin/bash

MYBOOL=`echo $1 | awk ‘$1 ~ /^[0-9]+-*[0-9]*$/ {print $0}’ | wc -l | awk ‘{print $1}’`

if [ “$MYBOOL” == “1” ]; then
/home/kale/bin/connect $1
else
exit 127
fi
[/code]

(where `connect’ is the path to my connect script, previously written)

This now allows me to do this awesome deal:

[code lang=”bash”]kale@bastion:~$ 12345
Connecting to server 12345…
root@12345:~#
[/code]

If you didn’t catch it, I don’t need to specify a command — just the argument. As there’s no application in my $PATH named `12345′, it falls through to the command_not_found_handle function, which then launches my connect script.

Who needs commands? I just saved hundreds of wasted seconds per night on typing “connect”!

Removing 1-liner malware code in webpages

More and more client workstations are being infected with keyloggers and trojans. In addition to stealing your WoW username and password (oh noes, my purpz!), they also have been stealing FTP logins.

This has manifested itself in the linux server world by seemingly legit users logging in over FTP, downloading a file, then uploading it a few seconds later with 100-ish bytes appended. A look at xferlog reveals this behavior, usually against a regex of pages (index.*, default.*, etc), and the connecting IP will often be foreign. A look at the secure log will reveal that the password was not brute-forced; rather, it was known.

The real solution is to change all passwords and force the end user to reformat their computer, since they’re infected and do not realize it. Alas, this is not quite practical (though if someone could invent a remote formatter, I’ll give you $10 for it). Rather, advise the end user of the situation and suggest reformatting — or, at the very least, using a collection of anti-spyware, anti-virus, and anti-everything software on their workstation. Change the affected user’s password.

To clean up the leftover malicious code that was appended, find out the exact string (usually a `tail index.php’ will reveal it) — it’s often a <JavaScript> line or an <iframe>. Copy the string completely and we’ll just sed it out:

[code lang=”bash”]sed -i “s#