Browse Tag: bash

Command-line Elasticsearch client

We use the ELK stack extensively at my job, thanks to my evangelizing and endless hard work. With all our servers logging to logstash and being pushed to Elasticsearch, logging into servers via ssh just to check logs is a thing of the past, and to help push that ideology along, I’ve hacked up a simple bash script to query Elasticsearch and return results in a manner that mimics running `tail` on a server’s logs. It quite literally just runs a query against Elasticsearch’s HTTP API, but I added some niceties so I can allow folks to make queries to ES without having to read a novel on how to do so.

[kstedman@kalembp:~/bin] $ ./ -h 10.x.x.x:9200 -q "+host:xxx6039 +type:syslog" -t 1000 -n 4
2014-07-09T05:05:26.000000+00:00 xxx6039 snmpd[15279]: Connection from UDP: [10.x.x.x]:57258
2014-07-09T05:05:26.000000+00:00 xxx6039 snmpd[15279]: Received SNMP packet(s) from UDP: [10.x.x.x]:57258
2014-07-09T05:05:26.000000+00:00 xxx6039 snmpd[15279]: Connection from UDP: [10.x.x.x]:57258
2014-07-09T05:05:27.000000+00:00 xxx6039 snmpd[15279]: Connection from UDP: [10.x.x.x]:57258

Of course, since this is text output on the console, you can use it as inputs/outputs to scripts, sed/grep/awk to your heart’s content, etc. Requires python datetime and json. Enjoy!

# Search server logs from the comfort of your terminal!
# This is a command-line wrapper for Elasticsearch's RESTful API.
# This is super-beta, version .000001-alpha. Questions/comments/hatemail to Kale Stedman,
# I'm so sorry. You should probably pipe the output to less.
# usage: ./ -u $USER -p $PASS -h es-hostname -q "$query" -t $time -n 500
# ex: ./ -u kstedman -p hunter2 -h -q "program:crond" -t 5 -n 50
# -h host      The Elasticsearch host you're trying to connect to.
# -u username  Optional: If your ES cluster is proxied through apache and you have http auth enabled, username goes here
# -p password  Optional: If your ES cluster is proxied through apache and you have http auth enabled, password goes here
# -q query     Optional: Query to pass to ES. If not given, "*" will be used.
# -t timeframe Optional: How far back to search. Value is in mimutes. If not given, defaults to 5.
# -n results   Optional: Number of results to return. If not given, defaults to 500.

# Declare usage fallback/exit
usage() { echo "Usage: $0 -h host [ -u USER ] [ -p PASS ] [ -q "QUERY" ] [ -t TIMEFRAME ] [ -n NUMRESULTS ]" 1>&2; exit 1; }

# Parse options
while getopts ":u:p:h:q:t:n:" o; do
    case "${o}" in
shift $((OPTIND-1))

if [ -z "${p}" ] && [ ! -z "${u}" ] ; then
  echo -n "Password: "
  read -s p

# Check for required variables
if [ -z "${h}" ] ; then

# Set defaults if not set
if [ -z "${n}" ] ; then
  # default: 500 results returned 

if [ -z "${q}" ] ; then
  # default: query "*"

if [ -z "${t}" ] ; then
  # default: 5 minutes ago

# cross-platform time compatibilities
FROMDATE=`python -c "from datetime import date, datetime, time, timedelta; print ( - timedelta(minutes=${t})).strftime('%s')"`
NOWDATE=`python -c "from datetime import date, datetime, time, timedelta; print ('%s')"`

# Build query

if [ ! -z "${u}" ] ; then

# run query and prettify the output
curl -s -XGET "${URL}" -d ''"${query}"'' | python -mjson.tool |grep '"message"' | awk -F\: -v OFS=':' '{ $1=""; print $0}' | sed -e 's/^: "//g' | sed -e 's/", $//g' | sed -e 's/\\n/\

Über-simple generic RHEL/CentOS init script

Fill in the indicated bits, drop in /etc/rc.d/init.d/ , chmod +x, and away you go!

# chkconfig: 2345 90 90
# description: program_name
# Provides: program_name
# Required-Start: network
# Required-Stop: network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Description: Start the program

### Fill in these bits:
START_CMD="java -jar /home/kale/bin/program_name.jar > /var/log/program_name/program_name.log 2>&1 &"

### No further muckin' about needed!


killproc() {
  pkill -u $USER -f $PGREP_STRING

start_daemon() {
  eval "$*"

log_success_msg() {
  echo "$*"
  logger "$_"

log_failure_msg() {
  echo "$*"
  logger "$_"

check_proc() {
  pgrep -u $USER -f $PGREP_STRING >/dev/null

start_script() {
  if [ "${CUR_USER}" != "root" ] ; then
    log_failure_msg "$NAME can only be started as 'root'."
    exit -1

  if [ $? -eq 0 ]; then
    log_success_msg "$NAME is already running."
    exit 0

  [ -d /var/run/$NAME ] || (mkdir /var/run/$NAME )

   # make go now 
    start_daemon /bin/su $USER -c "$START_CMD"

  # Sleep for a while to see if anything cries
  sleep 5

  if [ $? -eq 0 ]; then
    log_success_msg "Started $NAME."
    log_failure_msg "Error starting $NAME."
    exit -1

stop_script() {
  if [ "${CUR_USER}" != "root" ] ; then
    log_failure_msg "You do not have permission to stop $NAME."
    exit -1

  if [ $? -eq 0 ]; then
    killproc -p $PID_FILE >/dev/null

    # Make sure it's dead before we return
    until [ $? -ne 0 ]; do
      sleep 1

    if [ $? -eq 0 ]; then
      log_failure_msg "Error stopping $NAME."
      exit -1
      log_success_msg "Stopped $NAME."
    log_failure_msg "$NAME is not running or you don't have permission to stop it"

check_status() {
  if [ $? -eq 0 ]; then
    log_success_msg "$NAME is running."
    log_failure_msg "$NAME is stopped."
    exit -1

case "$1" in
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1

exit 0

bash, /dev/net and you

found during a random goog search on Dave Smith’s Blog:

exec 3<>/dev/tcp/
echo -e "GET / HTTP/1.1\n\n" >&3
cat <&3

seriously, wrap your head around that. /dev/net isn't a real device, it's a magical pseudodevice that bash intercepts and opens a socket as requested.

Backup/restore Elasticsearch index

[UPDATED 2017-03-09]
I still get comments/questions regarding this process I hacked together many moons ago. I must request that anybody who’s looking for a way to backup Elasticsearch indices STOP and do not follow the process described — it was for ES 0.00000000001, written back in 2011. You should not do what I suggest here! I’m saving this purely for historical purposes.

What you should do instead is save your events in flat text — in Logstash, output to both your ES index for searching via Kibana or whatnot, and also output your event to a flat file, likely periodic (per-day or month or whatever). Backup and archive these text files, since they compress quite well. When you want to restore data from a period, just re-process it through Logstash — CPU is cheap nowadays with cloud instances! The data is the important part — processed or not, if you have the data in an easily stored format, you can re-process it.

[Original post as follows]

I’ve been spending a lot of time with Elasticsearch recently, as I’ve been implementing logstash for our environment. Logstash, by the way, is a billion times awesome and I can’t recommend it enough for large-scale log management/search. Elasticsearch is pretty awesome too, but considering the sheer amount of data I was putting into it, I don’t feel satisfied with its replication-based redundancy — I need backups that I can save and restore at will. Since logstash creates a new Elasticsearch index for each day worth of logs, I want the ability to backup and restore arbitrary indices.

Elasticsearch has a concept of a gateway, wherein you can configure a gateway that maintains metadata and snapshots are regularly taken. “Regularly” as in every 10 seconds by default. The docs recommend using S3 as a gateway, meaning every 10s it’ll ship data up to S3 for backup purposes, and if a node ever needs to recover data, it can just look to S3 and get the metadata and fill in data from that source. However, this model does not support the “rotation”-style backup and restore I’m looking for, and it can’t keep up with the rate of data I’m sending it (my daily indices are about 15gb apiece, making for about 400k log entries an hour).

So I’ve come up with a pair of scripts that allow me to manage logstash/Elasticsearch index data, allowing for arbitrary restore of an index, as well as rotation so as to keep the amount of data that Elasticsearch keeps track of manageable. As always, I wrote my scripts for my environment, so I take no responsibility if they do not work in yours and instead destroy all your data (a distinct possibility). I include these scripts here because I spent a while trying to figure this out and couldn’t find any information elsewhere on the net.

The following script backs up today’s logstash index. I’m retarded at timezones, so I managed to somehow ship my logs to logstash in GMT, so my “day” ends at 5pm, when logstash closes its index and opens a new one for the new day. Shortly after logstash closes an index (stops writing to it, not “close” in the Elasticsearch sense), I run the following script in cron, which backs up the index, backs up the metadata, creates a restore script, and sticks it all in S3:

# herein we backup our indexes! this script should run at like 6pm or something, after logstash
# rotates to a new ES index and theres no new data coming in to the old one. we grab metadatas,
# compress the data files, create a restore script, and push it all up to S3.

TODAY=`date +"%Y.%m.%d"`
INDEXNAME="logstash-$TODAY" # this had better match the index name in ES
BACKUPCMD="/usr/local/backupTools/s3cmd --config=/usr/local/backupTools/s3cfg put"
YEARMONTH=`date +"%Y-%m"`

# create mapping file with index settings. this metadata is required by ES to use index file data
echo -n "Backing up metadata... "
curl -XGET -o /tmp/mapping "http://localhost:9200/$INDEXNAME/_mapping?pretty=true" > /dev/null 2>&1
sed -i '1,2d' /tmp/mapping #strip the first two lines of the metadata
echo '{"settings":{"number_of_shards":5,"number_of_replicas":1},"mappings":{' >> /tmp/mappost 
# prepend hardcoded settings metadata to index-specific metadata
cat /tmp/mapping >> /tmp/mappost
echo "DONE!"

# now lets tar up our data files. these are huge, so lets be nice
echo -n "Backing up data files (this may take some time)... "
mkdir -p $BACKUPDIR
nice -n 19 tar czf $BACKUPDIR/$INDEXNAME.tar.gz $INDEXNAME 
echo "DONE!"

echo -n "Creating restore script... "
# time to create our restore script! oh god scripts creating scripts, this never ends well...
cat << EOF >> $BACKUPDIR/$
# this script requires $INDEXNAME.tar.gz and will restore it into elasticsearch
# it is ESSENTIAL that the index you are restoring does NOT exist in ES. delete it
# if it does BEFORE trying to restore data.

# create index and mapping
echo -n "Creating index and mappings... "
curl -XPUT 'http://localhost:9200/$INDEXNAME/' -d '`cat /tmp/mappost`' > /dev/null 2>&1
echo "DONE!"

# extract our data files into place
echo -n "Restoring index (this may take a while)... "
tar xzf $BACKUPDIR/$INDEXNAME.tar.gz
echo "DONE!"

# restart ES to allow it to open the new dir and file data
echo -n "Restarting Elasticsearch... "
/etc/init.d/es restart
echo "DONE!"
echo "DONE!" # restore script done

# push both tar.gz and restore script to s3
echo -n "Saving to S3 (this may take some time)... "
echo "DONE!"

# cleanup tmp files
rm /tmp/mappost
rm /tmp/mapping

Restoring from this data is just as you would expect — download the backed up index.tar.gz and the associated to the same directory, chmod +x the, then run it. It will automagically create the index and put the data in place. This has the benefit of making backed up indices portable — you can “export” them from one ES cluster and import them to another.

As mentioned, because of logstash, I have daily indices that I back up; I also rotate them to prevent ES from having to search through billions of gigs of data over time. I keep 8 days worth of logs in ES (due to timezone issues) by doing the following:

# Performs 'rotation' of ES indices. Maintains only 8 indicies (1 week) of logstash logs; this script
# is to be run at midnight daily and removes the oldest one (as well as any 1970s-era log indices,
# as these are a product of timestamp fail).  Please note the insane amount of error-checking
# in this script, as ES would rather delete everything than nothing...

# Before we do anything, let's get rid of any nasty 1970s-era indices we have floating around
TIMESTAMPFAIL=`curl -s localhost:9200/_status?pretty=true |grep index |grep log |sort |uniq |awk -F\" '{print $4}' |grep 1970 |wc -l`
		curl -s localhost:9200/_status?pretty=true |grep index |grep log |sort |uniq |awk -F\" '{print $4}' |grep 1970 | while read line
				echo "Indices with screwed-up timestamps found; removing"
				echo -n "Deleting index $line: "
				curl -s -XDELETE http://localhost:9200/$line/
				echo "DONE!"

# Get list of indices; should we rotate?
INDEXCOUNT=`curl -s localhost:9200/_status?pretty=true |grep index |grep log |sort |uniq |awk -F\" '{print $4}' |wc -l`
if [ $INDEXCOUNT -lt "9" ] 
		echo "Less than 8 indices, bailing with no action"
		exit 0
		echo "More than 8 indices, time to do some cleaning"
		# Let's do some cleaning!
		OLDESTLOG=`curl -s localhost:9200/_status?pretty=true |grep index |grep log |sort |uniq |awk -F\" '{print $4}' |head -n1`
		echo -n "Deleting oldest index, $OLDESTLOG: "
		curl -s -XDELETE http://localhost:9200/$OLDESTLOG/
		echo "DONE!"

Sometimes, due to the way my log entries get to logstash, the timestamp is mangled, and logstash, bless its heart, tries so hard to index it. Since logstash is keyed on timestamps, though, this means every once in a while I get an index dated 1970 with one or two entries. There’s no harm save for any overhead of having an extra index, but it also makes it impossible to back those up or to be able to make any assumptions about the index names. I nuke the 1970s indices from orbit, and then, if there are more than 8 indices in logstash, drop the oldest. I run this script at midnight daily, after index backup. Hugest caveat in the world about the rotation: running `curl -s -XDELETE http://localhost:9200/logstash-10.14.2011/’ will delete index logstash-10.14.2011, as you’d expect. However, if that variable $OLDESTLOG is mangled somehow and this command is run: `curl -s -XDELETE http://localhost:9200//’, you will delete all of your indices. Just a friendly warning!

Add domains and users

Quick one liner to take a list of domains and create Apache vhosts from a template, create users, set their home dir, permissions etc

cat domains.out |while read line ; do DOMAIN=$line ; NODOTDOMAIN=`echo $DOMAIN | sed -e 's/\.//g'` ; mkdir -p /var/www/vhosts/$DOMAIN ; sed -e "s/$DOMAIN/g" /etc/httpd/vhost.d/default.vhost > /etc/httpd/vhost.d/$DOMAIN.conf ; useradd -d /var/www/vhosts/$DOMAIN $NODOTDOMAIN ; chown $NODOTDOMAIN:$NODOTDOMAIN /var/www/vhosts/$DOMAIN ; PASSWERD=`head -n 50 /dev/urandom | tr -dc A-Za-z0-9 | head -c8` ; echo $PASSWERD | passwd $NODOTDOMAIN --stdin ; echo "Domain: $DOMAIN" ; echo "User: $NODOTDOMAIN" ; echo "Password: $PASSWERD" ; echo ; done

Can’t fork?

Can’t fork but need to see what’s going on? Hint: a box that can’t fork can often `exec’.

Here are a pair of slick bash functions that can be lifesavers in dire situations:


$ myls() { while [ $# -ne 0 ] ; do echo "$1" ; shift ; done ; }
$ myls /etc/s*


$ mycat() { while IFS="" read l ; do echo "$l" ; done < $1 ; }
$ mycat /etc/shells

Auto-iptables off IPs with high connection counts

via Paul (

[code lang=”bash”]netstat -npa –inet | grep :80 | sed ‘s/:/ /g’ | awk ‘{print $6}’ | sort | uniq -c | sort -n | while read line; do one=`echo $line | awk ‘{print $1}’`; two=`echo $line | awk ‘{print $2}’`; if [ $one -gt 100 ];
then iptables -I INPUT -s $two -j DROP; fi; done; iptables-save | grep -P ‘^-A INPUT’ | sort | uniq -c | sort -n | while read line; do oneIp=`echo $line | awk ‘{print $1}’`; twoIp=`echo $line | awk ‘{print $5}’`; if [ $oneIp -gt 1 ]; then iptables -D INPUT -s $twoIp -j DROP; fi; done[/code]

This one-liner is quite effective when tossed into a file and run as a cronjob once per minute. Any IP with more than 100 concurrent connections — which, quite honestly, is far more than any one IP should ever have on a standard webserver — will be blocked via iptables. This script as a cronjob is extremely effective dealing with small-to-midsize DDoSes (too much traffic for Apache/whatever service to handle, but not saturating the pipe).

Combining text files as columns

To combine two (or more) text files as individual columns in the same file, such as:






[code]foo foobar
foo1 foobar1
foo2 foobar2
foo3 foobar3[/code]

rather than using an ugly combination of sed and awk, you can use the `paste’ command:

[code lang=”bash”]paste file1 file2[/code]

Extend bash functionality

[code lang=”bash”]# make bash autocomplete with up arrow
bind ‘”\e[A”:history-search-backward’
bind ‘”\e[B”:history-search-forward’

# make tab cycle through commands instead of listing
bind ‘”\t”:menu-complete'[/code]

Intercepting “command not found” in bash

On Debian, bash is patched with an interesting new function: command_not_found_handle. This intercepts exit code 127 (“command not found”) and allows you to do neat things. Debian uses it to pass it through the apt database, letting you know if a command you tried to invoke is not available, but can be found in the apt repos, and how to install it. Pretty spiffy.

This, of course, can be modified. Where I work, we use numbers to identify servers. I have a script that grabs login credentials from our internal systems and auto-logs me into servers based on their number. For example, I’d run `connect 12345′ to connect to server 12345.

By adding the following to my .bashrc:

[code lang=”bash”]function command_not_found_handle {
/home/kale/bin/command-not-found $1

And creating the following script, with regex in place to only care about numbers, placed in /home/kale/bin/command-not-found:

[code lang=”bash”]#!/bin/bash

MYBOOL=`echo $1 | awk ‘$1 ~ /^[0-9]+-*[0-9]*$/ {print $0}’ | wc -l | awk ‘{print $1}’`

if [ “$MYBOOL” == “1” ]; then
/home/kale/bin/connect $1
exit 127

(where `connect’ is the path to my connect script, previously written)

This now allows me to do this awesome deal:

[code lang=”bash”]kale@bastion:~$ 12345
Connecting to server 12345…

If you didn’t catch it, I don’t need to specify a command — just the argument. As there’s no application in my $PATH named `12345′, it falls through to the command_not_found_handle function, which then launches my connect script.

Who needs commands? I just saved hundreds of wasted seconds per night on typing “connect”!