The code is based on Matt Biddulph’s screenshot-tng script, but heavily modified to be more user friendly and provide more options. It uses embedded mozilla for rendering, and therefore requires the python-gtkmozembed package.

You can specify a resolution to take the screenshot at, and also a resolution for the thumbnail. When generating the thumbnail the aspect ratio will be preserved. You can also specify a delay, so that the screenshot is only taken so many seconds after loading the page. Here’s an example of running PyWebShot with 3 URLs, and the resulting images:

$ ./pywebshot.py -t 500x250 http://www.coderholic.com http://geomium.com/update/598/ http://jobs.plasis.co.uk
Loading http://www.coderholic.com... saved as www.coderholic.com.png
Loading http://geomium.com/update/598/... saved as geomium.com.update.598..png
Loading http://jobs.plasis.co.uk... saved as jobs.plasis.co.uk.png

It you have a huge list of URLs you’d like to generate screenshots for you can put them all into a file and generate images for them all with the following command:

$ cat urls.txt | xargs ./pywebshot.py

For more details and the source code see the PyWebShot project page on GitHub.

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!

watch ls -l

The watch program clears the screen and displays the output of “ls -l” every 2 seconds.

Sometimes I’ll want to monitor a command that only outputs a single line. If I wanted to see the total number of files in a directory rather than the files themselves I could use the command “ls -l | wc -l”. The fact that watch clears the whole screen can be a little annoying here though, because the command is only outputting a single line. That is why I came up with the following small bash script, linewatch.

Linewatch repeatedly calls any arguments passed to it every 2 seconds (in the same way watch does), but only clears a single line rather than the whole screen. Here is the code:

#!/bin/bash
clearline="\b\033[2K\r"
command=$@

while true
do
    eval "$command"
    sleep 2
    echo -n -e "$clearline"
done

And here is an example of how to call it:

$ ./linewatch "ls -l | wc -l"
24

The number of files in the current directory (24 in the example) will keep update every 2 seconds. Just hit Ctrl-C when you want to quit,

It wouldn’t take much to get it working with other version control systems such as Git or Bazaar, or to do some nice formatting of the output instead of outputting the raw svn log as-is. Let me know if you find it useful!

#!/bin/bash
# Shell script to email the latest changes in an SVN
# repsitory to a specified email address.
# Ben Dowling - wwww.coderholic.com

svnUrl="http://anonsvn.wireshark.org/wireshark/trunk/"
lastRevisionFile="./.last-revision"
mailto="ben@coderholic.com"

function getCurrentRevision {
  # Get the current SVN revision, eg. "r4670"
  currentRevision=$(svn log "$svnUrl" -r HEAD 2>/dev/null | head -n2 | grep -v -- "-------" | awk '{ print $1 }')
  # Strip off the 'r'
  currentRevision="${currentRevision:1}"
  echo "$currentRevision"
}

currentRevision=$(getCurrentRevision)

# If we've run this program before then we've stored the SVN revision at the time
if [ -f "$lastRevisionFile" ]
then
  lastRevision=$(cat "$lastRevisionFile")
  #  Check what the current revision is, and exit if there
  # haven't been any changes since we last checked
  if [ $currentRevision -lt $lastRevision ]
  then
      echo "No changes since last check"
      exit
  fi
else
  # We haven't run this program before, so set the last revision to the current revision - 10
  lastRevision=$(echo "$currentRevision - 10" | bc)
fi

# Mail the SVN changes
svn log "$svnUrl" -r "HEAD:${lastRevision}" | mail -s "SVN changes for $svnUrl" $mailto

# Store the current revision + 1 as the last revision
revision=$(echo "$currentRevision + 1" | bc)
echo "$revision" > "$lastRevisionFile"

The “Do you Scale” presentation I saw at PHP London a couple of months ago gave a good high level overview of scalability issues, and included some useful techniques to help you scale.

I think I’ve found the ultimate scalability presentation though: “Real World Web: Performance & Scalability”. The 189 slides contained within this presentation cover almost everything I’ve read elsewhere, and it’s packed full of practice advice!

Killing lots of connections can be tedious though. Say an application has created lots of persistent connections and not closed any of them, you’d have to go through the following steps:


mysql> SHOW FULL PROCESSLIST; /* shows all connections */
mysql> KILL <process_id>; /* for EVERY process you want to kill */


To make this process much simpler I wrote a little bash script that will kill all connections for a given MySQL user.

#!/bin/bash
# Kill all MySQL connections for a given user
# Ben Dowling - Apr 2009
# www.coderholic.com

user="root" # MySQL super user
pass="" # Above user's password
kill_user="myapp" # User who you want to kill all the connections for

# Get the connection IDs for all connections from the user $kill_user
processList=$(mysql -u$user -p$pass -s -e "SHOW FULL PROCESSLIST" | grep $kill_user | awk '{print $1}')
# Loop through all the connection IDs and kill the connection
for id in $processList
do
       result=$(mysql -u$user -p$pass -s -e "KILL $id")
       echo $result
done

The MySQL superuser account name and password, along with the user you want to kill all the connection for, are embedded within the script. It would be quite simple to pass these in as command line arguments if you wanted to though.

The solution I’ve come up with (and I’m sure others have too) is to put all my config files into into a Git repository on GitHub. Any changes to the config files can be committed to the repository, and than a quick update operation other machines keeps all the config files in sync. In this post I’ll describe how to set it all up.

Adding your config files to GitHub

You’ll obviously need to install Git. On any debian based system this is as simple as:

   sudo aptitude install git-core

You’ll also nee to Signup for a free GitHub account, and create a new repository. GitHub uses public-key cryptography and SSH for authentication, so you’ll need to create a SSH public key on each machine you’ll be accessing GitHub from. Details of how to do this are given in their guide. After adding your public key details to your GitHub account page you’re ready to add your config files:

  git init
  git add .bashrc
  git add .vimrc
  git add <any other config files you want to add>
  git commit -m 'Initial commit'
  git remote add origin git@github.com:coderholic/config.git # Your URL will differ
  git push origin master

Now all your config files are on GitHub!

Setting up a new machine

You’ll need to create a SSH public key on the new machine, just as you did on the previous one. Once you’ve done that:

Once you’ve done that:

  git init
  git pull git@github.com:coderholic/config.git
  git remote add origin git@github.com:coderholic/config.git

And your setup will match your other machine!

Making changes

The real benefit of storing your config files on GitHub is that it makes it really easy to synchronise changes between all your computers. After making a change on one computer:

  git commit -a -m "Useful commit message goes here"
  git push origin master

Then all you need to do on your other machines is issue the following command:

  git pull origin master

And you’re synced!

If you’re interested in seeing my config files you can see them on the GitHub page: http://github.com/coderholic/config/tree/master

The script requires mencoder, ffmpeg, dvdauthor and mkisofs, and will let you know you if any of these are missing. Use it like follows:

./dvd.sh input.avi

Once the script has finished a “dvd.iso” file will be created, which can then be burned to DVD using your favourite disc burning tool.

#!/bin/bash
# AVI to DVD Script
# Ben Dowling - www.coderholic.com

# Change to "ntsc" if you'd like to create NTSC disks
format="pal"

# Check we have enough command line arguments
if [ $# != 1 ]
then
	echo "Usage: $0 <input file>"
	exit
fi

# Check for dependencies
missing=0
dependencies=( "mencoder" "ffmpeg" "dvdauthor" "mkisofs" )
for command in ${dependencies[@]}
do
	if ! command -v $command &>/dev/null
	then
		echo "$command not found"
		missing=1
	fi
done

if [ $missing = 1 ]
then
	echo "Please install the missing applications and try again"
	exit
fi

function emphasise() {
	echo ""
	echo "********** $1 **********"
	echo ""
}

# Check the file exists
input_file=$1
if [ ! -e $input_file ]
then
	echo "Input file not found"
	exit
fi

emphasise "Converting AVI to MPG"

ffmpeg -i finalmovie.avi -y -target ${format}-dvd -sameq -aspect 16:9 finalmovie.mpg

if [ $? != 0 ]
then
	emphasise "Conversion failed"
	exit
fi

emphasise "Creating DVD contents"

dvdauthor --title -o dvd -f finalmovie.mpg
first=$?
dvdauthor -o dvd -T
second=$?

if [ $first != 0 || $second != 0 ]
then
	emphasise "DVD Creation failed"
	exit
fi

emphasise "Creating ISO image"

mkisofs -dvd-video -o dvd.iso dvd/

if [ $? != 0 ]
then
	emphasise "ISO Creation failed"
	exit
fi

# Everything passed. Cleanup
rm -f finalmovie.mpg
rm -rf dvd/

emphasise "Success: dvd.iso image created"

#!/bin/bash

# Query a supplied URL and output an error message if
# the status code was not 200. Can be used as a cron
# task to check if a server is up or not.

# Ben Dowling - http://www.coderholic.com

# The URL to query
if [ $# -ne 1 ]
then
	echo "Usage: $0 <URL>";
	exit;
fi

url=$1;
response=$(curl -s -I -L $url | grep HTTP); 

status=${response#* }; # Strip off characters up to the first space
status=${status:0:3}; # Just use the 3 digit status code

if [ "$status" != "200" ]
then
	echo "Error fetching $url. Status code '$status'";
fi

I then set this up as a cron task to run every 15 minutes, and to email me if the site doesn’t respond with a HTTP 200 OK status:

*/15 * * * * /path/to/checkserver.sh "http://www.coderholic.com" | mail ben@coderholic.com

bmd /dir: ls
-folder
bmd /dir: cd -folder
-bash: cd: -f: invalid option
cd: usage: cd [-L|-P] [dir]
bmd /dir: cd \-folder
-bash: cd: -f: invalid option
cd: usage: cd [-L|-P] [dir]
bmd /dir: cd “-folder”
-bash: cd: -f: invalid option
cd: usage: cd [-L|-P] [dir]

I wasn’t having much luck! It turns out solution is to pass “--” (two hypens) as the first argument to cd:

bmd /dir: cd — -folder
/home/bmd/dir/-folder
bmd /-folder:

I think I remember having to use this trick in the past, but it’s such an infrequent problem that I obviously forgot about it. Hopefully by posting about it I’ll have a better change of remembering it for next time!