A feature I’ve often wanted, that is currently missing from PHPUnit, is a simple command line display of code coverage statistics when you’re running tests. If you’re trying to improve the code coverage of a specific test it isn’t practical to generate a full HTML report of the test every time you run it, and then check it in a browser. It would be much easier if the code coverage could be reported next to the success or failure message on the command line.

After a little bit of investigation into the different PHPUnit reports, and some git, awk, and sed hackery I came up with the following PHPUnit wrapper that does exactly what I was after. Once you’ve run a test, or collection of tests, it’ll display a list of files affected by the test and the code coverage for each file. The code for this script is below:

#!/bin/bash
METRIC_FILE="/tmp/test-metrics.xml"
# Call PHPUnit will all provided command line arguments, and also generate a metrics file
phpunit --log-metrics "$METRIC_FILE" $@
# If phpunit exits with an error then exit here with the same error code
if [ $? -ne 0 ]
then
   exit $?
fi
# Do some manipulation of the metrics file and output each file and the coverage for that file
echo "*** SIMPLE COVERAGE REPORT ***"
cat "$METRIC_FILE" | grep "<file" | awk '{print $2 $10}' | sed
"s|$PWD||g" | awk -F\" '{printf "%-50s %.2f%\n", $2, $4}'
# Cleanup
rm "$METRIC_FILE"

I’ve place the script in /usr/bin/phpuc on my machine, and then I can invoke it like so:

$ phpcs tests/ExampleTest.xml

PHPUnit 3.3.16 by Sebastian Bergmann.

.........

Time: 0 seconds

OK (9 tests, 9 assertions)

Writing metrics report XML file, this may take a moment.
*** SIMPLE COVERAGE REPORT ***
Example.php                     89.37%
IncludedFile.php                 12.9%

Hopefully this will be something that’ll be built-in to PHPUnit in the future!

The code is based on Matt Biddulph’s screenshot-tng script, but heavily modified to be more user friendly and provide more options. It uses embedded mozilla for rendering, and therefore requires the python-gtkmozembed package.

You can specify a resolution to take the screenshot at, and also a resolution for the thumbnail. When generating the thumbnail the aspect ratio will be preserved. You can also specify a delay, so that the screenshot is only taken so many seconds after loading the page. Here’s an example of running PyWebShot with 3 URLs, and the resulting images:

$ ./pywebshot.py -t 500x250 http://www.coderholic.com http://geomium.com/update/598/ http://jobs.plasis.co.uk
Loading http://www.coderholic.com... saved as www.coderholic.com.png
Loading http://geomium.com/update/598/... saved as geomium.com.update.598..png
Loading http://jobs.plasis.co.uk... saved as jobs.plasis.co.uk.png

It you have a huge list of URLs you’d like to generate screenshots for you can put them all into a file and generate images for them all with the following command:

$ cat urls.txt | xargs ./pywebshot.py

For more details and the source code see the PyWebShot project page on GitHub.

Getting Starting with Clojure

Getting up and running with Clojure was made easy to to the wealth of documentation. There’s a great getting started guide, and a guide specifically for Clojure on Ubuntu. In terms of programming environment there’s a round up of Clojure IDEs. I stuck to Vim, but I didn’t take it as far as this guide, which describes turning Vim into a fairly comprehensive Clojure IDE.

Clojure has a REPL, which I always find makes learning a new language easier. When you want to find something out just type it in a see what the result is! The default REPL doesn’t support arrow navigation or pressing up to run previous commands though, so it can be a little frustrating. There are guides on enhacing the REPL with this functionality.

One of the first things I did was put together the following shell script which either runs the specified Clojure script, or gives you a REPL if no script was specified.

#!/bin/sh
CLOJURE_JAR="/opt/clojure-1.1.0/clojure.jar"
if [ -z "$1" ]; then 
        java -jar "${CLOJURE_JAR}" 
else
        java -jar "${CLOJURE_JAR}" "$@"
fi

Writing Code

I decided to write a port scanner in Clojure, which would introduce me to command line argument handling, the network API, and parallelization.

One of the great things about Clojure is that despite being such a new language there is so much example code available on the web. I was able to find a Clojure network scanner by Travis Whitton which detailed all of the network related code I’d need to my port scanner. Travis uses Clojure’s agents for parallelization, which is something else I borrowed from his script. I was amazed at how easy it was to parallelize the lookups. So much simpler than threading. If there is only one thing I take away from Clojure it’ll be its interesting approaches to parallelization.

So without further ado here is my Clojure port scanner:

(import '(java.io IOException)
        '(java.net Socket)
        '(java.net InetSocketAddress)
        '(java.net SocketTimeoutException)
        '(java.net UnknownHostException))
 
(if (== (count *command-line-args*) 1)
  (def hostname (first *command-line-args*))
  (
    (println "Usage: scanner <hostname>")
        (System/exit 1)
  ))
 
(defn port-open? [hostname port timeout]
  (let [sock-addr (InetSocketAddress. hostname port)]
    (try
     (with-open [sock (Socket.)]
       (. sock connect sock-addr timeout)
       port)
     (catch IOException e false)
     (catch SocketTimeoutException e false)
     (catch UnknownHostException e false))))
     	<li>
(defn host-port-open? [port]
  (port-open? hostname port 5000))
 
(def port-list (range 1 1024))
 
(def agents (for [port port-list] (agent port)))
 
(println (str "Scanning " hostname "..."))
 
(doseq [agent agents]
  (send-off agent host-port-open?))
 
(apply await agents)
 
(doseq [port (filter deref agents)]
       (println (str @port " is open")))
 
(shutdown-agents)

I’m sure it is far from an idiomatic solution, so any suggestions for improvement are welcome. Running the scanner with my bash script gives the following output:

$ ./clj.sh scanner
Usage: scanner <hostname>

$ ./clj.sh scanner github.com
Scanning github.com...
22 is open
80 is open
443 is open

What’s next?

I’ve barely scratched the surface of Clojure, but I’ve certainly become more aware of some of the concepts and idioms used by the language, which is what I was hoping for. I’ll be looking for more projects in the future where I can make use of it. For now though I need to move on to another language as part of my challenge. As I’m running behind I plan to go with one that I don’t think will be too unfamiliar, either Go or Fantom. I’ll keep you posted!

After writing about Munin last week and mentioning some of its uses outside of system performance tracking I decided to write a collection of plugins for tracking website popularity. The full code is available on GitHub.

The plugins include:

• Alexa traffic rank
• Number of pages in the Google index
• Technorati authority
• Feedburner RSS subscribers
• Twitter followers

Munin will automatically handle the generation of daily, weekly, monthly and yearly graphs. All of the plugins support graphing of multiple sites/accounts, so you can show several sites on the same graph, as is the case with the Google index graph above, or you can decide to show a single site/account per graph. Some example graphs are shown below:

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!

bmd /~: ./dbSize.sh coderholic root ********
wp_comments Data: 6.60MB Indexes: .15MB Total: 6.75MB
wp_links Data: 0MB Indexes: 0MB Total: 0MB
wp_options Data: 1.57MB Indexes: .01MB Total: 1.58MB
wp_postmeta Data: .01MB Indexes: .01MB Total: .02MB
wp_posts Data: .57MB Indexes: .02MB Total: .60MB
wp_term_relationships Data: 0MB Indexes: .01MB Total: .02MB
wp_term_taxonomy Data: 0MB Indexes: 0MB Total: 0MB
wp_terms Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_data Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_rss_map Data: 0MB Indexes: 0MB Total: 0MB
wp_usermeta Data: 0MB Indexes: 0MB Total: 0MB
wp_users Data: 0MB Indexes: 0MB Total: 0MB
*** 12 Tables | Data: 8.78MB Indexes: .25MB Total: 9.03MB ***

With a few small modifications I’m sure it’d be possible to get the script working PostgreSQL or other RDBMSes. The full code is below:

#!/bin/bash
# Calculate the storage space used up by all tables in a given MySQL database
# Ben Dowling - www.coderholic.com
database=$1
username=$2
password=$3

if [ ${#database} -eq 0 ]
then
	echo "Usage: $0 <database> [username [password]]"
	exit
fi

if [ "$password" ]
then
   password="-p$password"
fi

mysql="mysql -u $username $password $database"

$mysql -se "USE $database";

tables=$($mysql -se "SHOW TABLES")

totalData=0
totalIndex=0
totalTables=0

for table in $tables
do
   output=$($mysql -se "SHOW TABLE STATUS LIKE \"$table\"\G")
   data=$(echo "$output" | grep Data_length | awk -F': ' '{print $2}')
   dataMegs=$(echo "scale=2;$data/1048576" | bc)
   index=$(echo "$output" | grep Index_length | awk -F': ' '{print $2}')
   indexMegs=$(echo "scale=2;$index/1048576" | bc)
   total=$(($index+$data))
   totalMegs=$(echo "scale=2;$total/1048576" | bc)

   echo "$table Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB"

   totalData=$(($totalData+$data))
   totalIndex=$(($totalIndex+$index))
   totalTables=$(($totalTables+1))
done

dataMegs=$(echo "scale=2;$totalData/1048576" | bc)
indexMegs=$(echo "scale=2;$totalIndex/1048576" | bc)
total=$(($totalIndex+$totalData))
totalMegs=$(echo "scale=2;$total/1048576" | bc)

echo "*** $totalTables Tables | Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB ***"

watch ls -l

The watch program clears the screen and displays the output of “ls -l” every 2 seconds.

Sometimes I’ll want to monitor a command that only outputs a single line. If I wanted to see the total number of files in a directory rather than the files themselves I could use the command “ls -l | wc -l”. The fact that watch clears the whole screen can be a little annoying here though, because the command is only outputting a single line. That is why I came up with the following small bash script, linewatch.

Linewatch repeatedly calls any arguments passed to it every 2 seconds (in the same way watch does), but only clears a single line rather than the whole screen. Here is the code:

#!/bin/bash
clearline="\b\033[2K\r"
command=$@

while true
do
    eval "$command"
    sleep 2
    echo -n -e "$clearline"
done

And here is an example of how to call it:

$ ./linewatch "ls -l | wc -l"
24

The number of files in the current directory (24 in the example) will keep update every 2 seconds. Just hit Ctrl-C when you want to quit,

It wouldn’t take much to get it working with other version control systems such as Git or Bazaar, or to do some nice formatting of the output instead of outputting the raw svn log as-is. Let me know if you find it useful!

#!/bin/bash
# Shell script to email the latest changes in an SVN
# repsitory to a specified email address.
# Ben Dowling - wwww.coderholic.com

svnUrl="http://anonsvn.wireshark.org/wireshark/trunk/"
lastRevisionFile="./.last-revision"
mailto="ben@coderholic.com"

function getCurrentRevision {
  # Get the current SVN revision, eg. "r4670"
  currentRevision=$(svn log "$svnUrl" -r HEAD 2>/dev/null | head -n2 | grep -v -- "-------" | awk '{ print $1 }')
  # Strip off the 'r'
  currentRevision="${currentRevision:1}"
  echo "$currentRevision"
}

currentRevision=$(getCurrentRevision)

# If we've run this program before then we've stored the SVN revision at the time
if [ -f "$lastRevisionFile" ]
then
  lastRevision=$(cat "$lastRevisionFile")
  #  Check what the current revision is, and exit if there
  # haven't been any changes since we last checked
  if [ $currentRevision -lt $lastRevision ]
  then
      echo "No changes since last check"
      exit
  fi
else
  # We haven't run this program before, so set the last revision to the current revision - 10
  lastRevision=$(echo "$currentRevision - 10" | bc)
fi

# Mail the SVN changes
svn log "$svnUrl" -r "HEAD:${lastRevision}" | mail -s "SVN changes for $svnUrl" $mailto

# Store the current revision + 1 as the last revision
revision=$(echo "$currentRevision + 1" | bc)
echo "$revision" > "$lastRevisionFile"

<?php
	// Ensure logging is enabled. Ideally this will be set in php.ini
	ini_set('log_errors', 'On');
	// Call a non-existant function to generate an error
	doesNotExist();
?>

You’ll see something like the following entry in the error log, which includes a description of the problem and the file and line number of where the problem occurred:


[Tue May 26 19:54:53 2009] [error] [client 127.0.0.1] PHP Fatal error: Call to undefined function doesNotExist() in /var/www/index.php on line 6


I’ve written a small bash script to mail me all of the PHP errors that it finds in the log. The results are sorted and piped through the “uniq” command to remove any duplicate errors. I’ve set it up to run as a cron task once a day, so if there are ever any problems it isn’t long before I know about them. The code is below:

#!/bin/bash
# Mail out PHP errors that are in the apache error log.
# Note PHP's log_errors must be turned on
# Ben Dowling - www.coderholic.com

errorLog=/var/log/apache2/error.log # Error log location
email=you-email-address@example.com # Send report here

# Pull out the lines that mention PHP, and use AWK to get the column we're interested in
errors=$(cat $errorLog | grep PHP | awk -F'] ' '{print $4}')
# Remove referer information, sort, and remove duplicate entries
errors=$(echo "$errors" | awk -F', referer' '{print $1}' | sort | uniq)
# Check that we actually have some errors
if [ -n "$errors" ]
then
	echo "$errors" | mail "$email" -s "PHP Errors"
fi

If you need to it should be quite easy to modify the script for a slightly different use, such as printing out all the fatal PHP errors that are in the log.

Hope you find it useful!

The script requires mencoder, ffmpeg, dvdauthor and mkisofs, and will let you know you if any of these are missing. Use it like follows:

./dvd.sh input.avi

Once the script has finished a “dvd.iso” file will be created, which can then be burned to DVD using your favourite disc burning tool.

#!/bin/bash
# AVI to DVD Script
# Ben Dowling - www.coderholic.com

# Change to "ntsc" if you'd like to create NTSC disks
format="pal"

# Check we have enough command line arguments
if [ $# != 1 ]
then
	echo "Usage: $0 <input file>"
	exit
fi

# Check for dependencies
missing=0
dependencies=( "mencoder" "ffmpeg" "dvdauthor" "mkisofs" )
for command in ${dependencies[@]}
do
	if ! command -v $command &>/dev/null
	then
		echo "$command not found"
		missing=1
	fi
done

if [ $missing = 1 ]
then
	echo "Please install the missing applications and try again"
	exit
fi

function emphasise() {
	echo ""
	echo "********** $1 **********"
	echo ""
}

# Check the file exists
input_file=$1
if [ ! -e $input_file ]
then
	echo "Input file not found"
	exit
fi

emphasise "Converting AVI to MPG"

ffmpeg -i finalmovie.avi -y -target ${format}-dvd -sameq -aspect 16:9 finalmovie.mpg

if [ $? != 0 ]
then
	emphasise "Conversion failed"
	exit
fi

emphasise "Creating DVD contents"

dvdauthor --title -o dvd -f finalmovie.mpg
first=$?
dvdauthor -o dvd -T
second=$?

if [ $first != 0 || $second != 0 ]
then
	emphasise "DVD Creation failed"
	exit
fi

emphasise "Creating ISO image"

mkisofs -dvd-video -o dvd.iso dvd/

if [ $? != 0 ]
then
	emphasise "ISO Creation failed"
	exit
fi

# Everything passed. Cleanup
rm -f finalmovie.mpg
rm -rf dvd/

emphasise "Success: dvd.iso image created"