I’m a big fan of unit testing, and for PHP projects I use PHPUnit. Combined with XDebug PHPUnit provides some great code coverage reporting options, including detailed HTML reports and a variety of different XML report options that work nicely with various different continuous integration servers and other tools.

A feature I’ve often wanted, that is currently missing from PHPUnit, is a simple command line display of code coverage statistics when you’re running tests. If you’re trying to improve the code coverage of a specific test it isn’t practical to generate a full HTML report of the test every time you run it, and then check it in a browser. It would be much easier if the code coverage could be reported next to the success or failure message on the command line.

After a little bit of investigation into the different PHPUnit reports, and some git, awk, and sed hackery I came up with the following PHPUnit wrapper that does exactly what I was after. Once you’ve run a test, or collection of tests, it’ll display a list of files affected by the test and the code coverage for each file. The code for this script is below:

#!/bin/bash
METRIC_FILE="/tmp/test-metrics.xml"
# Call PHPUnit will all provided command line arguments, and also generate a metrics file
phpunit --log-metrics "$METRIC_FILE" $@
# If phpunit exits with an error then exit here with the same error code
if [ $? -ne 0 ]
then
   exit $?
fi
# Do some manipulation of the metrics file and output each file and the coverage for that file
echo "*** SIMPLE COVERAGE REPORT ***"
cat "$METRIC_FILE" | grep "<file" | awk '{print $2 $10}' | sed
"s|$PWD||g" | awk -F\" '{printf "%-50s %.2f%\n", $2, $4}'
# Cleanup
rm "$METRIC_FILE"

I’ve place the script in /usr/bin/phpuc on my machine, and then I can invoke it like so:

$ phpcs tests/ExampleTest.xml

PHPUnit 3.3.16 by Sebastian Bergmann.

.........

Time: 0 seconds

OK (9 tests, 9 assertions)

Writing metrics report XML file, this may take a moment.
*** SIMPLE COVERAGE REPORT ***
Example.php                     89.37%
IncludedFile.php                 12.9%

Hopefully this will be something that’ll be built-in to PHPUnit in the future!

There have been lots of links to automatic website thumbnail generators on sites like reddit and hacker news today, including webkit2png and CutyCapt. Well it just so happens that a few weeks ago I wrote my own website thumbnail generator, and today I got around to putting it on GitHub.

The code is based on Matt Biddulph’s screenshot-tng script, but heavily modified to be more user friendly and provide more options. It uses embedded mozilla for rendering, and therefore requires the python-gtkmozembed package.

You can specify a resolution to take the screenshot at, and also a resolution for the thumbnail. When generating the thumbnail the aspect ratio will be preserved. You can also specify a delay, so that the screenshot is only taken so many seconds after loading the page. Here’s an example of running PyWebShot with 3 URLs, and the resulting images:

$ ./pywebshot.py -t 500x250 http://www.coderholic.com http://geomium.com/update/598/ http://jobs.plasis.co.uk
Loading http://www.coderholic.com... saved as www.coderholic.com.png
Loading http://geomium.com/update/598/... saved as geomium.com.update.598..png
Loading http://jobs.plasis.co.uk... saved as jobs.plasis.co.uk.png

It you have a huge list of URLs you’d like to generate screenshots for you can put them all into a file and generate images for them all with the following command:

$ cat urls.txt | xargs ./pywebshot.py

For more details and the source code see the PyWebShot project page on GitHub.

At the start of the year I announced that I was setting myself the challenge of learning 12 new programming languages during 2010. That works out at a language a month, so seeing as it’s almost the end of March you might expect me to be wrapping up my third language. Unfortunately that isn’t the case. I’m just about to move on to my second. I’m still optimistic I can achieve the target of 12 new languages this year though, so expect future updates to be more regular.

Getting Starting with Clojure

Getting up and running with Clojure was made easy to to the wealth of documentation. There’s a great getting started guide, and a guide specifically for Clojure on Ubuntu. In terms of programming environment there’s a round up of Clojure IDEs. I stuck to Vim, but I didn’t take it as far as this guide, which describes turning Vim into a fairly comprehensive Clojure IDE.

Clojure has a REPL, which I always find makes learning a new language easier. When you want to find something out just type it in a see what the result is! The default REPL doesn’t support arrow navigation or pressing up to run previous commands though, so it can be a little frustrating. There are guides on enhacing the REPL with this functionality.

One of the first things I did was put together the following shell script which either runs the specified Clojure script, or gives you a REPL if no script was specified.

#!/bin/sh
CLOJURE_JAR="/opt/clojure-1.1.0/clojure.jar"
if [ -z "$1" ]; then 
        java -jar "${CLOJURE_JAR}" 
else
        java -jar "${CLOJURE_JAR}" "$@"
fi

Writing Code

I decided to write a port scanner in Clojure, which would introduce me to command line argument handling, the network API, and parallelization.

One of the great things about Clojure is that despite being such a new language there is so much example code available on the web. I was able to find a Clojure network scanner by Travis Whitton which detailed all of the network related code I’d need to my port scanner. Travis uses Clojure’s agents for parallelization, which is something else I borrowed from his script. I was amazed at how easy it was to parallelize the lookups. So much simpler than threading. If there is only one thing I take away from Clojure it’ll be its interesting approaches to parallelization.

So without further ado here is my Clojure port scanner:

(import '(java.io IOException)
        '(java.net Socket)
        '(java.net InetSocketAddress)
        '(java.net SocketTimeoutException)
        '(java.net UnknownHostException))
 
(if (== (count *command-line-args*) 1)
  (def hostname (first *command-line-args*))
  (
    (println "Usage: scanner <hostname>")
        (System/exit 1)
  ))
 
(defn port-open? [hostname port timeout]
  (let [sock-addr (InetSocketAddress. hostname port)]
    (try
     (with-open [sock (Socket.)]
       (. sock connect sock-addr timeout)
       port)
     (catch IOException e false)
     (catch SocketTimeoutException e false)
     (catch UnknownHostException e false))))
     	<li>
(defn host-port-open? [port]
  (port-open? hostname port 5000))
 
(def port-list (range 1 1024))
 
(def agents (for [port port-list] (agent port)))
 
(println (str "Scanning " hostname "..."))
 
(doseq [agent agents]
  (send-off agent host-port-open?))
 
(apply await agents)
 
(doseq [port (filter deref agents)]
       (println (str @port " is open")))
 
(shutdown-agents)

I’m sure it is far from an idiomatic solution, so any suggestions for improvement are welcome. Running the scanner with my bash script gives the following output:

$ ./clj.sh scanner
Usage: scanner <hostname>

$ ./clj.sh scanner github.com
Scanning github.com...
22 is open
80 is open
443 is open

What’s next?

I’ve barely scratched the surface of Clojure, but I’ve certainly become more aware of some of the concepts and idioms used by the language, which is what I was hoping for. I’ll be looking for more projects in the future where I can make use of it. For now though I need to move on to another language as part of my challenge. As I’m running behind I plan to go with one that I don’t think will be too unfamiliar, either Go or Fantom. I’ll keep you posted!

After writing about Munin last week and mentioning some of its uses outside of system performance tracking I decided to write a collection of plugins for tracking website popularity. The full code is available on GitHub.

The plugins include:

• Alexa traffic rank
• Number of pages in the Google index
• Technorati authority
• Feedburner RSS subscribers
• Twitter followers

Munin will automatically handle the generation of daily, weekly, monthly and yearly graphs. All of the plugins support graphing of multiple sites/accounts, so you can show several sites on the same graph, as is the case with the Google index graph above, or you can decide to show a single site/account per graph. Some example graphs are shown below:

Munin is an excellent open source tool for monitoring and graphing server performance metrics. It can be configured to send out alert emails when something goes wrong with your server, and the graphs make it easy to view trends over time: You could see that your site gets much less traffic on a Sunday, for example, or that the number of database queries performed per day has doubled in the last 2 months.

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!