I’ve been a long time user of MySQL, and despite having had some frustrations I’ve been pretty happy with MySQL. I’m comfortable with it, I know where its weaknesses lie and where it’s strong. I’ve never really had to look at alternatives.

This all changed for a project that has been taking up all of my spare time lately, Geomium, a location-based communication tool. MySQL’s geospacial support is almost completely non-existant. PostgreSQL, on the other had, when combined with the PostGIS extension, has fantastic support. So PostgreSQL was what the project needed. I thought it’d be fairly easy to transition from MySQL to PostgresSQL but the command line clients of these two RDBMSes are very different. Here are some of the things I’ve learnt.

A few other things to note: PostgreSQL only supports single quotes in queries not double quotes, so

SELECT * FROM table WHERE column = "value"

would need to be rewritten as

SELECT * FROM table WHERE column = 'value'

PostgreSQL also defaults to being case-sensitive, whereas MySQL defaults to case-insensitive, so you might want to modify your queries to something like

SELECT * FROM table WHERE lower(column) = 'value'

although this can prevent an index on “column” from being used.

At first I thought PostgreSQL didn’t support some of the handy MySQL date functions (such as NOW() – INTERVAL 1 DAYS), but it turns out that it does support them, and more. It’s just fussier about the format. For details see the datetime documentation.

Munin is an excellent open source tool for monitoring and graphing server performance metrics. It can be configured to send out alert emails when something goes wrong with your server, and the graphs make it easy to view trends over time: You could see that your site gets much less traffic on a Sunday, for example, or that the number of database queries performed per day has doubled in the last 2 months.

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!

I’ve written a small shell script to report how much disk space each table in a given MySQL database is using. For example, below is the output of the script when run against this site’s database:

bmd /~: ./dbSize.sh coderholic root ********
wp_comments Data: 6.60MB Indexes: .15MB Total: 6.75MB
wp_links Data: 0MB Indexes: 0MB Total: 0MB
wp_options Data: 1.57MB Indexes: .01MB Total: 1.58MB
wp_postmeta Data: .01MB Indexes: .01MB Total: .02MB
wp_posts Data: .57MB Indexes: .02MB Total: .60MB
wp_term_relationships Data: 0MB Indexes: .01MB Total: .02MB
wp_term_taxonomy Data: 0MB Indexes: 0MB Total: 0MB
wp_terms Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_data Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_rss_map Data: 0MB Indexes: 0MB Total: 0MB
wp_usermeta Data: 0MB Indexes: 0MB Total: 0MB
wp_users Data: 0MB Indexes: 0MB Total: 0MB
*** 12 Tables | Data: 8.78MB Indexes: .25MB Total: 9.03MB ***

With a few small modifications I’m sure it’d be possible to get the script working PostgreSQL or other RDBMSes. The full code is below:

#!/bin/bash
# Calculate the storage space used up by all tables in a given MySQL database
# Ben Dowling - www.coderholic.com
database=$1
username=$2
password=$3

if [ ${#database} -eq 0 ]
then
	echo "Usage: $0 <database> [username [password]]"
	exit
fi

if [ "$password" ]
then
   password="-p$password"
fi

mysql="mysql -u $username $password $database"

$mysql -se "USE $database";

tables=$($mysql -se "SHOW TABLES")

totalData=0
totalIndex=0
totalTables=0

for table in $tables
do
   output=$($mysql -se "SHOW TABLE STATUS LIKE \"$table\"\G")
   data=$(echo "$output" | grep Data_length | awk -F': ' '{print $2}')
   dataMegs=$(echo "scale=2;$data/1048576" | bc)
   index=$(echo "$output" | grep Index_length | awk -F': ' '{print $2}')
   indexMegs=$(echo "scale=2;$index/1048576" | bc)
   total=$(($index+$data))
   totalMegs=$(echo "scale=2;$total/1048576" | bc)

   echo "$table Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB"

   totalData=$(($totalData+$data))
   totalIndex=$(($totalIndex+$index))
   totalTables=$(($totalTables+1))
done

dataMegs=$(echo "scale=2;$totalData/1048576" | bc)
indexMegs=$(echo "scale=2;$totalIndex/1048576" | bc)
total=$(($totalIndex+$totalData))
totalMegs=$(echo "scale=2;$total/1048576" | bc)

echo "*** $totalTables Tables | Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB ***"

I was really pleased to come across the “SQL Antipatterns Stirke Back” presentation recently, which discusses common mistakes with SQL database design. It gives some really good advice on how best to design databases to avoid these issues. I’ve certainly made some of the mistakes mentioned, and I’m sure I’ll be referring back to this presentation again and again!

At work we’re experiencing some fairly rapid growth, and our single production server is starting the feel the strain. I’ve been doing a lot of investigation into how we can scale the site, and thankfully there is lots of information out there.

The “Do you Scale” presentation I saw at PHP London a couple of months ago gave a good high level overview of scalability issues, and included some useful techniques to help you scale.

I think I’ve found the ultimate scalability presentation though: “Real World Web: Performance & Scalability”. The 189 slides contained within this presentation cover almost everything I’ve read elsewhere, and it’s packed full of practice advice!