This all changed for a project that has been taking up all of my spare time lately, Geomium, a location-based communication tool. MySQL’s geospacial support is almost completely non-existant. PostgreSQL, on the other had, when combined with the PostGIS extension, has fantastic support. So PostgreSQL was what the project needed. I thought it’d be fairly easy to transition from MySQL to PostgresSQL but the command line clients of these two RDBMSes are very different. Here are some of the things I’ve learnt.

A few other things to note: PostgreSQL only supports single quotes in queries not double quotes, so

SELECT * FROM table WHERE column = "value"

would need to be rewritten as

SELECT * FROM table WHERE column = 'value'

PostgreSQL also defaults to being case-sensitive, whereas MySQL defaults to case-insensitive, so you might want to modify your queries to something like

SELECT * FROM table WHERE lower(column) = 'value'

although this can prevent an index on “column” from being used.

At first I thought PostgreSQL didn’t support some of the handy MySQL date functions (such as NOW() – INTERVAL 1 DAYS), but it turns out that it does support them, and more. It’s just fussier about the format. For details see the datetime documentation.

On a Debian-based system installing Munin is as simple as running the following command, and then going to http://your-server/munin/ in your browser:

sudo aptitude install munin

Munin comes with lots of monitoring plugins by default, including those for MySQL, PostgresSQL, Apache, Tomcat, Squid, and for things such a CPU and memory usage, load average, network traffic, and many more. You can also find lots of user submitted plugins on sites like Munin Exchange.

Munin doesn’t have to be used solely for monitoring server performance though. Being so easy to extend Munin is also a great tool for tracking non-server performance related trends over time. In just a few lines of code you could write plugins to track the following stats about your website:

• Number of User signups
• Google PageRank
• Pages in Google’s index
• Number of backlinks
• Number of twitter mentions
• Alexa traffic rank

The number of pages in Google’s index is actually a plugin I’ve written. Simple put the following code in your /etc/munin/plugins directory to see it in action:

#!/bin/sh
# Munin Plugin to display the number of pages in the
# google index for all of the given websites
# Ben Dowling - www.coderholic.com

# Change this to whatever sites you're interested in
websites="www.yahoo.com www.google.com www.twitter.com"

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

if [ "$1" = "config" ]; then

        echo 'graph_title Number of Pages in Google Index'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_vlabel number of pages'
        echo 'graph_category google'
        echo 'graph_info This graph shows the number of pages in the Google index for a given website.'

        i=0
        for site in $websites
        do
                name="site_${i}"
                echo "${name}.label ${site}"
                echo "${name}.draw LINE2"
                echo "${name}.info The number of pages in the google index."
                i=$((i+1))
        done
        exit 0
fi

i=0
for site in $websites
do
        name="site_${i}"
		value=$(wget -q --user-agent=Firefox -O - "http://www.google.com/search?q=site:${site}" | grep -E "of about [0-9,]+" -o | grep -E "[0-9,]+" -o | sed "s/,//g")
        echo "${name}.value ${value}"

        i=$((i+1))
done

For more details about Munin see their homepage, which also includes detailed documentation on writing your own plugins.

Let me know if you can think of any more Munin plugins that could be interesting, or if you’ve used any yourself!

bmd /~: ./dbSize.sh coderholic root ********
wp_comments Data: 6.60MB Indexes: .15MB Total: 6.75MB
wp_links Data: 0MB Indexes: 0MB Total: 0MB
wp_options Data: 1.57MB Indexes: .01MB Total: 1.58MB
wp_postmeta Data: .01MB Indexes: .01MB Total: .02MB
wp_posts Data: .57MB Indexes: .02MB Total: .60MB
wp_term_relationships Data: 0MB Indexes: .01MB Total: .02MB
wp_term_taxonomy Data: 0MB Indexes: 0MB Total: 0MB
wp_terms Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_data Data: 0MB Indexes: 0MB Total: 0MB
wp_tla_rss_map Data: 0MB Indexes: 0MB Total: 0MB
wp_usermeta Data: 0MB Indexes: 0MB Total: 0MB
wp_users Data: 0MB Indexes: 0MB Total: 0MB
*** 12 Tables | Data: 8.78MB Indexes: .25MB Total: 9.03MB ***

With a few small modifications I’m sure it’d be possible to get the script working PostgreSQL or other RDBMSes. The full code is below:

#!/bin/bash
# Calculate the storage space used up by all tables in a given MySQL database
# Ben Dowling - www.coderholic.com
database=$1
username=$2
password=$3

if [ ${#database} -eq 0 ]
then
	echo "Usage: $0 <database> [username [password]]"
	exit
fi

if [ "$password" ]
then
   password="-p$password"
fi

mysql="mysql -u $username $password $database"

$mysql -se "USE $database";

tables=$($mysql -se "SHOW TABLES")

totalData=0
totalIndex=0
totalTables=0

for table in $tables
do
   output=$($mysql -se "SHOW TABLE STATUS LIKE \"$table\"\G")
   data=$(echo "$output" | grep Data_length | awk -F': ' '{print $2}')
   dataMegs=$(echo "scale=2;$data/1048576" | bc)
   index=$(echo "$output" | grep Index_length | awk -F': ' '{print $2}')
   indexMegs=$(echo "scale=2;$index/1048576" | bc)
   total=$(($index+$data))
   totalMegs=$(echo "scale=2;$total/1048576" | bc)

   echo "$table Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB"

   totalData=$(($totalData+$data))
   totalIndex=$(($totalIndex+$index))
   totalTables=$(($totalTables+1))
done

dataMegs=$(echo "scale=2;$totalData/1048576" | bc)
indexMegs=$(echo "scale=2;$totalIndex/1048576" | bc)
total=$(($totalIndex+$totalData))
totalMegs=$(echo "scale=2;$total/1048576" | bc)

echo "*** $totalTables Tables | Data: ${dataMegs}MB Indexes: ${indexMegs}MB Total: ${totalMegs}MB ***"

The “Do you Scale” presentation I saw at PHP London a couple of months ago gave a good high level overview of scalability issues, and included some useful techniques to help you scale.

I think I’ve found the ultimate scalability presentation though: “Real World Web: Performance & Scalability”. The 189 slides contained within this presentation cover almost everything I’ve read elsewhere, and it’s packed full of practice advice!

Killing lots of connections can be tedious though. Say an application has created lots of persistent connections and not closed any of them, you’d have to go through the following steps:


mysql> SHOW FULL PROCESSLIST; /* shows all connections */
mysql> KILL <process_id>; /* for EVERY process you want to kill */


To make this process much simpler I wrote a little bash script that will kill all connections for a given MySQL user.

#!/bin/bash
# Kill all MySQL connections for a given user
# Ben Dowling - Apr 2009
# www.coderholic.com

user="root" # MySQL super user
pass="" # Above user's password
kill_user="myapp" # User who you want to kill all the connections for

# Get the connection IDs for all connections from the user $kill_user
processList=$(mysql -u$user -p$pass -s -e "SHOW FULL PROCESSLIST" | grep $kill_user | awk '{print $1}')
# Loop through all the connection IDs and kill the connection
for id in $processList
do
       result=$(mysql -u$user -p$pass -s -e "KILL $id")
       echo $result
done

The MySQL superuser account name and password, along with the user you want to kill all the connection for, are embedded within the script. It would be quite simple to pass these in as command line arguments if you wanted to though.

What I came up with was a LoggedPDO class that extends PHP’s PDO class. If you’re already using PDO in your project you only need to change a single line of code to enable database query logging:

// Standard PDO...
$db = new PDO($dsn, $username, $password);
// ...should be changed to
$db = new LoggedPDO($dsn, $username, $password);

And then all your database queries will be logged, timed and all information printed out at the end of each page. My code also includes a decorator for PDOStatements so any prepared statements that get executed also get logged. Below is a small sample of the code in use:

$db = new LoggedPDO("mysql:dbname=test;host=localhost", "root");
$results = $db->query("SHOW TABLES");
$tables = $results->fetchAll(PDO::FETCH_COLUMN);
foreach($tables AS $table) {
	$statement = $db->prepare("DESCRIBE :table");
	$statement->bindParam(":table", $table, PDO::PARAM_STR);
	$results = $statement->execute();
	$columns = $statement->fetchAll(PDO::FETCH_ASSOC);
}

Which would output:

Full code for the both the LoggedPDO and LoggedPDOStatement classes is below:

/**
* Extends PDO and logs all queries that are executed and how long
* they take, including queries issued via prepared statements
*/
class LoggedPDO extends PDO
{
	public static $log = array();

	public function __construct($dsn, $username = null, $password = null) {
		parent::__construct($dsn, $username, $password);
	}

	/**
	 * Print out the log when we're destructed. I'm assuming this will
	 * be at the end of the page. If not you might want to remove this
	 * destructor and manually call LoggedPDO::printLog();
	 */
	public function __destruct() {
		self::printLog();
	}

	public function query($query) {
		$start = microtime(true);
		$result = parent::query($query);
		$time = microtime(true) - $start;
		LoggedPDO::$log[] = array('query' => $query,
				                  'time' => round($time * 1000, 3));
		return $result;
	}

	/**
	 * @return LoggedPDOStatement
	 */
	public function prepare($query) {
		return new LoggedPDOStatement(parent::prepare($query));
	}

	public static function printLog() {
		$totalTime = 0;
		echo '<table border=1><tr><th>Query</th><th>Time (ms)</th></tr>';
		foreach(self::$log as $entry) {
			$totalTime += $entry['time'];
			echo '<tr><td>' . $entry['query'] . '</td><td>' . $entry['time'] . '</td></tr>\n';
		}
		echo '<tr><th>' . count(self::$log) . ' queries</th><th>' . $totalTime . '</th></tr>\n';
		echo '</table>';
	}
}

/**
* PDOStatement decorator that logs when a PDOStatement is
* executed, and the time it took to run
* @see LoggedPDO
*/
class LoggedPDOStatement {
	/**
	 * The PDOStatement we decorate
	 */
	private $statement;

	public function __construct(PDOStatement $statement) {
		$this->statement = $statement;
	}

	/**
	* When execute is called record the time it takes and
	* then log the query
	* @return PDO result set
	*/
	public function execute() {
		$start = microtime(true);
		$result = $this->statement->execute();
		$time = microtime(true) - $start;
		LoggedPDO::$log[] = array('query' => '[PS] ' . $this->statement->queryString,
				                  'time' => round($time * 1000, 3));
		return $result;
	}
	/**
	* Other than execute pass all other calls to the PDOStatement object
	* @param string $function_name
	* @param array $parameters arguments
	*/
	public function __call($function_name, $parameters) {
		return call_user_func_array(array($this->statement, $function_name), $parameters);
	}
}

This is one approach to the failed contact form problem, but it isn’t going to help much if the problem is actually with your email address. For example, you may have exceeded your disk quota, or you might have a badly configured mail server.

Another problem with the mail solution is that it requires the user to perform another action. They’ve gone to the trouble of filling out a contact form. If that fails they might not bother to send to send you an email too (although the suggested approach in the article does reduce the effort required).

An Alternative Approach

So what other approaches are there? What I do is store all contact form information in a database. Below is the SQL to create the basic messages table:

CREATE TABLE messages (
    id int auto_increment NOT NULL,
    sent timestamp NOT NULL,
    sender varchar(255) NOT NULL,
    message text,

    PRIMARY KEY(id)
);

Then when a contact form gets submitted you can store the information in the database before sending an email. Below is some sample PHP code, using PDO to access the database:

// Get the form data
$sender = $_POST['sender'];
$message = $_POST['message'];

// Add the message to the database
$query = "INSERT INTO messages(sender, message) VALUES (:sender, :message);
$stmt = $pdo->prepare($query);
$stmt->bindParam(':sender', $sender, PDO::PARAM_STR);
$stmt->bindParam(':message', $message, PDO::PARAM_STR);
$stmt->execute();

// Send the mail
mail("me@my.domain", "Contact Form Message", $message, "From: {$sender}");

All that is left to do is create an admin page that lists all messages, and allows us to delete them. We could even include a link to the admin page in the emails we send to ourselves. By using this approach we never lose the message content, even if the mail call fails, because they all get stored in the database. All we need to do is check the admin page every so often and we’ll soon see if there are any new messages there.

Of course, this approach doesn’t have to be used in isolation. We can combine it with the approach suggested in the DZone article, and present the user with a mail link if sending the mail does fail. If the user decides not to send a mail though, we haven’t lost anything.

Using a third party tool tedia2sql, you can automatically generate SQL schema for a number of databases, including Oracle, MySQL, Postgres, based on your database diagram. In this post I’ll describe exactly how.

The diagram

To create the database diagram we’ll be using Dia’s UML tools. Classes represent database tables, and associations between classes represent foreign key constrains.

In this post I’ll go over a simple example database with just three tables: Film, Film_Actor and Actor.

To create our database diagram we’re going to use Dia’s UML shapes. When you first start Dia it defaults to displaying the “Assorted” set of shapes, so you’ll need to change the drop down to “UML”

Then we can click on the UML class shape, which is at the top left.Once we’ve done that we should have a diagram that looks like the one below:

Double clicking on the class will bring up a properties window, which has several tabs. On the first “Class” tab we can set the name. For the first table that is “Film”. On the “Attributes” tab we can enter the rows that this table will have. Setting the attribute visibility to protected signifies that it is a primary key.

After adding all three tables and attributes we end up with a diagram like the one below:

At this point we could already generate a database schema, but it wouldn’t contain any foreign key contraints. If you’re not interested in adding them then you can skip straight to the generation part.

Foreign key contraints must be modelled using the UML aggregation tool: The line with the white diamond on one end. Select that tool, and then click on one table and drag and drop onto another table. When you let go of the mouse button a link should be drawn between the two. Double clicking on that line brings up a properties dialog.

Enter the foreign key in one end, and the row to which the foreign key referrs. The link between the Film_Actors table and the Films table is shown above, where the link is from film_id to id.

After adding a similar link between the Film_Actors and Actor table we end up with our complete diagram:

Generating the SQL

Now that our database diagram is compete we can generate the SQL commands to create the database. The command to generate the code for MySQL InnoDB is:

tedia2sql -i diagram.dia -o schema.sql -t innodb -f

If we want to generate SQL for a different database (such as Oracle) then we just need to change the -t argument. The generated SQL contains lots of comments, but the main sections are shown below:

-- Film
create table Film (
  id                        int not null,
  title                     varchar,
  year                      int,
  constraint pk_Film primary key (id)
) type = InnoDB ;

-- Actor
create table Actor (
  id                        int not null,
  name                      varchar,
  dob                       date,
  constraint pk_Actor primary key (id)
) type = InnoDB ;

-- Film_Actors
create table Film_Actors (
  film_id                   int not null,
  actor_id                  int not null,
  constraint pk_Film_Actors primary key (film_id,actor_id)
) type = InnoDB ;

alter table Film_Actors add constraint film_Actors_fk_Film_id
  foreign key (film_id)
  references Film (id)  ;
alter table Film_Actors add constraint film_Actors_fk_Actor_id
  foreign key (actor_id)
  references Actor (id)  ;

Other Dia tools

In this post I described how easy it is to generate SQL from a Dia diagram using tedia2sql. There are lots of other great third party tools to automatically generate output based on your diagram. The official Dia links page lists many of them.

#!/bin/bash
# Copy a remote database to a local one

REMOTE_HOST=''
REMOTE_USER=''
REMOTE_PWD=''
REMOTE_DB=''

LOCAL_USER=''
LOCAL_PWD=''
LOCAL_DB=''

mysqldump -h$REMOTE_HOST -u$REMOTE_USER -p$REMOTE_PWD --opt --compress $REMOTE_DB | mysql -u$LOCAL_USER -p$LOCAL_PWD $LOCAL_DB