Today I’d like to start a series on website optimization. The order in which I will do things is not necessarily the best order to optimize a website. However it will be full of tips and explanations for every single aspect of website optimization. A better structured article will later be based off this series and put in Web:Extend’s documentation. We know, the documentation is pretty empty right now. We intend to fix this before we reach beta.

Context

I currently have a contract with a client whose website had huge performance issues. Our job was first to port his code to our framework, then optimize both the code and the server configuration. When he contacted me he had peaks of 130 registered users at the same time with loads between 15 and 30. Some pages could take up to 30 seconds to load.

The rewrite

So I did just that, port the code to use Web:Extend instead of various PEAR and non-pear libraries. At the same time we would remove the biggest problems in the database schema and in the queries. Let me get this straight: the website was slow not because it used PEAR; it was because the queries would sometimes select thousands of rows that would then be filtered by PHP, and because some operations generated a huge number of queries because JOINs were not used as often as it should have been. The use of the framework was motivated by the client’s will to move to PHP5 instead of the now deprecated PHP4, and this framework was chosen because I’m its main developer and I know it very well.

Once the code was fully ported and tested (fixing at the same time a few bugs in the original code), we put it on the production server. The load reduced to an average of 3 at peak times. That’s nice, but it can be better. I should point here that any user on the website produce an AJAX query (producing several SQL queries) every 10 seconds. That means at least 13 queries per second at peak times. It’s actually more, since that number does not include non-registered users.

The biggest point of slowness

The queries were not optimized yet.

We looked into it and decided to first try to find the biggest performance problem. The chat functionality was the biggest problem. Whenever someone posted a new chat message, a noticeable delay would happen. All chat messages since the launch of the website were stored in one table… containing about 1.5 million rows. The table was big and not properly indexed nor optimized (it’s still not perfectly optimized but I’ll get back on this point later). Most chat messages in this table were never used since they were read and long forgotten. We decided to archive the table periodically (deleting wasn’t an option). A copy of the table was thus created and a cron job would run the PHP code to archive all the read messages sent more than 15 days ago and all unread messages sent more than 3 months ago. This gave us a nice 150 thousand rows on the main table instead of the 1.5 million we started with.

This last point helped a lot; the slowness went away. But this didn’t fix everything. More queries needed help, and the database schema was still in need of huge changes.

EXPLAIN ALL QUERIES

This part is very technical. Do not hesitate to post a comment asking for answers or clarification. Thank you.

This brings us to the most interesting part of this post. I needed to analyze the SELECT queries used by the application. All the queries. My first thought was: is there a way to analyze everything in a completely automated way without having to copy/paste the queries or rewrite anything? If such a beast exist, I didn’t find it. So I wrote it myself.

This website uses MySQL. A good way to check whether your queries are optimized is to run an EXPLAIN statement. An EXPLAIN statement is basically just a SELECT query with “EXPLAIN ” put at the start. Instead of running the query, it will analyze it and return various informations about how MySQL will run it. This basically means that I just have to put “EXPLAIN ” at the start of all my SQL queries to get a full report on all the queries. And this is basically what I did, but without changing the application’s code.

In Web:Extend, we used the concept of sets to define the model part of applications. A set contains elements, just like a table contains rows, or like a path contains files, directories or links. You can retrieve either “exactly 1″ or “n (with n>=0)” elements from a set. Let’s take an example: the users set. This set have one user with nickname "essen", and 42 users with the "invisible" option activated. In our framework, "essen" is an individual element while the "invisible" users are a subset of the users set. Mathematically, "essen" is both an individual element and a subset, but for the sake of simplicity we consider it to be nothing more than an individual element.

Web:Extend defines base classes for both a set and an individual element. All operations performed on a set (e.g. all SELECT statements, along with the UPDATE and DELETE operations potentially affecting n rows) are written in the set class; all operations performed on an individual element (e.g. INSERT, UPDATE, DELETE affecting only 1 row) are written in the element class. When a set returns elements, it creates the correct instance for the element associated with the given set.

You probably understood by now that all SELECT queries are stored in only one type of class, the set classes, and that these classes do not contain any overhead related to the handling of individual elements. In other words, we’re free to instanciate them at will and call its methods without any worry. Most methods are of the type count, fetch or fetchAll, which respectively count the number of elements in the set (or in a subset), retrieve exactly 1 element and retrieve n elements.

All we have to do now is put an “EXPLAIN ” at the start of all these queries. Now that we saw how it works, it should be quite easy, right? As long as we resolve the following problems:

1. When fetching exactly one element, the model will return exactly one row and not the result of the query containing all the rows returned by the EXPLAIN query.
2. When counting rows, the row is fetched from the database and the value returned directly.

Let’s take this code for example:

class demoUsersSet extends weeDbSet
{
	protected $sModel = 'demoUsersModel';

	/**
		@return int The number of users.
	*/

	public function count()
	{
		$a = $this->getDb()->query('
			SELECT COUNT(*) AS c FROM users
		')->fetch();

		return $a['c'];
	}

	/**
		@param $iUserId The user identifier.
		@return demoUsersModel The requested user.
	*/

	public function fetch($iUserId)
	{
		return $this->getDb()->query('
			SELECT * FROM users WHERE user_id=? LIMIT 1
		', $iUserId)->rowClass($this->sModel)->fetch();
	}

	/**
		@return weeDatabaseResults All the users.
	*/

	public function fetchAll()
	{
		return $this->getDb()->query('
			SELECT * FROM users
		')->rowClass($this->sModel);
	}
}

We have the array access in count and fetch methods to bypass. We are going to bypass them by writing weeDatabaseResult child classes that will always return $this for both array access and fetch methods. Let’s call the resulting class weeExplainSQLResult (no pun intended).

This won’t be enough, however; we still need to tell the database driver used by the model to return “EXPLAIN” result objects instead of normal results. We can do this simply by extending the database driver classes, make the required changes, and tell the model classes to use that driver for their queries. Extending was easy for both MySQL and PgSQL.

We finished writing the classes that will bypass the execution of all the queries in the model and execute EXPLAIN queries instead.

All SELECT queries explained

Let’s try it!

$oDb = new weeExplainMySQLDatabase(array(
	'host'		=> 'localhost',
	'user'		=> 'wee',
	'password'	=> 'wee',
	'dbname'	=> 'wee_tests',
));

$oSet = new demoUsersSet;
$oSet->setDb($oDb);

foreach ($oSet->count() as $aExplainRow)
	var_dump($aExplainRow);

foreach ($oSet->fetch(42) as $aExplainRow)
	var_dump($aExplainRow);

foreach ($oSet->fetchAll() as $aExplainRow)
	var_dump($aExplainRow);

When ran, this code will output this:

array(10) {
  ["id"]=>
  string(1) "1"
  ["select_type"]=>
  string(6) "SIMPLE"
  ["table"]=>
  NULL
  ["type"]=>
  NULL
  ["possible_keys"]=>
  NULL
  ["key"]=>
  NULL
  ["key_len"]=>
  NULL
  ["ref"]=>
  NULL
  ["rows"]=>
  NULL
  ["Extra"]=>
  string(28) "Select tables optimized away"
}
array(10) {
  ["id"]=>
  string(1) "1"
  ["select_type"]=>
  string(6) "SIMPLE"
  ["table"]=>
  string(5) "users"
  ["type"]=>
  string(5) "const"
  ["possible_keys"]=>
  string(7) "PRIMARY"
  ["key"]=>
  string(7) "PRIMARY"
  ["key_len"]=>
  string(1) "4"
  ["ref"]=>
  string(5) "const"
  ["rows"]=>
  string(1) "1"
  ["Extra"]=>
  string(0) ""
}
array(10) {
  ["id"]=>
  string(1) "1"
  ["select_type"]=>
  string(6) "SIMPLE"
  ["table"]=>
  string(5) "users"
  ["type"]=>
  string(3) "ALL"
  ["possible_keys"]=>
  NULL
  ["key"]=>
  NULL
  ["key_len"]=>
  NULL
  ["ref"]=>
  NULL
  ["rows"]=>
  string(1) "3"
  ["Extra"]=>
  string(0) ""
}

By applying this on all the queries of our application we will get the complete analyze of all our SELECT queries, available for review and optimization to anyone working on the project. These “EXPLAIN” classes will allow us to test the results of changes on the database over all the queries rather than looking at them one by one and possibly missing things by making changes that affect a query we already looked at and fixed.

Feel free to use, improve or adapt this method in your projects. If you do, please let us know about it so we can use your improvements.

Conclusion

We’ll take a break from the query optimization in the next post of this series to talk about Simple is Hard and its implications for our framework, our thoughts about it and what are our plans to improve performance as a whole in the framework. I’ll only say for now that our framework performs really, really well. 3rd part will talk about query optimization and the importance of database design, highlighting errors and explaining good practices.

Thanks for reading, see you soon.

Disclaimer: If you feel like exploring the framework then have fun, but remember that the documentation is still lacking and that some parts of the API may change before we reach beta. We advise you to wait for the beta release before trying to use it, unless you feel adventurous or just plain curious.