Redundant and Fault Tolerant PHP Session Storage

As a new PHP application grows from a single server the database is usually the bottleneck. When the original server begins to struggle the database is moved to a dedicated machine or two and perhaps read-only slaves are added (this may require some application code changes -  MySQL Proxy offers a neat alternative with read/write connection splitting). These are both relatively trivial and effective solutions until application traffic grows and the web server becomes the bottleneck.

RESTful Statelessness

HTTP conforms to the principle of REST, which require a that state (i.e. a small amount of session data) "is kept entirely on the client" (original PhD). Adopting this precept improves the fault tolerance and scalability of a a load-balanced web server architecture: as the user's session is not stored on a single server, any server can fail with minimum impact. The user's next request hits the load balancer and is routed to an active server, where their session continues from its last saved state. This configuration can be scaled easily using many identical machines, and it eliminates any individual web server as single points of failure.

If a PHP application has deeply embedded usage of the $_SESSION superglobal, removing state is difficult. Instead removing the dependency between a user's session data and the single server it's stored on achieves the same fault tolerance. By changing  PHP's session handler to write the serialised session to shared storage state is still being stored by the application, but it has the properties of statelessness so any server can handle the request. This is not true RESTful statelessness, in which the small amounts of user data such as a primary key are stored client-side in cookies or headers, but is a suitable alternative.

PHP's default session storage mechanism uses file on the local filesystem. When the session starts, PHP locks the session file with the operating system's flock() (file lock) call, which queues any subsequent requests until the current request has called sessionwriteclose(), which releases the lock. This prevents race conditions between fast or concurrent requests (likely in AJAX rich applications) that could result in unpredictable behaviour. It's also important that all of a session's locks are dropped when a connection to the session store is closed in case the application crashes without releasing the lock. These essential behaviours are not replicated by some of the more frequently touted alternatives.

Memcached

PHP's memcached extension makes it trivial to plug memcached into PHP's session handler. It's possible to run a pool of memcached servers, however it is a RAM-only cache. This means that when a server goes down its data is lost, moving the single point of failure from the web server's filesystem to the cache server's volatile RAM.

There is generally less RAM available than disk, increasing the chance of cache overflow (old but valid data being dropped in favour of newer data under low memory conditions). It's possible to implement atomic increments for simple locks, although this doesn't provide request queueing. Since version 3.0.4 (which is still beta) the memcached extension supports session locking and node mirroring, so although it's currently not robust there is future potential.

DRBD

DRBD in dual-primary mode provides a redundant, drop-in replacement for the local filesystem. By mounting a clustered filesytem such as OCFS onto DRBD the only configuration change is updating the session storage path. Although DRBD is robust it can not currently be expanded to more than two nodes, and configuring a heartbeat to enforce failover can be complicated.

There are other flavours possible, including GlusterFS (which supports flock() natively but has significant overhead for small files) and NFS (with an inability to release the locks of crashed clients until they've reconnected and problematic lock recovery), but they introduce substantial complexity - something the default session handler gracefully avoids.

Sharedance

Sharedance mimics PHP's filesystem locking on a remote server, keeping the file descriptor open until the client closes the session or the connection drops. The PHPDance library supports writing to a pair of  servers (although this could easily be modified to add further redundancy) and fails-over if the first server is unreachable. Because this extension writes sequentially but reads from only a single server a lock is lost when the primary server fails, although the data is safe on the other server.

Both Sharedance and PHPDance are old projects; this is a good sign of their stability but as they are not actively maintained any newly discovered bugs are likely to be the discoverer's responsibility. Here's a configuration guide.

NoSQL

Many NoSQL solutions regard record locking and atomicity as overhead. MongoDB has implemented atomic writes which facilitates application-level locking, but there is no native queuing. NoSQL datastores rarely requires authentication by default, which could leave session data vulnerable to manipulation or reading if the local network is not secured.

Corey Ballou presents a partial solution that may be satisfactory for low traffic or non-critical session handling on his blog.

Hazelcast

Hazelcast is an open source DHT - a decentralised, fault-tolerant, massively scalable key-value store. It supports locking and queueing, authentication, inter-node encryption, and can speak the memcached protocol. All this would make Hazelcast the perfect distributed session store, however the memcache protocol doesn't support the concept of locking, so the PHP memcached extension can't be used.

Hazelcast exposes a RESTful API that doesn't currently support locking either. Until the API is upgraded or a native PHP extension is released Hazelcast remains tantalisingly out of reach.

MySQL

Web applications already have an authenticated connection to a database in the cluster which (if required) is highly available. Almost all relational databases can perform read locking comparable to that of a filesystem, can queue requests for a locked resource and can be tuned to serve the data they store quickly. MySQL fulfils all these prerequisites.

Using a database server as a sesson store slightly increases load - concurrent requests to the locked resource result in multiple wait locks. Persistent database connections should be avoided so the server can release locks in case of an application crash.

Conclusion

Volatile storage may be suitable for session data if your application can repopulate the session in case of failure (if you don't store the session key this would also require the user to log in again), and memcached v3 offers some promising redundancy features. Hazelcast is the most fault-tolerant and easily scalable solution, but is yet to integrate sufficiently with PHP to be viable.

For critical systems only Sharedance and MySQL effectively mimic the local filesystem's functionality. Sharedance is almost an exact functional clone of the default storage mechanism, but the PHPDance interface has clunky redundancy (sequential writes and constant retries of the primary server, even in case of failure) and the project is not actively maintained. MySQL is able to duplicate the default session storage behaviour, offers various levels of redundancy that are abstracted from the session handler's implementation, and is likely to be already available to the majority of web applications.

In the next post I will write a fault tolerant MySQL session handler optimised for high traffic volume.