Be careful with memcached keys

Using invalid characters in a memcached key can cause disasterous problems on your server, but first, the details...

Memcached's text protocol uses the space character as its delimiter. For example, the following line:

set mykey somevalue\r\n

will set the key "mykey" to the value "somevalue".

If you know that the protocol uses the space character as a delimiter it is very obvious that key's should never contain spaces as it will mess with the memcached parser. However, if you use the PHP PECL extension to access the memcached server you don't necessarily know the protocol as it is hidden from you and in none of the PHP documentation is it mentioned to not use spaces in your keys.

 

So what is the big deal though, surely it will simply not work when using an invalid key to get/set data?

Sadly, this is not the case. In fact, what will happen is that your PHP instance/script will halt completely and hold the Apache thread forever, causing Apache to start new threads/workers until your server runs out of resources and completely falls over.

 

How we discovered the problem.

We have an application that was never supposed to send keys that contained spaces in them in the first place but sadly, out of millions of requests received every few hours, some actually did contain spaces that managed to slip past our checks (thanks to a crappy reg-ex check mainly). I noticed our server would simply start running out of resources and the number of Apache processes kept climbing and climbing.

Debugging the problem took 2 full days as I didn't know what was causing the issue in the first place and at first couldn't replicate the problem on my test server. What i did know was that amongst one of the many features we added to our application, one was to use memcached for some additional memory storage we required caching and in the back of my mind I had a feeling that memcached had something to do with the problem. I just couldn't replicate it, not knowing that spaces in keys caused the problem or that there was keys with spaces in them.

Through process of elimination I eventually found the problem and I will show an example below on how you can replicate the problem yourself. Don't run this on a live server, but you can easily test it on your development machine. This simple PHP script will crash the PHP instance/Apache thread and is one of the very few things you can do to cause such a serious problem:

<?php
    $memcached = new Memcached();
    $memcached->addServer("localhost", 11211);
    echo "Setting key (and hanging instance)<br>";
    $memcached->set('spaced key', 'Some data', 60);
   
    echo "Your script will never reach this point and will never timeout either<br>";
    echo $memcached->get('spaced key');
?>

To recover from the problem simply restart your Apache server:

/etc/init.d/apache2 restart

 

How you can avoid the problem.

Avoiding the problem is very simply. One method is to convert all keys to valid keys before getting or setting values. As an example, you can use the function below:

function memcachedKey($key) {
    return preg_replace("/[^A-Za-z0-9]/", "", $key);
}

You can then use the function like this:

$memcached->set(memcachedKey('spaced key'), 'Some data', 60);

$memcached->get(memcachedKey('spaced key'));

Another method (but which I personally don't like) is to MD5 your keys before setting and getting them. The reason I don't like it of course is that an infinite number of key names may end up with the same names so use this MD5 hashing only when you know that you won't deal with an infinite number of different strings as keys.

 

Other things to check.

Make sure your key is never empty either as it will produce the same problem.

 

Conclusion

I sincerely hope that others that run into the same problem doesn't have to spend so much time trying to fix it and that they'd come upon this blog post before pulling their hair out. Google, now go forth and do your job. :)