Our MogileFS setup allows the disk space on each web server (normally unutilized) to form a cheap storage node, and make use of space that would otherwise go entirely unused.

On Monday 1/21 the database servers behind MogileFS paged with too many connections, which leads to Mogile going very slowly for a while, and sometimes requiring a restart of some of the nodes.

This database issue cascaded into us asking our Mogile client for item A, but receiving item B in response…

The problem turned out to be at the lowest level, that of the socket communication with Mogile. The request chain asked for item A, but since the database was responding very slowly, the “fread” waiting for the response timed out. Unfortunately, the timeout simply passes back up the chain as a “not found”. In the background, the data for the response to the request was buffered by TCP. We handled the error, told our daemon to check later for that item, and proceeded to ask for item B. When doing another “fread” for the response, we received the data from the original request. Since all was well, we continued requesting items and receiving a non-matching item in response.

In order to fix this on our end, we simply made the connection terminate when any of these timeouts occurs. Additionally, we’ll be adding application-level matching for these requests and responses (the data we store in Mogile will likely include the key so when we pull it back out, it can be check-summed against the requested key).

However, I wonder if it’d make sense for the MogileFS protocol itself to support this matching. It seems key that this type of problem not affect others, so I encourage you to check your client for such a problem, and I plan to file bugs against those who have the bug we found. We were using an old PEAR module as we sought a client with a non-GPL license for inclusion in our codebase. However, a cursory examination of the PHP extension we were coveting seems it might be subject to the same problem.

Other than this issue (and the large headache it caused), which mostly lies on the client side (unless I get any pickup on the protocol-level support above), MogileFS has proved to be a great asset to our infrastructure and I look forward to seeing it scale to much more data.

declare(ticks=1);

class DaemonRunner {
    public static function exec($className) {
        $argv = $_SERVER['argv'];
        $daemon = new $className($argv);
        $daemon->init();
        while (!$daemon->isDone()) {
            $daemon->run();
        }
        $daemon->shutdown();
    }
}

The first statement is a PHPism. This allows PHP to check for signals every 1 tick. A tick is simply a low-level PHP interpreter step. The way I’ve setup my daemon and signal handling by default, the PHP will finish its current iteration and then quit when receiving a signal. This ensures that the database, and whatever background stuff aren’t left in inconsistent states. As with anything, you’re free to modify, extend, or otherwise alter this behavior.

great for background/async processing of:

• photos
• movies
• reports
• imports of data

advantages:

• use the same codebase as your app
• use the same expertise on your team of programmers

challenges:

• signals
• one thread
• memory (this will have to be a whole separate article)
• starting and stopping

A typical Daemon class I use:

abstract class Daemon {
    protected $done = false;

    public function __construct($argv) {
    }
    public function init() {
        pnctl_signal(SIGTERM, array($this, "onSignal"));
        pnctl_signal(SIGINT, array($this, "onSignal"));

    }
    abstract public function run();
    public function shutdown() {}
    public function isDone() {
        return $this->done;
    }
    public onSignal($signal){
        switch ($signal) {
             case SIGTERM:
             case SIGINT:
                $this->done = true;
                 break;
             default:
                 // handle all other signals
        }
    }
}

As you can see, we’ve started to tackle some of the challenges here. The idea is to handle signals, specifically I usually care about SIGTERM (default when using ‘kill’ on *nix) and SIGINT (from pressing Ctrl-C). Both will cause the program to exit gracefully.

I’ll be posting the class I use to run Daemon’s tomorrow.