Our MogileFS setup allows the disk space on each web server (normally unutilized) to form a cheap storage node, and make use of space that would otherwise go entirely unused.

On Monday 1/21 the database servers behind MogileFS paged with too many connections, which leads to Mogile going very slowly for a while, and sometimes requiring a restart of some of the nodes.

This database issue cascaded into us asking our Mogile client for item A, but receiving item B in response…

The problem turned out to be at the lowest level, that of the socket communication with Mogile. The request chain asked for item A, but since the database was responding very slowly, the “fread” waiting for the response timed out. Unfortunately, the timeout simply passes back up the chain as a “not found”. In the background, the data for the response to the request was buffered by TCP. We handled the error, told our daemon to check later for that item, and proceeded to ask for item B. When doing another “fread” for the response, we received the data from the original request. Since all was well, we continued requesting items and receiving a non-matching item in response.

In order to fix this on our end, we simply made the connection terminate when any of these timeouts occurs. Additionally, we’ll be adding application-level matching for these requests and responses (the data we store in Mogile will likely include the key so when we pull it back out, it can be check-summed against the requested key).

However, I wonder if it’d make sense for the MogileFS protocol itself to support this matching. It seems key that this type of problem not affect others, so I encourage you to check your client for such a problem, and I plan to file bugs against those who have the bug we found. We were using an old PEAR module as we sought a client with a non-GPL license for inclusion in our codebase. However, a cursory examination of the PHP extension we were coveting seems it might be subject to the same problem.

Other than this issue (and the large headache it caused), which mostly lies on the client side (unless I get any pickup on the protocol-level support above), MogileFS has proved to be a great asset to our infrastructure and I look forward to seeing it scale to much more data.

• Team grew from 10 to 22
• Down from 12 mail servers to 10!
• From Four database clusters to six
• Built and launched a Community
• From 50-80+servers
• Created and Grew Infrastructure team from 2 people to 9 people under Carl’s leadership
• Alan and Geoff reoriented our development teams to create the Middleware and User-experience teams
• Fixed 1048 bugs, (of 1683 total resolved)
• 3102 logged hours of development time vs. 2255 hours of development time in 2006
• 2393 Maint Tickets vs. 1194 in 2006
• 1141 Systems/Helpdesk Tickets vs. 1137 in 2006 (amazing, a difference of only 4 tickets)
• 33k lines of code added to the Publisher (170k lines of core code of the Publisher vs 147k at the end of 2006)
• 71k LOC created in the Community (71k LOC in the Community vs. 266 at the end of 2006)
• 1778 commits to the Publisher tool (that’s a lot of code reviews)

At home

• We’ve taken 3001 photos totaling 3.6 GB
• We’ve taken 327 movies totaling 9.4 GB
• I’ve archived 1180 emails totaling 263 MB (theoretically, I read most of these)
• I’ve Traded 9191 Instant messages
• I’ve written 25 blog entries (up until this one)
• We’ve helped at least 22 dogs find new homes (probably many more, but I’m counting those I have photos of)
• DRNA spent over $8,528 vetting these dogs
• Rented 103 movies from Netflix (approximately 200 hours of our year)

We were reminded of this lesson at iContact. Just because your software ‘should’ never do something, doesn’t mean you shouldn’t make these rules explicit. An emergency bugfix to our queues at iContact had a trojan horse to allow messages from Client A to be matched with subscribers from Client B. Clearly this is bad, and now we’re putting a patch in to detect and throw exceptions if any situation like this occurs. This is, however, a great example of where things that should never happen are still worthy of writing software to prevent. It’s not shameful, it’s insightful to admit that as a programmer we are human and don’t understand as much as we might like.

• From 3500-7200+ customers
• From one database cluster to four
• From eight mail servers to 12
• Creating and maturing the development processing inspired by good things from Scrum, Extreme Programming, and Flickr (credit to Geoff and Alan)
• One-click deployment tool
• Bug triage – bugs find their way to solution instead of hanging in purgatory forever
• Grown from 20 to 50 servers
• Two four-person development teams
• From 1.5 to 2.5 Systems people
• Event-based Billing system (almost all credit here goes to Michael Best)
• Got rid of Albatross upgrade code
• Two mailhandlers in two locations
• Agnification (web servers can serve up from any DB cluster)
• Message desmurfing interface (credit to Vaughn)
• RSS feeds
• Public Newsletter Archive
• Role-based servers (credit to Jay)

• Mac/Yahoo Desktop Widget
• Cleanup of old files in codebase
• Consistent mail sending for messages other than broadcasts
• Firefox/Thunderbird Extension for showing message stats
• Refactoring message sending to show customers progress in sending and allow support team to ‘finish’ messages that get stuck

Where do RSS feeds go? They are a publication method, so maybe they belong under Send. But, they are targets when broadcasting a message (like lists) so maybe they belong under My Contacts. Maybe they are something else that you Create, Send, and Track therefore they go on the Create tab with hints about their status on the Send and Track tabs.

The answer may seem obvious to you, but it’s likely a different answer from that of your friends.

Most importantly, I welcome such a discussion and challenge. A victory is ours when our interface brings the simplification to a necessity rather than a desire as we improve IntelliContact.

[tags]intellicontact, ui, design, human factors[/tags]

Two of the issues that bit us this past week would have been caught within 5 minutes by tests we placed near the top of our priority list at the beginning of this sprint (8/7) and are still scheduled to be completed by the close of the sprint.

On the up side, we’ve had only a handful of minor issues related to our 4.0 release. The worst problems have been a few features we didn’t bring over from the 3.0 interface such as paying for additional sends above and beyond those included with your account.

[tags]triage, qa, intellicontact[/tags]

The challenge comes in distilling the feedback and determining the value of what has been said and suggested, and what hasn’t been said. For example, if everyone talks about the size of the new icons (which are intentionally quite large) does it mean that they thought most everything else was great? Or does it mean they stopped looking after they were scared away by the large icons?

The triage process as will begin next week where we decide what of the feedback gets rolled into the software now, what goes into the product backlog for later, and what gets ‘held’ for further input, suggestion, or interest.

Meanwhile, the development teams have been hard at work playing with buzzwords like RSS and REST.

[tags]intellicontact, beta, rss, rest, scrum[/tags]