As any readers of the iContact blog may have learned, MogileFS has become an integral part of our infrastructure at iContact. Rather than store the bodies of messages in our database, we moved them to a quick&dirty storage method in our infrastructure long ago. This method was essentially a cheap WebDAV server and on each STORE command it would write to two backend servers and issue a GET from only one. About a month ago, we migrated most of our messages away from this older, less scalable method to our newer MogileFS backend.
Our MogileFS setup allows the disk space on each web server (normally unutilized) to form a cheap storage node, and make use of space that would otherwise go entirely unused.
On Monday 1/21 the database servers behind MogileFS paged with too many connections, which leads to Mogile going very slowly for a while, and sometimes requiring a restart of some of the nodes.
This database issue cascaded into us asking our Mogile client for item A, but receiving item B in response…
(more…)
Related posts
In high-school, the teacher who taught a programming class and worked to write a Java-based voting system insisted they build in logging functionality, in spite of the iron-clad storage of data into text files. This discussion made an impression upon me because the best and worst thing about programming and computers is they do exactly what they’re told to do. Unfortunately, this means that the error of lowly humans can easily seep into what the code or program asks the computer to do.
We were reminded of this lesson at iContact. Just because your software ’should’ never do something, doesn’t mean you shouldn’t make these rules explicit. An emergency bugfix to our queues at iContact had a trojan horse to allow messages from Client A to be matched with subscribers from Client B. Clearly this is bad, and now we’re putting a patch in to detect and throw exceptions if any situation like this occurs. This is, however, a great example of where things that should never happen are still worthy of writing software to prevent. It’s not shameful, it’s insightful to admit that as a programmer we are human and don’t understand as much as we might like.
Related posts
I decided to make list of things of which I’ve been a part of accomplishing over the last twelve months at IntelliContact not any without assistance, and some for which I’m only happy to have had a role.
- From 3500-7200+ customers
- From one database cluster to four
- From eight mail servers to 12
- Creating and maturing the development processing inspired by good things from Scrum, Extreme Programming, and Flickr (credit to Geoff and Alan)
- One-click deployment tool
- Bug triage - bugs find their way to solution instead of hanging in purgatory forever
- Grown from 20 to 50 servers
- Two four-person development teams
- From 1.5 to 2.5 Systems people
- Event-based Billing system (almost all credit here goes to Michael Best)
- Got rid of Albatross upgrade code
- Two mailhandlers in two locations
- Agnification (web servers can serve up from any DB cluster)
- Message desmurfing interface (credit to Vaughn)
- RSS feeds
- Public Newsletter Archive
- Role-based servers (credit to Jay)
Related posts
