By drasch, on January 29th, 2008%
As any readers of the iContact blog may have learned, MogileFS has become an integral part of our infrastructure at iContact. Rather than store the bodies of messages in our database, we moved them to a quick&dirty storage method in our infrastructure long ago. This method was essentially a cheap WebDAV server and on each STORE command it would write to two backend servers and issue a GET from only one. About a month ago, we migrated most of our messages away from this older, less scalable method to our newer MogileFS backend.
Our MogileFS setup allows the disk space on each web server (normally unutilized) to form a cheap storage node, and make use of space that would otherwise go entirely unused.
On Monday 1/21 the database servers behind MogileFS paged with too many connections, which leads to Mogile going very slowly for a while, and sometimes requiring a restart of some of the nodes.
This database issue cascaded into us asking our Mogile client for item A, but receiving item B in response…
Continue reading MogileFS and race condition
Related posts
By drasch, on December 24th, 2007%
iContact Accomplishments
Team grew from 10 to 22
Down from 12 mail servers to 10!
From Four database clusters to six
Built and launched a Community
From 50-80+servers
Created and Grew Infrastructure team from 2 people to 9 people under Carl’s leadership
Alan and Geoff reoriented our development teams to create the Middleware and User-experience teams
Fixed 1048 bugs, (of 1683 total resolved)
3102 logged . . . → Read More: 2007 iContact and personal redux
By drasch, on September 20th, 2007%
In high-school, the teacher who taught a programming class and worked to write a Java-based voting system insisted they build in logging functionality, in spite of the iron-clad storage of data into text files. This discussion made an impression upon me because the best and worst thing about programming and computers is they do exactly . . . → Read More: stopgaps
By drasch, on December 17th, 2006%
I decided to make list of things of which I’ve been a part of accomplishing over the last twelve months at IntelliContact not any without assistance, and some for which I’m only happy to have had a role.
From 3500-7200+ customers
From one database cluster to four
From eight mail servers to 12
Creating and maturing the development processing inspired . . . → Read More: 2006 IntelliContact Accomplishments
|
|
|
