We didn’t read the contract carefully enough. We assumed that Yahoo had some deal with … magnets … and, uh, electricity … where harddrives would never die once we signed. But it turns out that that was wrong and the ill-paired failure of two different drives within an hour of each other and some not-as-advertised raid controller flakiness in one of the database masters meant that we had to take the site down to reconfigure. Since it is down, we are running some power maintenance as well and plugging various servers into other slots. Should be back up well before 8pm pacific.
Fun fact: we have more than 200 harddrives in production in Flickr. Crazy! And 200 GB of RAM.
Oy. We were back up before 8pm, but had some serious database replication issues. Working on it now and will report back with an estimate soon.
Now it all works again. Yay!