mikepk.com about current projects contact

Server Outages

July 17, 2007

It's been a long time since I've had to write a server update, we've taken the reliability of our back end *very* seriously and I'd like to think people have noticed. Other than the occasional network hiccup, our system has been extremely solid for many months now. Unfortunately we've been in full on firefighting mode for the past couple of days as a a set of interdependent code bugs have reared their ugly heads and have been bringing down pieces of our server infrastructure at random intervals (since about Friday). We have significant levels of redundancy so most of these shouldn't have resulted in serious downtime. The past 6 hours, however, have seen the problem escalate, bringing down multiples of our machines and causing Grazr to fail to load for longer stretches of time. I apologize for anyone's site that's been affected. We're still working on it now and I promise we will get this resolved and continue to build out our system.