Some of the computer science behind the internet giants
I recently became interested in the structure of large internet systems but did not get to take a deep dive until today. I followed a bunch of links, ended up wiki-ing NoSQL and landed on a couple of excellent articles that explain the architectural concepts underlying some of the infrastructure technology deployed by internet giants like Amazon, Google, Facebook, eBay and Twitter.
Chirag Mehta is the pellucid writer of an excellent blog cloud computing and his March 5 article NoSQL is Not SQL and that’s a Problem is a good primer on the complexities of implementing internet-level distributed systems. Many, if not most, concepts of distributed systems were well understood even as early as 20 years ago but Chirag’s article sheds light on the issues faced by commercial internet distributed computing implementations and their approaches to dealing with well understood problems.
For a deeper dive into fundamental concepts consider this excellent writeup on Brewer’s CAP Theorem by Julian Browne, who I would argue is an even better writer given his ability to explain core technical concepts in simple language. This article can be followed by anybody with some systems-level (probably not with just programming alone) experience and explains Berkeley professor Eric Brewer’s CAP Theorem and its proof.
These two articles were tremendously helpful for understanding the conceptual domain, historical context and next level of technical issues surrounding the development of technologies such as Hadoop, BigTable, Map Reduce, PIG, Cassandra and others. I will write a technical overview of these technologies in a forthcoming post.
Update (3/15/10): tweet by julianbrowne: Interesting debate on Google Groups over whether CAP Theorem is false or not: http://bit.ly/ayUxLw 6:37 AM Mar 2nd from bit.ly.
