Papers I've read.

Technical discussions where a group of folks sit down and talk about interesting technical papers is probably one of the best ways to build a robust learning community. One of the first steps to do that is identifying some papers worth chatting about that can create lively discussion. Most of the suggestions below have come via the excellent list from Will Larson the author of An Elegant Puzzle

The must reads

Big Ball of Mud
or PDF Probably the best paper on system architecture in the real world ever.
Fielding's REST Paper
The full orignal can be found here. Its also worth reading Roy Fielding's Misappropriated REST Dissertation for context on how REST is often abused in the real world.
On Designing and Deploying Internet-Scale Services
The system-to-administrator ratio is commonly used as a rough metric to understand administrative costs in high-scale services. With smaller, less automated services this ratio can be as low as 2:1, whereas on industry leading, highly automated services, there are ratios as high as 2,500:1. Services that are operations friendly require little human intervention, and both detect and recover from all but the most obscure failures without administrative intervention. This paper summarizes the best practices accumulated over many years in scaling some of the largest services at MSN and Windows Live. This is a true checklist of how to design and evaluate large scale systems (almost like The Twelve Factor App wants to be for a checklist for operable applications).
CAP Twelve Years Later: How the Rules Have Changed
Eric Brewer posited the CAP theorem in the early 2000s, and twelve years later he wrote this excellent overview and review of CAP (which argues distributed systems have to pick between either availability or consistency during partitions), in part because: In the decade since its introduction, designers and researchers have used (and sometimes abused) the CAP theorem as a reason to explore a wide variety of novel distributed systems. The NoSQL movement also has applied it as an argument against traditional databases. CAP is interesting because there is not a "seminal CAP paper", but this article serves well in such a paper's stead. These ideas are expanded on in the Harvest and Yield paper below.
Harvest, Yield, and Scalable Tolerant Systems
This paper builds on the concepts from CAP Twelve Years Later, introducing the concepts of harvest and yield to add more nuance to the AP vs CP discussion. The harvest and yield concepts are particularly interesting because they are both self-evidence and very rarely explicitly used, instead distributed systems continue to fail in mostly undefined ways.
Searching for Build Debt: Experiences Managing Technical Debt at Google
This paper is an interesting cover of how to perform large-scale migrations in living codebases. Google software engineers are constantly paying interest on various forms of technical debt. Google engineers also make efforts to pay down that debt, whether through special Fixit days, or via dedicated teams, variously known as janitors, cultivators, or demolition experts. The paper describe several related efforts to measure and pay down technical debt found in Google's BUILD files and associated dead code. It also addresses debt found in dependency specifications, unbuildable targets, and unnecessary command line flags. These efforts often expose other forms of technical debt that must first be managed. Using broken builds as the running example, they break down their strategy to three pillars: automation, make it easy to do the right thing, and make it hard to do the wrong thing.
Dynamo: Amazon's Highly Available Key-value Store
This paper is a phenomenal introduction to eventual consistency, coordinating state across distributed storage, reconciling data as it diverges across replicas and much more.
Design patterns for container-based distributed systems
The move to containers-based deployment and orchestration has introduced a whole new set of vocabulary like sidecars and adapters, and this paper provides a survey of the patterns which have evolved over the past decade as microservices and containers have become increasingly prominent infrastructure components The "sidecar" term in particular likely originated in this blog post from Netflix, which is a worthy read in its own right.

The best of the rest

Hints for Computer System Design
Butler Lampson is an ACM Turning Award winner (among other awards), and worked at the Xerox PARC. This paper concisely summarizes many of his ideas around system design, and is a great read.
The Google File System
The inspiration for the ground breaking Hadoop File system.
MapReduce: Simplified Data Processing on Large Clusters
The MapReduce paper is an excellent example of an idea which has been so successful that it now seems self-evident. The idea of applying the concepts of functional programming at scale became a clarion call, provoking a shift from data warehousing to a new paradigm for data analysis, this paper was itself a major inspiration for Hadoop.
Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
The Dapper paper introduces a performant approach to tracing requests across many services, which has become increasingly relevant as more companies refactor core monolithic applications into dozens or hundreds of micro-services. Introduces the design of Dapper, Google's production distributed systems tracing infrastructure, and describes how the design goals of low overhead, application-level transparency, and ubiquitous deployment on a very large scale system were met. Dapper shares conceptual similarities with other tracing systems, particularly Magpie and X-Trace. However certain design choices were made that have been key to its success, such as the use of sampling and restricting the instrumentation to a rather small number of common libraries. The ideas from Dapper have since made their way into open source, especially in Zipkin and OpenTracing.
Kafka: a Distributed Messaging System for Log Processing
Apache Kafka has become a core piece of infrastructure for many internet companies. It's versatility lends it to many roles, serving as the ingress point to "data land" for some, a durable queue for others, and that's just scratching the surface. Not only a useful addition to your toolkit, Kafka is also a beautifully designed system. In particular, Kafka's partitions do a phenomenal job of forcing application designers to make explicit tradeoffs about trading off performance for predictable message ordering.
Wormhole: Reliable Pub-Sub to Support Geo-replicated Internet Services
In many ways similar to Kafka, Facebook's Wormhole is another highly scalable approach to messaging. In particular, note the approach to supporting lagging consumers without sacrificing overall system throughput.
Borg, Omega, and Kubernetes
While the individual papers for each of Google's orchestration systems (Borg, Omega and Kubernetes) are worth reading in their own right, this article is an excellent overview of the three. This article describes the lessons learned from developing and operating them.
Large-scale cluster management at Google with Borg
Borg has been orchestrating much of Google's infrastructure for quite some time (significantly predating Omega, although fascinatingly the Omega paper predates the Borg paper by two years). Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. This paper takes a look at Borg's centralized scheduling model, which was both effective and efficient, although it became increasingly challenging to modify and scale over time, inspiring both Omega and Kubernetes within Google (the former to optimistically replace it, and the later seemingly to commercialize their learnings, or at least prevent Mesos from capturing too much mindshare).
Omega: flexible, scalable schedulers for large compute clusters
Omega is, among many other things, an excellent example of the second-system effect, where an attempt to replace a complex existing system with something far more elegant ends up being more challenging than anticipated. In particular, Omega is a reaction against the realities of extending the aging Borg system. Perhaps also an example of worse is better once again taking the day.
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
This paper describes the design of Apache Mesos, in particular its distinctive two-level scheduler. Used heavily by Twitter and Apple, Mesos was for some time the only open-source general scheduler with significant adoption, and is now in a fascinating competition for mindshare with Kubernetes.
Raft: In Search of an Understandable Consensus Algorithm
and Paxos Made Simple. Rather complex papers on concensus algorithms. Raft is simpler and is used by etcd and influxdb among many others. Paxos is a deeply innovative concept, and is the algorithm behind Google's Chubby and Apache Zookeeper, among many others.
SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol
The majority of consensus algorithms focus on being consistent during partition, but SWIM goes the other direction and focuses on availability. Several distributed peer-to-peer applications require weakly-consistent knowledge of process group membership information at all participating processes. SWIM is a generic software module that offers this service for large scale process groups. The SWIM effort is motivated by the unscalability of traditional heart-beating protocols, which either impose network loads that grow quadratically with group size, or compromise response times or false positive frequency w.r.t. detecting process crashes. This paper reports on the design, implementation and performance of the SWIM sub-system on a large cluster of commodity PCs. SWIM is used in Hashicorp's software, as well as Uber's Ringpop.
Out of the Tar Pit
Out of the Tar Pit bemoans unnecessary complexity in software, and proposes that that functional programming and better data modeling can help us reduce accidental complexity (arguing that most unnecessary complexity comes from state). Certainly a good read. Reading it a decade later it's fascinating to see that neither of those approaches have taken off. Instead the closest "universal" approach to reducing complexity seems to be the move to numerous mostly stateless services. Which is perhaps more a reduction of local complexity, at the expense of larger systemic complexity, whose maintenance is then delegated to more specialized systems engineers.
Bigtable: A Distributed Storage System for Structured Data
One of Google's preeminent papers and technologies is Bigtable, which was an early (early in the internet era, anyway) NoSQL datastore, operating at extremely high scale and built on top of Chubby. From the SSTable design to the bloom filters, Cassandra inherits significantly from the Bigtable paper, and is probably rightfully considered a merging of the Dynamo and Bigtable papers.
Spanner: Google's Globally-Distributed Database
Where many early NoSQL storage systems traded eventual consistency for increased resiliency, building on top of eventually consistent systems can be harrowing. Spanner represents an approach from Google to offering both strong consistency and distributed reliability, relying in part on a novel approach to managing time.
Security Keys: Practical Cryptographic Second Factors for the Modern Web
Security keys like the YubiKey have emerged as the most secure second authentication factor, and this paper out of Google explains the motivations that lead to their creation, and the design that makes them work. They're also remarkably cheap.
BeyondCorp: Design to Deployment at Google
Building on the original BeyondCorp paper in 2014, this paper is slightly more detailed and benefits from two more years of migration-fueled wisdom. That said, the big ideas have remained fairly consistent and there is not much new relative to the BeyondCorp paper itself.
Availability in Globally Distributed Storage Systems
This paper explores how to think about availability in replicated distributed systems, and is a useful starting point for those of us who are trying to determine the correct way to measure uptime for their storage layer or any other sufficiently complex system. A simple definition of availability was in this case: A storage node becomes unavailable when it fails to respond positively to periodic health checking pings sent by our monitoring system. The node remains unavailable until it regains responsiveness or the storage system reconstructs the data from other surviving nodes. Often discussions of availability become arbitrarily complex so this is a good exmaple of simplicity.
Still All on One Server: Perforce at Scale
This paper is particularly good at considering the difficulties that companies run into scaling Git monorepos.
Source Code Rejuvenation is not Refactoring
This paper introduces the concept of "Code Rejuvenation", a unidirectional process of moving towards cleaner abstractions as new language features and libraries become available, which is particularly applicable to sprawling, older code bases. There are some strong echoes of this work in Google's ClangMR paper.

Papers I've read.

The must reads

The best of the rest

Additional amazing papers