Aleksandar Dragojevic, Dushyanth Narayanan, Ed Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, Miguel Castro Published in Symposium on Operating Systems Principles (SOSP), 2015
In this paper, we show that there is no need to compromise in modern data centers. We show that a main memory distributed computing platform called FaRM can provide distributed transactions with strict serializability, high performance, durability, and high availability.
Mark Russinovich, Edward Ashton, Christine Avanessians, Miguel Castro, Amaury Chamayou, Sylvan Clebsch, Manuel Costa, Cedric Fournet, Matthew Kerner, Sid Krishna, Julien Maffre, Thomas Moscibroda, Kartik Nayak, Olga Ohrimenko, Felix Schuster, Roy Schuster, Alex Shamis, Olga Vrousgou, Christoph M. Wintersteiger Technical Report, 2019
This paper present CCF, a framework to build premissioned confidential blockchains. CCF provides a simple programming model of a highly-available data store and a universally-verifiable log that implements a ledger abstraction. CCF leverages trust in a consortium of governing members and in a network of replicated hardware-protected execution environments to achieve high throughput, low latency, strong integrity and strong confidentiality for application data and code executing on the ledger.
Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J. Parkinson, Alex Shamis, Christoph M. Wintersteiger, David Chisnall Published in 2019 ACM SIGPLAN International Symposium on Memory Management, 2019
This paper presents snmalloc, a new point in the allocator/deallocator design space. Instead of thread-caching, we use lightweight lock-free message-passing to send batches of deallocations to the originating thread.
Alex Shamis, Matthew Renzelmann, Stanko Novakovic, Georgios Chatzopoulos, Aleksandar Dragojevic, Dushyanth Narayanan, Miguel Castro Published in ACM SIGMOD International Conference on Management of Data, 2019
This paper extends the design of FaRM — which provides strict serializability only for committed transactions — to provide opacity while maintaining FaRM’s high throughput, low latency, and high availability within a modern data center. It uses timestamp ordering based on real time with clocks synchronized to within tens of microseconds across a cluster, and a failover protocol to ensure correctness across clock master failures. Best Paper - Honorable Mention
Paul England, Henrique S Malvar, Eric Horvitz, Jack W Stokes, Cédric Fournet, Amaury Chamayou, Sylvan Clebsch, Manuel Costa, Shabnam Erfani, Kevin Kane, Alex Shamis Technical Report, 2020
Advances in graphics and machine learning algorithms and processes have led to the general availability of easy-to-use tools for modifying and synthesizing media. The proliferation of these tools threatens democracies around the world by enabling wide-spread distribution of false information to billions of individuals via social media platforms. One approach to thwarting the flow of fake media is to detect synthesized or modified media via the use of pattern recognition methods, including statistical classifiers developed via machine learning. While detection may help in the short-term, we believe that it is destined to fail as the quality of the fake media generation continues to improve. Within a short period of time, neither humans nor algorithms will be able to reliably distinguish fake versus real content. Thus, pipelines for assuring the source and integrity of media will be required—and will be increasingly relied upon. We propose AMP, a system that ensures authentication of a media contents source via provenance.
During my time at Microsoft, I have worked on several different distributed systems that the company uses internally. In this talk, I will describe two of them and also discuss the trade-offs and design choices that were made to build and run these systems. I will firstly talk about Microsoft Cosmos which is Microsoft’s internal big data framework that runs dryad-like jobs which are programmed using the SCOPE language. Then, I will describe FaRM which is a main-memory distributed transactional store that exploits new hardware trends such as RDMA and non-volatile memory to achieve performance that is several orders of magnitude faster than systems that offer similar capability but run using TCP. – LSDS - Seminar
FaRM is a main memory distributed computing platform that provides distributed transactions with strict serializability, high performance, durability, and high availability. FaRM’s transactional reads however, do not provide opacity. Transactions sometimes operate on inconsistent state before aborting, which can result in program crashes. Programmers using FaRM are required to introduce consistency checks, similar to the checks used in lock-free algorithms to prevent this. This is a major issue as it significantly complicates the programming model, even for the experience programmers that use FaRM to build production systems. To fix this problems, we have changed the FaRM protocols to provide opacity using Multiversion Concurrency Control (MVCC). – UK Systems Workshop
FaRM is an in-memory transactional object store for random-access latency-sensitive workloads, such as graphs and key-value stores. It provides high throughput and low microsecond latency by using remote direct memory access (RDMA) and novel transactional protocols. In addition to presenting FaRM, we have built A1, a graph store built on top of FaRM, and which is being adopted as the next generation graph platform targeting several scenarios in Bing, Office, and Cortana. – Microsoft Research
FaRM is an in-memory transactional object store for random-access latency-sensitive workloads, such as graphs and key-value stores. It provides high throughput and low microsecond latency by using remote direct memory access (RDMA) and novel transactional protocols. In this talk I present FaRM, discuss it’s history, and the the direction of the project. Finally, I show how FaRM is being used by Microsoft as a production service.
Transactions can simplify distributed applications by hiding data distribution, concurrency, and failures from the application developer. Ideally the developer would see the abstraction of a single large machine that runs transactions sequentially and never fails. This requires the transactional subsystem to provide opacity (strict serializability for both committed and aborted transactions), as well as transparent fault tolerance with high availability. As even the best abstractions are unlikely to be used if they perform poorly, the system must also provide high performance.
The Confidential Consortium Framework (CCF) is an open-source framework for building permissioned confidential applications. CCF uses replicated hardware-protected trusted execution environments to ensure confidentiality and integrity whilst providing high throughput and low latency. CCF exposes a transactional programming model atop a key-value store, and supports transparent programmable governance.