Sitemap

Page Not Found

Page not found. Your pixels are in another canvas.

About me

CV

Publications

Sitemap

Talk map

Talks and presentations

Jupyter notebook markdown generator

No compromises: distributed transactions with consistency, availability, and performance

In this paper, we show that there is no need to compromise in modern data centers. We show that a main memory distributed computing platform called FaRM can provide distributed transactions with strict serializability, high performance, durability, and high availability.

Download here

CCF: A Framework for Building Confidential Verifiable Replicated Services

This paper present CCF, a framework to build premissioned confidential blockchains. CCF provides a simple programming model of a highly-available data store and a universally-verifiable log that implements a ledger abstraction. CCF leverages trust in a consortium of governing members and in a network of replicated hardware-protected execution environments to achieve high throughput, low latency, strong integrity and strong confidentiality for application data and code executing on the ledger.

Download here

snmalloc: A Message Passing Allocator

This paper presents snmalloc, a new point in the allocator/deallocator design space. Instead of thread-caching, we use lightweight lock-free message-passing to send batches of deallocations to the originating thread.

Download here

Fast General Distributed Transactions with Opacity

This paper extends the design of FaRM — which provides strict serializability only for committed transactions — to provide opacity while maintaining FaRM’s high throughput, low latency, and high availability within a modern data center. It uses timestamp ordering based on real time with clocks synchronized to within tens of microseconds across a cluster, and a failover protocol to ensure correctness across clock master failures.
Best Paper - Honorable Mention

Download here

A1: A Distributed In-Memory Graph Database

A1 is an in-memory distributed database used by the Bing search engine to support complex queries over structured data. The key enablers for A1 are availability of cheap DRAM and high speed RDMA (Remote Direct Memory Access) networking in commodity hardware. A1 uses FaRM as its underlying storage layer and builds the graph abstraction and query engine on top. The combination of in-memory storage and RDMA access requires rethinking how data is allocated, organized and queried in a large distributed system. A single A1 cluster can store tens of billions of vertices and edges and support a throughput of 350+ million of vertex reads per second with end to end query latency in single digit milliseconds. In this paper we describe the A1 data model, RDMA optimized data structures and query execution.

Download here

Multi-stakeholder media provenance management to counter synthetic media risks in news publishing

Three major global news organizations and a leading technology provider have come together to demonstrate a mechanism to tackle this problem that can operate at scale. The BBC, The New York Times Company, and CBC/Radio‐Canada in cooperation with Microsoft have developed a proposed open standards approach which can be used by large and small news organizations to protect the provenance of news stories in audio/visual/textual media.

Download here

AMP: Authentication of Media via Provenance

Advances in graphics and machine learning algorithms and processes have led to the general availability of easy-to-use tools for modifying and synthesizing media. The proliferation of these tools threatens democracies around the world by enabling wide-spread distribution of false information to billions of individuals via social media platforms. One approach to thwarting the flow of fake media is to detect synthesized or modified media via the use of pattern recognition methods, including statistical classifiers developed via machine learning. While detection may help in the short-term, we believe that it is destined to fail as the quality of the fake media generation continues to improve. Within a short period of time, neither humans nor algorithms will be able to reliably distinguish fake versus real content. Thus, pipelines for assuring the source and integrity of media will be required—and will be increasingly relied upon. We propose AMP, a system that ensures authentication of a media contents source via provenance.

Download here

IA-CCF: Individual Accountability for Permissioned Ledgers

We describe IA-CCF, a new permissioned ledger system that provides individual accountability. It can assign blame to the individual members that operate misbehaving replicas regardless of the number of misbehaving replicas or members. IA-CCF achieves this by signing and logging BFT protocol messages in the ledger, and by using Merkle trees to provide clients with succinct, universally-verifiable receipts as evidence of successful transaction execution. Anyone can audit the ledger against a set of receipts to discover inconsistencies and identify replicas that signed contradictory statements. IA-CCF also supports changes to consortium membership and replicas by tracking signing keys using a sub-ledger of governance transactions. IA-CCF provides strong disincentives to misbehavior with low overhead: it executes 47,000 tx/s while providing clients with receipts in two network round trips.

Download here

CTR: Checkpoint, Transfer, and Restore for Secure Enclaves

We present CTR, a software-only design to retrofit migration functionality into existing TEE architectures, whilst maintaining their expected security guarantees. Our design allows TEEs to be interrupted and migrated at arbitrary points in their execution, thus maintaining compatibility with existing VM and process migration techniques. By cooperatively involving the TEE in the migration process, our design also allows application developers to specify stateful migration-related policies, such as limiting the number of times a particular TEE may be migrated. Our prototype implementation for Intel SGX demonstrates that migration latency increases linearly with the size of the TEE memory and is dominated by TEE system operations.

Download here

Dropbear: Machine Learning Marketplaces made Trustworthy with Byzantine Model Agreement

We describe Dropbear, the first ML model marketplace that provides clients with strong integrity guarantees by combining results from multiple models in a trustworthy fashion. Dropbear replicates inference computation across a model group, which consists of multiple cloud-based GPU nodes belonging to different model owners. Clients receive inference certificates that prove agreement using a Byzantine consensus protocol, even under model heterogeneity and concurrent model updates. To improve performance, Dropbear batches inference and consensus operations separately: it first performs the inference computation across a model group, before ordering requests and model updates. Despite its strong integrity guarantees, Dropbears performance matches that of state-of-the-art ML inference systems: deployed across 3 cloud sites, it handles 800 requests/s with ImageNet models.

Download here

Personal experiences building Microsoft’s Cosmos and FaRM

Published: November 30, 2017

During my time at Microsoft, I have worked on several different distributed systems that the company uses internally. In this talk, I will describe two of them and also discuss the trade-offs and design choices that were made to build and run these systems. I will firstly talk about Microsoft Cosmos which is Microsoft’s internal big data framework that runs dryad-like jobs which are programmed using the SCOPE language. Then, I will describe FaRM which is a main-memory distributed transactional store that exploits new hardware trends such as RDMA and non-volatile memory to achieve performance that is several orders of magnitude faster than systems that offer similar capability but run using TCP. – LSDS - Seminar

FaRM and MVCC

Published: March 21, 2018

FaRM is a main memory distributed computing platform that provides distributed transactions with strict serializability, high performance, durability, and high availability. FaRM’s transactional reads however, do not provide opacity. Transactions sometimes operate on inconsistent state before aborting, which can result in program crashes. Programmers using FaRM are required to introduce consistency checks, similar to the checks used in lock-free algorithms to prevent this. This is a major issue as it significantly complicates the programming model, even for the experience programmers that use FaRM to build production systems. To fix this problems, we have changed the FaRM protocols to provide opacity using Multiversion Concurrency Control (MVCC). – UK Systems Workshop

Microsoft Research Faculty Summit 2018 - FaRM

Published: August 02, 2018

FaRM is an in-memory transactional object store for random-access latency-sensitive workloads, such as graphs and key-value stores. It provides high throughput and low microsecond latency by using remote direct memory access (RDMA) and novel transactional protocols. In addition to presenting FaRM, we have built A1, a graph store built on top of FaRM, and which is being adopted as the next generation graph platform targeting several scenarios in Bing, Office, and Cortana. – Microsoft Research

FaRM Distributed transactions with consistency, availability, performance, and opacity

Published: September 21, 2018

FaRM is an in-memory transactional object store for random-access latency-sensitive workloads, such as graphs and key-value stores. It provides high throughput and low microsecond latency by using remote direct memory access (RDMA) and novel transactional protocols. In this talk I present FaRM, discuss it’s history, and the the direction of the project. Finally, I show how FaRM is being used by Microsoft as a production service.

Fast General Distributed Transactions with Opacity

Published: July 04, 2019

Transactions can simplify distributed applications by hiding data distribution, concurrency, and failures from the application developer. Ideally the developer would see the abstraction of a single large machine that runs transactions sequentially and never fails. This requires the transactional subsystem to provide opacity (strict serializability for both committed and aborted transactions), as well as transparent fault tolerance with high availability. As even the best abstractions are unlikely to be used if they perform poorly, the system must also provide high performance.

The Confidential Consortium Framework

Published: February 21, 2020

The Confidential Consortium Framework (CCF) is an open-source framework for building permissioned confidential applications. CCF uses replicated hardware-protected trusted execution environments to ensure confidentiality and integrity whilst providing high throughput and low latency. CCF exposes a transactional programming model atop a key-value store, and supports transparent programmable governance.