The Derecho Project

Derecho is an open-source C++ distributed computing toolkit that provides strong forms of distributed coordination and consistency at RDMA speeds. Its goal is to be a library for building high-speed replicated systems on RDMA networks, with a simple and flexible interface that is much easier to use than standard RDMA drivers. The system is complete and in use by other projects (notably Cornell’s Cascade and Cascade DDS efforts). Derecho was created by a team that includes Jonathan Behrens (@fintelia), Sagar Jha (@sagarjha), Matthew Milano (@mpmilano), Edward Tremel (@etremel), and Weijia Song (@songweijia), as well as Professors Ken Birman and Robbert van Renesse.

Assumptions

Derecho aims at developers of new microservices that would run in cluster or cloud-hosted settings, with a good quality of resources (our experiments tend to be on dedicated machines, with RDMA NICs, and without much contention on the particular links the RDMA traffic travels over, although we have done some work in less performant environments). We aren’t trying to replace Zookeeper, the popular system configuration tool, and you might want to consider using it if you can already see that Derecho won’t match your needs.

Derecho itself has C++ APIs, but you can also create a library in which you write stub methods that don’t have type parameters, but that turn around and relay incoming requests to Derecho’s typed generic methods. By doing this and creating a DLL, you can load (via “using” or “import” statements) the DLL from languages like Java, Python or C# and then call into Derecho. Just keep in mind that if you are passing some form of managed data (if the language is garbage collected), you need to tell the garbage collector not to move the data you are passing to us while a call is active. There are always ways to do that (search for “pin memory” in the documentation for your favorite language).

You can configure Derecho to run over TCP instead of RDMA, and if you do that, could develop services on less beefy platforms with some degree of virtualization. But read more to make sure that your development and testing goals match our system assumptions. Derecho can run in a virtualized setting over TCP, but wouldn’t work well with substantial scheduling delays, very slow links, high rates of failures or membership changes, and the list goes on. We don’t recommend using Derecho for purposes very different from what we had in mind when we designed it, and we won’t be able to help if you try to do something really bizarre and it doesn’t work.

Many of our assumptions were based on the idea that one primary use case for the system would be to create new microservices for use in Azure’s IoT Edge (or the Azure Intelligent Edge: the IoT Edge is a cluster for potentially disconnected uses, and the Intelligent one is the outer tier of the Azure cloud). Azure IoT is mostly an Ubuntu environment, and we actually do our own testing on Ubuntu, although we should be able to run on other Linux platforms and perhaps directly on Windows too – we haven’t tried that out. We suggest that you play with the system exactly as we intended it, and gain some experience, before trying anything that pushes the envelope even in the most minor ways.

For the highest performance on genuine RDMA or similar interconnects, your code needs to avoid any form of copying or locking (we really mean this). Even a single copying operation might easily be 10x slower than our peak data replication speeds. We don’t have any magic ideas for how you would achieve this, but do keep it very much in mind. Even with C++, even using the most modern features, it is very easy to end up with lots of locks and lots of copying on the critical paths. Scheduling delays would kill performance too, or paging delays. Thus even common ideas like using multiple cores with multiple parallel threads can compromise performance – not “will” because there are ways to make things work, but “can”, because without care, you’ll have slow events on the critical paths that turn out to be performance-limiting, and sadly, those will absolutely kill peak performance. So we welcome users, but you’ll need to become fairly sophisticated to achieve the top levels of performance of which Derecho is capable.

Source Code

You can download the source code for Derecho from our main repository at https://github.com/derecho-project/derecho/. The README hosted at that repository provides installation and usage instructions.

Documentation

The GitHub README at the derecho repository contains a brief guide to setting up and running a distributed service using Derecho. You can also read it here. In addition, most of the developer-facing code in Derecho is documented with Doxygen comments, and the generated Doxygen HTML is hosted at this website for your convenience. We hope to publish a more detailed user guide on this website soon.