Derecho  1.2
Distributed systems toolkit for RDMA

This is the main repository for the Derecho project. It unifies the RDMC, SST, and Derecho modules under a single, easy-to-use repository.


The code for this project is split into modules that interact only through each others' public interfaces. Each module resides in a directory with its own name, which is why the repository's root directory is mostly empty. External dependencies are located in the third_party directory, and will be pointers to Git submodules whenever possible to reduce code duplication.


Derecho is a library that helps you build replicated, fault-tolerant services in a datacenter with RDMA networking. Here's how to start using it in your projects.


Getting Started

Since this repository uses Git submodules to refer to some bundled dependencies, a simple git clone will not actually download all the code. To download a complete copy of the project, run

git clone --recursive

Once cloning is complete, to compile the code cd into the derecho-unified directory and run:

This will place the binaries and libraries in the sub-dierectories of Release. The other build type is Debug. If you need to build the Debug version, replace Release by Debug in the above instructions. We explicitly disable in-source build, so running cmake . in derecho-unified will not work.

To add your own executable (that uses Derecho) to the build system, simply add an executable target to CMakeLists.txt with derecho as a "linked library." You can do this either in the top-level CMakeLists.txt or in the CMakeLists.txt inside the "derecho" directory. It will look something like this:

add_executable(my_project_main my_project_main.cpp)
	  target_link_libraries(my_project_main derecho)

To use Derecho in your code, you simply need to include the header derecho/derecho.h in your *.h or *.cpp files:

Testing (and some hidden gotchas)

There are many experiment files in derecho/experiments that can be run to test the installation. To be able to run the tests, you need a minimum of two machines connected by RDMA. The RDMA devices on the machines should be active. In addition, you need to run the following commands to install and load the required kernel modules:

Depending on your system, some of the modules might not load which is fine.

RDMA requires memory pinning of memory regions shared with other nodes. There's a limit on the maximum amount of memory a process can pin, typically 64 KB, which Derecho easily exceeds. Therefore, you need to set this to unlimited. To do so, append the following lines to /etc/security/limits.conf:

where username is your linux username. A * in place of username will set this limit to unlimited for all users. Log out and back in again for the limits to reapply. You can test this by verifying that ulimit -l outputs unlimited in bash.

We currently do not have a systematic way of asking the user for RDMA device configuration. So, we pick an arbitrary RDMA device in functions resources_create in sst/verbs.cpp and verbs_initialize in rdmc/verbs_helper.cpp. Look for the loop for(i = 1; i < num_devices; i++). If you have a single RDMA device, most likely you want to start i from 0. If you have multiple devices, you want to start i from the order (zero-based) of the device you want to use in the list of devices obtained by running ibv_devices in bash.

To test if one of the experiments is working correctly, go to two of your machines (nodes), cd to Release/derecho/experiments and run ./derecho_bw_test 0 10000 15 1000 1 0 on both. The programs will ask for input.

The input to the first node is: