Represents the set of RDMA resources needed to maintain a two-way connection to a single remote node. More...

#include <lf.hpp>

Inheritance diagram for sst::_resources:

Collaboration diagram for sst::_resources:

Public Member Functions
	_resources (int r_id, char write_addr, char read_addr, int size_w, int size_r, int is_lf_server)
	Constructor Initializes the resources. More...

virtual	~_resources ()
	Destroys the resources. More...

	_resources (int r_index, char write_addr, char read_addr, int size_w, int size_r)
	Constructor; initializes Queue Pair, Memory Regions, and `remote_props`. More...

virtual	~_resources ()
	Destroys the resources. More...

Public Attributes
int	remote_id
	ID of the remote node. More...

struct fid_ep *	ep
	tx/rx completion queue More...

struct fid_mr *	write_mr
	memory region for remote writer More...

struct fid_mr *	read_mr
	memory region for remote writer More...

char *	write_buf
	Pointer to the memory buffer used for local writes. More...

char *	read_buf
	Pointer to the memory buffer used for the results of RDMA remote reads. More...

uint64_t	mr_lrkey
	key for local read buffer More...

uint64_t	mr_lwkey
	key for local write buffer More...

uint64_t	mr_rwkey
	key for remote write buffer More...

fi_addr_t	remote_fi_addr
	remote write memory address More...

struct fid_eq *	eq
	the event queue More...

int	remote_index
	Index of the remote node. More...

struct ibv_qp *	qp
	Handle for the IB Verbs Queue Pair object. More...

struct ibv_mr *	write_mr
	Memory Region handle for the write buffer. More...

struct ibv_mr *	read_mr
	Memory Region handle for the read buffer. More...

struct cm_con_data_t	remote_props
	Connection data values needed to connect to remote side. More...

Protected Member Functions
int	post_remote_send (struct lf_sender_ctxt *ctxt, const long long int offset, const long long int size, const int op, const bool completion)
	post read/write request More...

int	post_remote_send (const uint32_t id, const long long int offset, const long long int size, const int op, const bool completion)
	Post a remote RDMA operation. More...

Private Member Functions
void	connect_endpoint (bool is_lf_server)
	Connect the queue pair. More...

int	init_endpoint (struct fi_info *fi)
	Initialize resource endpoint using fi_info. More...

void	set_qp_initialized ()
	Initializes the queue pair. More...

void	set_qp_ready_to_receive ()
	Transitions the queue pair to the ready-to-receive state. More...

void	set_qp_ready_to_send ()
	Transitions the queue pair to the ready-to-send state. More...

void	connect_qp ()
	Connect the queue pairs. More...

Detailed Description

Represents the set of RDMA resources needed to maintain a two-way connection to a single remote node.

Definition at line 31 of file lf.hpp.

Constructor & Destructor Documentation

◆ _resources() [1/2]

sst::_resources::_resources	(	int	r_id,
		char *	write_addr,
		char *	read_addr,
		int	size_w,
		int	size_r,
		int	is_lf_server
	)

Constructor Initializes the resources.

Implementation for Public APIs.

Registers write_addr and read_addr as the read and write buffers and connects a queue pair with the specified remote node.

Parameters

r_id	The node id of the remote node to connect to.
write_addr	A pointer to the memory to use as the write buffer. This is where data should be written locally in order to send it in an RDMA write to the remote node.
read_addr	A pointer to the memory to use as the read buffer. This is where the results of RDMA reads from the remote node will arrive.
size_w	The size of the write buffer (in bytes).
size_r	The size of the read buffer (in bytes).
is_lf_server	Is local node a libfabric server or client. A libfabric client initiates connection to the passive endpoint of the remote node, while a libfabric server waiting for the conneciton using its local passive endpoint.

Definition at line 311 of file lf.cpp.

◆ ~_resources() [1/2]

sst::_resources::~_resources ( )

virtual

Destroys the resources.

Cleans up all IB Verbs resources associated with this connection.

Definition at line 368 of file lf.cpp.

◆ _resources() [2/2]

sst::_resources::_resources	(	int	r_index,
		char *	write_addr,
		char *	read_addr,
		int	size_w,
		int	size_r
	)

Constructor; initializes Queue Pair, Memory Regions, and remote_props.

Initializes the resources.

Registers write_addr and read_addr as the read and write buffers and connects a queue pair with the specified remote node.

Parameters

r_index	The node rank of the remote node to connect to.
write_addr	A pointer to the memory to use as the write buffer. This is where data should be written locally in order to send it in an RDMA write to the remote node.
read_addr	A pointer to the memory to use as the read buffer. This is where the results of RDMA reads from the remote node will arrive.
size_w	The size of the write buffer (in bytes).
size_r	The size of the read buffer (in bytes).

Definition at line 95 of file verbs.cpp.

◆ ~_resources() [2/2]

virtual sst::_resources::~_resources ( )

virtual

Destroys the resources.

Member Function Documentation

◆ connect_endpoint()

void sst::_resources::connect_endpoint ( bool is_lf_server )

private

Connect the queue pair.

Parameters

is_lf_server This parameter decide local role in connection. If is_lf_server is true, it waits on PEP for connection from remote side. Otherwise, it initiate a connection to remote side.

Definition at line 226 of file lf.cpp.

◆ connect_qp()

void sst::_resources::connect_qp ( )

private

Connect the queue pairs.

This method implements the entire setup of the queue pairs, calling all the modify_qp_* methods in the process.

Definition at line 254 of file verbs.cpp.

◆ init_endpoint()

int sst::_resources::init_endpoint ( struct fi_info * fi )

private

Initialize resource endpoint using fi_info.

Parameters

fi	The fi_info object

Returns: 0 for success.

Definition at line 198 of file lf.cpp.

◆ post_remote_send() [1/2]

int sst::_resources::post_remote_send	(	const uint32_t	id,
		const long long int	offset,
		const long long int	size,
		const int	op,
		const bool	completion
	)

protected

Post a remote RDMA operation.

This is used for both reads and writes.

Parameters

id
offset	The offset within the remote buffer to start the operation at.
size	The number of bytes to read or write.
op	The operation mode; 0 is for read, 1 is for write.

Returns: The return code of the IB Verbs post_send operation.

Definition at line 317 of file verbs.cpp.

◆ post_remote_send() [2/2]

int sst::_resources::post_remote_send	(	struct lf_sender_ctxt *	ctxt,
		const long long int	offset,
		const long long int	size,
		const int	op,
		const bool	completion
	)

protected

post read/write request

Parameters

ctxt	- pointer to the sender context, caller should maintain the ownership of this context until completion.
offset	- The offset within the remote buffer to read/write
size	- The number of bytes to read/write
op	- 0 for read and 1 for write
return	the return code for operation.

Definition at line 384 of file lf.cpp.

◆ set_qp_initialized()

void sst::_resources::set_qp_initialized ( )

private

Initializes the queue pair.

This transitions the queue pair to the init state.

Definition at line 178 of file verbs.cpp.

◆ set_qp_ready_to_receive()

void sst::_resources::set_qp_ready_to_receive ( )

private

Transitions the queue pair to the ready-to-receive state.

Definition at line 197 of file verbs.cpp.

◆ set_qp_ready_to_send()

void sst::_resources::set_qp_ready_to_send ( )

private

Transitions the queue pair to the ready-to-send state.

Definition at line 232 of file verbs.cpp.

Member Data Documentation

◆ ep

struct fid_ep* sst::_resources::ep

tx/rx completion queue

Handle for the LibFabric endpoint.

Definition at line 67 of file lf.hpp.

◆ eq

struct fid_eq* sst::_resources::eq

the event queue

Definition at line 85 of file lf.hpp.

◆ mr_lrkey

uint64_t sst::_resources::mr_lrkey

key for local read buffer

Definition at line 77 of file lf.hpp.

◆ mr_lwkey

uint64_t sst::_resources::mr_lwkey

key for local write buffer

Definition at line 79 of file lf.hpp.

◆ mr_rwkey

uint64_t sst::_resources::mr_rwkey

key for remote write buffer

Definition at line 81 of file lf.hpp.

◆ qp

struct ibv_qp* sst::_resources::qp

Handle for the IB Verbs Queue Pair object.

Definition at line 58 of file verbs.hpp.

◆ read_buf

char * sst::_resources::read_buf

Pointer to the memory buffer used for the results of RDMA remote reads.

Definition at line 75 of file lf.hpp.

◆ read_mr [1/2]

struct ibv_mr* sst::_resources::read_mr

Memory Region handle for the read buffer.

Definition at line 62 of file verbs.hpp.

◆ read_mr [2/2]

struct fid_mr* sst::_resources::read_mr

memory region for remote writer

Definition at line 71 of file lf.hpp.

◆ remote_fi_addr

fi_addr_t sst::_resources::remote_fi_addr

remote write memory address

Definition at line 83 of file lf.hpp.

◆ remote_id

int sst::_resources::remote_id

ID of the remote node.

Definition at line 63 of file lf.hpp.

◆ remote_index

int sst::_resources::remote_index

Index of the remote node.

Definition at line 56 of file verbs.hpp.

◆ remote_props

struct cm_con_data_t sst::_resources::remote_props

Connection data values needed to connect to remote side.

Definition at line 64 of file verbs.hpp.

◆ write_buf

char * sst::_resources::write_buf

Pointer to the memory buffer used for local writes.

Definition at line 73 of file lf.hpp.

◆ write_mr [1/2]

struct ibv_mr* sst::_resources::write_mr

Memory Region handle for the write buffer.

Definition at line 60 of file verbs.hpp.

◆ write_mr [2/2]

struct fid_mr* sst::_resources::write_mr

memory region for remote writer

Definition at line 69 of file lf.hpp.

The documentation for this class was generated from the following files:

include/derecho/sst/detail/lf.hpp
include/derecho/sst/detail/verbs.hpp
src/sst/lf.cpp
src/sst/verbs.cpp

Public Member Functions

Public Attributes

Protected Member Functions

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ _resources() [1/2]

◆ ~_resources() [1/2]

◆ _resources() [2/2]

◆ ~_resources() [2/2]

Member Function Documentation

◆ connect_endpoint()

◆ connect_qp()

◆ init_endpoint()

◆ post_remote_send() [1/2]

◆ post_remote_send() [2/2]

◆ set_qp_initialized()

◆ set_qp_ready_to_receive()

◆ set_qp_ready_to_send()

Member Data Documentation

◆ ep

◆ eq

◆ mr_lrkey

◆ mr_lwkey

◆ mr_rwkey

◆ qp

◆ read_buf

◆ read_mr [1/2]

◆ read_mr [2/2]

◆ remote_fi_addr

◆ remote_id

◆ remote_index

◆ remote_props

◆ write_buf

◆ write_mr [1/2]

◆ write_mr [2/2]