How to do RDMA programming in Python

Last updated on February 13, 2023 by Dan Nanni

RDMA (Remote Direct Memory Access) is a technology that allows direct access to memory on a remote computer over a network, without involving the remote computer's CPU or operating system. This results in very low latency and high-speed data transfer between machines. RDMA is widely used in many applications that require fast and efficient data transfer between machines, such as scientific simulation and modeling, big data analytics, cloud storage, distributed machine learning, etc.

RDMA can be used with a variety of programming languages including C/C++, Java, and Python. In general, the choice of programming language will depend on the requirements of your application and your development preferences. While C/C++ may be the best choice for high-performance computing applications, you may want to try a higher-level language like Python for its fast prototyping capability and more flexible abstraction.

In this tutorial, let me show how to do RDMA programming in Python.

Prerequisite

If you would like to use RDMA in your network application, you need RDMA-capable network interface card (NIC) on your servers (e.g., NVIDIA ConnectX-5/6) and compatible hardware switches that can interconnect these adapters. On software side, you need to have necessary NIC driver and RDMA software stack (e.g., Mellanox OFED) installed.

Keep in mind that RDMA programming can be complex, and requires a good understanding of low-level network programming concepts and RDMA verbs APIs. If you are new to RDMA, it is recommended that you start with basic RDMA tutorials.

Python Installation for RDMA Programming

You can use RDMA in Python by utilizing libraries such as Pyverbs, which is a Python wrapper for the RDMA verbs APIs. To install the Pyverbs library on an Ubuntu system, you can use the pip package manager. Here are the steps to follow.

First, install the dependencies required by Pyverbs:

$ sudo apt install libibverbs-dev libibverbs1

Then install Pyverbs using pip:

sudo pip install pyverbs

After you follow these steps, Pyverbs should be installed and ready to use in your Python environment. You can import the pyverbs module in your Python code and start using it to issue RDMA verbs.

Python Example of RDMA READ and WRITE Verbs

Here is a Python code example of how you can issue RDMA READ and WRITE verbs using the Pyverbs library in Python.

import pyverbs

# Create a context for the InfiniBand device
ctx = pyverbs.Context(pyverbs.IBV_SOCKET_RAW, 1)

# Create an RDMA endpoint (QP)
qp = ctx.qp(
    pyverbs.PD(),
    pyverbs.RCQP(),
    {
        'qp_type': pyverbs.IBV_QPT_RC,
        'send_cq': ctx.cq(ctx.comp_channel(), 1000),
        'recv_cq': ctx.cq(ctx.comp_channel(), 1000),
        'cap': {
            'max_send_wr': 1000,
            'max_recv_wr': 1000,
            'max_send_sge': 1,
            'max_recv_sge': 1,
        },
    }
)

# Register a memory buffer for RDMA operations
buf = ctx.buf(1024)
mr = ctx.mr(buf, pyverbs.IBV_ACCESS_LOCAL_WRITE | pyverbs.IBV_ACCESS_REMOTE_WRITE |
    pyverbs.IBV_ACCESS_REMOTE_READ)

# Issue an RDMA READ operation
read_wr = pyverbs.RDMAWrite(
    qp,
    {
        'remote_addr': mr.addr,
        'rkey': mr.rkey,
        'wr_id': 1,
        'opcode': pyverbs.IBV_WR_RDMA_READ,
    }
)

# Issue an RDMA WRITE operation
write_wr = pyverbs.RDMAWrite(
    qp,
    {
        'remote_addr': mr.addr,
        'rkey': mr.rkey,
        'wr_id': 2,
        'opcode': pyverbs.IBV_WR_RDMA_WRITE,
    }
)

# Wait for the completion of the RDMA operations
ev = ctx.comp_channel().poll()
print("RDMA operation completed with status:", ev.status)

This is just a simple example to illustrate the basic steps involved in issuing RDMA READ and WRITE verbs using Pyverbs. In a real-world scenario, you would need to handle additional details such as connection establishment, error handling, and so on. Additionally, you would also need to coordinate the actions between the local and remote endpoints to ensure that the RDMA operations are performed correctly.

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal or credit card

Please note that this article is published by Xmodulo.com under a Creative Commons Attribution-ShareAlike 3.0 Unported License. If you would like to use the whole or any part of this article, you need to cite this web page at Xmodulo.com as the original source.