<div id="readability-page-1" class="page"><div id="container">
<p>1.<br>
RDMA와 관련한 글을 하나더 올립니다. 2010년을 전후한 때에 북마크했던 곳인데 지금은 거의 글이 올라오지 않습니다.</p>
<p><a href="https://zcopy.wordpress.com/" target="_blank" rel="noopener">ZCopy:Education and Sample Code for RDMA Programming</a>에 첫번째로 올라왔던 <a href="https://zcopy.wordpress.com/2010/10/08/quick-concepts-part-1-%e2%80%93-introduction-to-rdma/" target="_blank" rel="noopener">Quick Concepts Part 1 – Introduction to RDMA</a>입니다. 아쉽게도 이후 글이 이어지지 않고 현재에 이릅니다. 주인장이 어딜론가 떠난 듯 합니다. 그래서 PDF로 변화하여 올립니다. 개인 기록을 겸합니다.</p>

<p><a href="http://smallake.kr/wp-content/uploads/2025/05/Quick-Concepts-Part-1-%E2%80%93-Introduction-to-RDMA.pdf">Download (PDF, 168KB)</a></p>
<p>2.<br>
또다른 곳은 <a href="https://www.rdmamojo.com/" target="_blank" rel="noopener">RDMAmojo</a>입니다. 블로그 제목처럼 “RDMAmojo – blog on RDMA technology and programming by Dotan Barak”이라는 블로그 제목처럼 Dotan Barak이 운영하는 사이트입니다. Dotab Barak은 Mellanox에서 개발자로 근무하였던 분입니다. 현재 2018년까지 글이 올라왔습니다. 주로 RDMA와 관련한 함수 설명이 많습니다. 이와 관련해서  <a href="https://docs.nvidia.com/networking/display/rdmaawareprogrammingv17" rel="noopener" target="_blank">RDMA Aware Networks Programming User Manual</a>을 같이 참고하세요. Mellanox를 인수한 Nvidia가 제공하는 문서입니다.</p>

<p><a href="http://smallake.kr/wp-content/uploads/2025/05/rdma-aware-networks-programming-user-manual-1-7.pdf">Download (PDF, 911KB)</a></p>
<p>제가 관심을 가지고 본 글입니다. libvma를 튜닝할 때 혹 도움이 될까해서 참고하고 있습니다.</p>
<blockquote>
<h3><span id="General_tips">General tips</span></h3>
<h3><span id="Avoid_using_control_operations_in_the_data_path">Avoid using control operations in the data path</span></h3>
<p>Unlike the data operations that stay in the same context that they were called in (i.e. don’t perform a context switch) and they are written in optimized way, the control operations (all create/destroy/query/modify) operations are very expensive because:</p>
<ul>
<li>Most of the time, they perform a context switch</li>
<li>Sometimes they allocate or free dynamic memory</li>
<li>Sometimes they involved in accessing the RDMA device</li>
</ul>
<p>As a general rule of thumb, one should avoid calling control operations or decrease its use in the data path.</p>
<p>The following verbs are considered as data operations:</p>
<ul>
<li><strong>ibv_post_send()</strong></li>
<li><strong>ibv_post_recv()</strong></li>
<li><strong>ibv_post_srq_recv()</strong></li>
<li><strong>ibv_poll_cq()</strong></li>
<li><strong>ibv_req_notify_cq</strong></li>
</ul>
<h3><span id="When_posting_multiple_WRs_post_them_in_a_list_in_one_call">When posting multiple WRs, post them in a list in one call</span></h3>
<p>When posting several Work Requests to one of the <strong>ibv_post_*()</strong> verbs, posting multiple Work Requests as a linked list in one call instead of several calls each time with one Work Request will provide better performance since it allows the low-level driver to perform optimizations.</p>
<h2><span id="When_using_Work_Completion_events_acknowledge_several_events_in_one_call">When using Work Completion events, acknowledge several events in one call</span></h2>
<p>When handling Work Completions using events, acknowledging several completions in one call instead of several calls each time will provide better performance since less mutual exclusion locks are being performed.</p>
<h2><span id="Avoid_using_many_scattergather_entries">Avoid using many scatter/gather entries</span></h2>
<p>Using several scatter/gather entries in a Work Request (either Send Request or Receive Request) mean that the RDMA device will read those entries and will read the memory that they refer to. Using one scatter/gather entry will provide better performance than more than one.</p>
<h2><span id="Avoid_using_Fence">Avoid using Fence</span></h2>
<p>Send Request with the fence flag set will be blocked until all prior RDMA Read and Atomic Send Requests will be completed. This will decrease the BW.</p>
<h2><span id="Avoid_using_atomic_operations">Avoid using atomic operations</span></h2>
<p>Atomic Operations allow to perform read-modify-write in an atomic way. This usually will decrease the performance since doing this usually involved in locking the access to the memory (implementation dependent).</p>
<h2><span id="Read_multiple_Work_Completions_at_once">Read multiple Work Completions at once</span></h2>
<p><strong>ibv_poll_cq()</strong> allows to reading multiple completions at once. If the number of Work Completions in the CQ is less than the number of Work Completion that one tried to read, it means that the CQ is empty and there isn’t any need to check if there are more Work Completions in it.</p>
<h2><span id="Set_processor_affinity_for_a_certain_task_or_process">Set processor affinity for a certain task or process</span></h2>
<p>When working with a Symmetric MultiProcessing (SMP) machines, binding the processes to a specific CPU(s)/core(s) may provide better utilization of the CPU(s)/core(s) thus provide better performance. Executing processes as the number of CPU(s)/core(s) in a machine and spread a process to each CPU(s)/core(s) may be a good practice. This can be done with the “taskset” utility.</p>
<h2><span id="Work_with_local_NUMA_node">Work with local NUMA node</span></h2>
<p>When working on a Non-Uniform Memory Access (NUMA) machines, binding the processes to CPU(s)/core(s) which are considered local NUMA nodes for the RDMA device may provide better performance because of faster CPU access. Spreading the processes to all of the local CPU(s)/core(s) may be a good practice.</p>
<h2><span id="Work_with_cache-line_aligned_buffers">Work with cache-line aligned buffers</span></h2>
<p>Working with cache-line aligned buffers (in S/G list, Send Request, Receive Request and data) will improve performance compared to working with unaligned memory buffers; it will decrease the number of CPU cycles and number of memory accesses.</p>
<h2><span id="Avoid_getting_into_retransmission_flows">Avoid getting into retransmission flows</span></h2>
<p>Retransmission is a performance killer. There are 2 major reasons for retransmission in RDMA:</p>
<ul>
<li>Transport retransmission – remote QP isn’t at a state that can process incoming messages, i.e. didn’t get to, at least, RTR state, or moved to Error state</li>
<li>RNR retransmission – there is a message that should consume a Receive Request in the responder side, but there isn’t any Receive Request in the Receive Queue</li>
</ul>
<p>There are RDMA devices that provide counters to indicate that retry flows occurred, but not all of them.</p>
<p>Setting QP.retry_cnt and QP.rnr_retry to zero will cause a failure (i.e. Completion with error) when the QP enters those flows.</p>
<p>However, if retry flows can’t be avoided, use low (as possible) delay between the retransmission.</p>
<h2><span id="Improving_the_Bandwidth">Improving the Bandwidth</span></h2>
<h2><span id="Find_the_best_MTU_for_the_RDMA_device">Find the best MTU for the RDMA device</span></h2>
<p>The MTU value specifies the maximum packet payload size (i.e. excluding the packet headers) that can be sent. As a rule of thumb since the packet header sizes are the same for all MTU values, using the maximum available MTU size will decrease the “paid price” per packet; the percent of the payload data in the total used BW will be increased. However, there are RDMA devices which provide the best performance for MTU values which are lower than the maximum supported value. One should perform some testing in order to find the best MTU for the specific device that he works with.</p>
<h2><span id="Use_big_messages">Use big messages</span></h2>
<p>Sending a few big messages is more effective than sending a lot of small messages. In application level one should collect data and send big messages over RDMA.</p>
<h2><span id="Work_with_multiple_outstanding_Send_Requests">Work with multiple outstanding Send Requests</span></h2>
<p>Working with multiple outstanding Send Requests and keeping the Send Queue always full (i.e. for every polled Work Completion post a new Send Request) will keep the RDMA device busy and prevents it from being idle.</p>
<h2><span id="Configure_the_Queue_Pair_to_allow_several_RDMA_Reads_and_Atomic_in_parallel">Configure the Queue Pair to allow several RDMA Reads and Atomic in parallel</span></h2>
<p>If one uses RDMA Read or Atomic operations, it is advised to configure the QP to work with several RDMA Read and Atomic operations in flight since it will provide higher BW.</p>
<h2><span id="Work_with_selective_signaling_in_the_Send_Queue">Work with selective signaling in the Send Queue</span></h2>
<p>Working with selective signaling in the Send Queue means that not every Send Request will produce a Work Completion when it ends and this will reduce the number of Work Completions that should be handled.</p>
<h2><span id="Reducing_the_latency">Reducing the latency</span></h2>
<h2><span id="Read_Work_Completions_by_polling">Read Work Completions by polling</span></h2>
<p>In order to read the Work Completion as soon as they are added to the Completion Queue, polling will provide the best results (rather than working with Work Completion events).</p>
<h2><span id="Send_small_messages_as_inline">Send small messages as inline</span></h2>
<p>In RDMA devices which supports sending data as inline, sending small messages as inline will provide better latency since it eliminates the need of the RDMA device to perform extra read (over the PCIe bus) in order to read the message payload.</p>
<h2><span id="Use_low_values_in_QPs_timeout_and_min_rnr_timer">Use low values in QP’s timeout and min_rnr_timer</span></h2>
<p>Using lower values in the QP’s timeout and min_rnr_timer means that in case that something gets wrong and retry is required (whether if because the remote QP doesn’t answer or if it doesn’t have outstanding Receive Request) the waited time before a retransmission will be short.</p>
<h2><span id="If_immediate_data_is_used_use_RDMA_Write_with_immediate_instead_of_Send_with_immediate">If immediate data is used, use RDMA Write with immediate instead of Send with immediate</span></h2>
<p>When sending a message that has only immediate data, RDMA Write with immediate will provide better performance than Send With immediate since the latter causes the outstanding posted Receive Request to be read (in the responder side) and not only be consumed.</p>
<h2><span id="Reducing_memory_consumption">Reducing memory consumption</span></h2>
<h2><span id="Use_Shared_Receive_Queue_SRQ">Use Shared Receive Queue (SRQ)</span></h2>
<p>Instead of posting many Receive Requests for each Queue Pair, using SRQ can save the total number of outstanding Receive Request thus reduce the total consumed memory.</p>
<h2><span id="Register_physical_contiguous_memory">Register physical contiguous memory</span></h2>
<p>Register physical contiguous memory, such as huge pages, can allow the low-level driver(s) to perform optimizations since lower amount of memory address translations will be required (compared to 4KB memory pages buffer).</p>
<h2><span id="Reduce_the_size_of_the_used_Queues_to_the_minimum">Reduce the size of the used Queues to the minimum</span></h2>
<p>Creating the various Queues (Queue Pairs, Shared Receive Queues, Completion Queues) may consume a lot of memory. One should set the size of them to the minimum that is required by his application.</p>
<h2><span id="Reducing_CPU_consumption">Reducing CPU consumption</span></h2>
<h2><span id="Work_with_Work_Completion_events">Work with Work Completion events</span></h2>
<p>Reading the Work Completions using events will eliminate the need to perform constant polling on the CQ since the RDMA device will send an event when a Work Completion was added to the CQ.</p>
<h2><span id="Work_with_solicited_events_in_Responder_side">Work with solicited events in Responder side</span></h2>
<p>When reading the Work Completions in the Responder side, the solicited event can be a good way to the Requestor to provide a hint that now is a good time to read the completions. This reduces the total number of handled Work Completions.</p>
<h2></h2>
<p>Using the same CQ with several Queues and reducing the total number of CQs will eliminate the need to check several CQs in order to understand if an outstanding Work Request was completed. This can be done by sharing the same CQ with multiple Send Queues, multiple Receive Queues or with a mix of them.</p>
<h2><span id="Increase_the_scalability">Increase the scalability</span></h2>
<h2><span id="Use_collective_algorithms">Use collective algorithms</span></h2>
<p>Using collective algorithms will reduce the total number of messages that cross the wire and will decrease the total number of messages and resources that each node in a cluster will use. There are RDMA devices that provide special collective offload operations that will help reducing the CPU utilization.</p>
<h2><span id="Use_Unreliable_Datagram_UD_QP">Use Unreliable Datagram (UD) QP</span></h2>
<p>If every node needs to be able to receive or send a message to any other node in the subnet, using a connected QP (either Reliable or Unreliable) may be a bad solution since many QPs will be created in every node. Using a UD QP is better since it can send and receive messages from any other UD QP in the subnet.</p>
<p><a href="https://www.rdmamojo.com/2013/06/08/tips-and-tricks-to-optimize-your-rdma-code/">Tips and tricks to optimize your RDMA code</a>중에서</p></blockquote>
<p>또다른 사항은 <a href="https://www.rdmamojo.com/2015/01/24/verify-rdma-working/" target="_blank" rel="noopener">Verify that RDMA is working</a>입니다. 역시나 libvma를 위한 공부입니다. </p>
<p> RDMA를 매매시스템에서 사용할 수 있는 곳이 있을까 생각해봅니다. 아마도 전략시스템과 FEP사이가 아닐까 합니다. 고객 맞춤형 FEP를 공급하는 곳도 있기때문에 부분적인 수정개발을 통하여 적용해보면 어떨까 합니다. </p>

  </div></div>

그대안의 작은 호수

1. RDMA와 관련한 글을 하나더 올립니다. 2010년을 전후한 때에 북마크했던 곳인데 지금은 거의 글이 올라오지 않습니다. ZCopy:Education and Sample Code for RDMA Programming에 첫번째로 올라왔던 Quick Concepts Part 1 – Introduction to RDMA입니다. 아쉽게도 이후 글이 이어지지 않고 현재에 이릅니다. 주인장이 어딜론가 떠난 듯 합니다. 그래서 PDF로 변화하여 올립니다. 개인 기록을 겸합니다. 2. 또다른 곳은 RDMAmojo입니다. 블로그 제목처럼 “RDMAmojo –...
Read more
RDMA 프로그래밍과 튜닝 first appeared on 그대안의작은호수.

RDMA 프로그래밍과 튜닝

댓글