GitLab

Right now, two pools are initialized within the Infiniband node. When a Receive Work Request is placed into the Receive Queue, the node gets a buffer out of the pool, and places the address into the Receive Queue. So, if the HCA receives a packet, it writes it into this buffer and generates a Work Completion with that address.

Then, when the nodes' read function ib_read is called, the function takes the address out of the Work Completion and copies the content of the buffer into a second buffer outside of the node.

A solution to this problem is to directly submit the addresses which are passed to ib_read to the Receive Queue. Pseudo code of this solution is:

ib_read(*sample) {
  ret = poll_cq()
  if(ret > 0) {
    //get sample from queue
    //change address of *sample
    return 
  } else if(receive queue not full) {
    //Add address of *sample to Receive Queue
  }
  return 0;
}

A problem is that the sample buffers which are passed to ib_read have to be registered to the protection domain of the IB node. This can be solved with a special memory type, as introduced in c01393bc