diff options
Diffstat (limited to 'Documentation/admin-guide/cgroup-v1/rdma.rst')
| -rw-r--r-- | Documentation/admin-guide/cgroup-v1/rdma.rst | 191 |
1 files changed, 191 insertions, 0 deletions
diff --git a/Documentation/admin-guide/cgroup-v1/rdma.rst b/Documentation/admin-guide/cgroup-v1/rdma.rst new file mode 100644 index 000000000..03e1f1b78 --- /dev/null +++ b/Documentation/admin-guide/cgroup-v1/rdma.rst @@ -0,0 +1,191 @@ +=============== +RDMA Controller +=============== + +.. Contents + + 1. Overview + 1-1. What is RDMA controller? + 1-2. Why RDMA controller needed? + 1-3. How is RDMA controller implemented? + 2. Usage Examples + 3. RDMA Interface Files + +1. Overview +=========== + +1-1. What is RDMA controller? +----------------------------- + +RDMA controller allows user to limit RDMA/IB specific resources that a given +set of processes can use. These processes are grouped using RDMA controller. + +RDMA controller defines two resources which can be limited for processes of a +cgroup. + +1-2. Why RDMA controller needed? +-------------------------------- + +Currently user space applications can easily take away all the rdma verb +specific resources such as AH, CQ, QP, MR etc. Due to which other applications +in other cgroup or kernel space ULPs may not even get chance to allocate any +rdma resources. This can lead to service unavailability. + +Therefore RDMA controller is needed through which resource consumption +of processes can be limited. Through this controller different rdma +resources can be accounted. + +1-3. How is RDMA controller implemented? +---------------------------------------- + +RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains +resource accounting per cgroup, per device using resource pool structure. +Each such resource pool is limited up to 64 resources in given resource pool +by rdma cgroup, which can be extended later if required. + +This resource pool object is linked to the cgroup css. Typically there +are 0 to 4 resource pool instances per cgroup, per device in most use cases. +But nothing limits to have it more. At present hundreds of RDMA devices per +single cgroup may not be handled optimally, however there is no +known use case or requirement for such configuration either. + +Since RDMA resources can be allocated from any process and can be freed by any +of the child processes which shares the address space, rdma resources are +always owned by the creator cgroup css. This allows process migration from one +to other cgroup without major complexity of transferring resource ownership; +because such ownership is not really present due to shared nature of +rdma resources. Linking resources around css also ensures that cgroups can be +deleted after processes migrated. This allow progress migration as well with +active resources, even though that is not a primary use case. + +Whenever RDMA resource charging occurs, owner rdma cgroup is returned to +the caller. Same rdma cgroup should be passed while uncharging the resource. +This also allows process migrated with active RDMA resource to charge +to new owner cgroup for new resource. It also allows to uncharge resource of +a process from previously charged cgroup which is migrated to new cgroup, +even though that is not a primary use case. + +Resource pool object is created in following situations. +(a) User sets the limit and no previous resource pool exist for the device +of interest for the cgroup. +(b) No resource limits were configured, but IB/RDMA stack tries to +charge the resource. So that it correctly uncharge them when applications are +running without limits and later on when limits are enforced during uncharging, +otherwise usage count will drop to negative. + +Resource pool is destroyed if all the resource limits are set to max and +it is the last resource getting deallocated. + +User should set all the limit to max value if it intents to remove/unconfigure +the resource pool for a particular device. + +IB stack honors limits enforced by the rdma controller. When application +query about maximum resource limits of IB device, it returns minimum of +what is configured by user for a given cgroup and what is supported by +IB device. + +Following resources can be accounted by rdma controller. + + ========== ============================= + hca_handle Maximum number of HCA Handles + hca_object Maximum number of HCA Objects + ========== ============================= + +RDMA devices from all network namespaces are listed. Each line starts with +the device name. If more than one device has the same name, ``index=N`` +follows the name, where ``N`` is the system-wide RDMA device index, unique +among registered devices. When configuring a limit, the index is optional +for a globally unique name and required for a duplicate name. A write without +the required index fails with ``-ENOTUNIQ``. + +2. Usage Examples +================= + +(a) Configure resource limit:: + + echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max + echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max + echo "rxe0 index=5 hca_handle=2" > /sys/fs/cgroup/rdma/3/rdma.max + +(b) Query resource limit:: + + cat /sys/fs/cgroup/rdma/2/rdma.max + #Output: + mlx4_0 hca_handle=2 hca_object=2000 + ocrdma1 hca_handle=3 hca_object=max + +(c) Query current usage:: + + cat /sys/fs/cgroup/rdma/2/rdma.current + #Output: + mlx4_0 hca_handle=1 hca_object=20 + ocrdma1 hca_handle=1 hca_object=23 + +(d) Delete resource limit:: + + echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max + +3. RDMA Interface Files +======================== + +The following interface files are available in each non-root RDMA cgroup. + + rdma.max + A read-write file which describes the configured resource limit + for an RDMA/IB device. See the Usage Examples above. + + rdma.current + A read-only file which describes the current resource usage. + + rdma.peak + A read-only nested-keyed file which shows the historical high + watermark of resource usage per device since the cgroup was created. + + An example for mlx4 and ocrdma device follows:: + + mlx4_0 hca_handle=1 hca_object=20 + ocrdma1 hca_handle=0 hca_object=23 + + rdma.events + A read-only nested-keyed file which exists on non-root cgroups + and contains the following keys: + + max + The number of times a process in this cgroup or its + descendants attempted an RDMA resource allocation that + was rejected because a rdma.max limit in the subtree + was reached. This is a hierarchical counter propagated + upward to all ancestor cgroups. A value change in this + file generates a file modified event. + + alloc_fail + The number of RDMA resource allocation attempts that + originated in this cgroup or its descendants and failed + due to a rdma.max limit being reached. This is a + hierarchical counter propagated upward. + + An example for mlx4 device follows:: + + mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0 + + rdma.events.local + Similar to rdma.events but the fields are local to the cgroup, + i.e. not hierarchical. The file modified event generated on this + file reflects only the local events. + + The following nested keys are defined. + + max + The number of times a process in this cgroup or its + descendants attempted an RDMA resource allocation that + was rejected because this cgroup's own rdma.max limit + was reached. + + alloc_fail + The number of RDMA resource allocation attempts + originating from this cgroup that failed due to this + cgroup's or an ancestor's rdma.max limit. + + An example for mlx4 device follows:: + + mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0 |