sched_ext: Maintain an online cid mask in the scheduler arenagrafted

Schedulers on the default cid mapping treat [0, nr_online_cids) as the
online set and restart on hotplug. Schedulers that install their own mapping
with scx_bpf_cid_override() have no way to learn which cids are online: the
count no longer identifies members and the CPU-form cpumask is unusable from
cid programs. This is an obvious hole in the cid API.

Add scx_bpf_online_cmask(), a kernel-maintained cmask in the scheduler's
arena, allocated alongside the per-CPU scratch masks and populated after the
cid mapping is finalized and before ops.init(), for child schedulers too.
The pointer stays valid through ops.exit() with no reference to take. It is
the arena offset as a void pointer, the same form struct_ops arena arguments
arrive in. The verifier types the void return as a scalar for the program's
arena cast.

The mask follows the SCX hotplug notifications: seeded from cpu_active_mask
and updated before ops.cid_online/offline() runs, so it lags cpu_online_mask
only inside a hotplug transition. Updates walk the scheduler list under the
lock that also serializes unlinking. Reads are live, not atomic snapshots.
Root initialization excludes hotplug.

v2: Reworded the getter kerneldoc (Andrea Righi).

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

1 files changed, 133 insertions, 0 deletions

diff --git a/Documentation/ABI/testing/sysfs-class-rnbd-client b/Documentation/ABI/testing/sysfs-class-rnbd-client
new file mode 100644
index 000000000..0b5997ab3
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-rnbd-client
@@ -0,0 +1,133 @@
+What:		/sys/class/rnbd-client
+Date:		Feb 2020
+KernelVersion:	5.7
+Contact:	Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
+Description:	Provide information about RNBD-client.
+		All sysfs files that are not read-only provide the usage information on read:
+
+		Example::
+
+		    # cat /sys/class/rnbd-client/ctl/map_device
+
+		    > Usage: echo "sessname=<name of the rtrs session> path=<[srcaddr,]dstaddr>
+		    > [path=<[srcaddr,]dstaddr>] device_path=<full path on remote side>
+		    > [access_mode=<ro|rw|migration>] > map_device
+		    >
+		    > addr ::= [ ip:<ipv4> | ip:<ipv6> | gid:<gid> ]
+
+What:		/sys/class/rnbd-client/ctl/map_device
+Date:		Feb 2020
+KernelVersion:	5.7
+Contact:	Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
+Description:	Expected format is the following::
+
+		    sessname=<name of the rtrs session>
+		    path=<[srcaddr,]dstaddr> [path=<[srcaddr,]dstaddr> ...]
+		    device_path=<full path on remote side>
+		    [access_mode=<ro|rw|migration>]
+
+		Where:
+
+		sessname:
+		    accepts a string not bigger than 256 chars, which identifies
+		    a given session on the client and on the server.
+		    I.e. "clt_hostname-srv_hostname" could be a natural choice.
+
+		path:
+		    describes a connection between the client and the server by
+		    specifying destination and, when required, the source address.
+		    The addresses are to be provided in the following format::
+
+			ip:<IPv6>
+			ip:<IPv4>
+			gid:<GID>
+
+		for example::
+
+		    path=ip:10.0.0.66
+
+		The single addr is treated as the destination.
+		The connection will be established to this server from any client IP address.
+
+		::
+
+		    path=ip:10.0.0.66,ip:10.0.1.66
+
+		First addr is the source address and the second is the destination.
+
+		If multiple "path=" options are specified multiple connection
+		will be established and data will be sent according to
+		the selected multipath policy (see RTRS mp_policy sysfs entry description).
+
+		device_path:
+		    Path to the block device on the server side. Path is specified
+		    relative to the directory on server side configured in the
+		    'dev_search_path' module parameter of the rnbd_server.
+		    The rnbd_server prepends the <device_path> received from client
+		    with <dev_search_path> and tries to open the
+		    <dev_search_path>/<device_path> block device.  On success,
+		    a /dev/rnbd<N> device file, a /sys/block/rnbd<N>/
+		    directory and an entry in /sys/class/rnbd-client/ctl/devices
+		    will be created.
+
+		If 'dev_search_path' contains '%SESSNAME%', then each session can
+		have different devices namespace, e.g. server was configured with
+		the following parameter "dev_search_path=/run/rnbd-devs/%SESSNAME%",
+		client has this string "sessname=blya device_path=sda", then server
+		will try to open: /run/rnbd-devs/blya/sda.
+
+		access_mode:
+		    the access_mode parameter specifies if the device is to be
+		    mapped as "ro" read-only or "rw" read-write. The server allows
+		    a device to be exported in rw mode only once. The "migration"
+		    access mode has to be specified if a second mapping in read-write
+		    mode is desired.
+
+		By default "rw" is used.
+
+		nr_poll_queues
+		  specifies the number of poll-mode queues. If the IO has HIPRI flag,
+		  the block-layer will send the IO via the poll-mode queue.
+		  For fast network and device the polling is faster than interrupt-base
+		  IO handling because it saves time for context switching, switching to
+		  another process, handling the interrupt and switching back to the
+		  issuing process.
+
+		  Set -1 if you want to set it as the number of CPUs
+		  By default rnbd client creates only irq-mode queues.
+
+		  NOTICE: MUST make a unique session for a device using the poll-mode queues.
+
+		Exit Codes:
+
+		If the device is already mapped it will fail with EEXIST. If the input
+		has an invalid format it will return EINVAL. If the device path cannot
+		be found on the server, it will fail with ENOENT.
+
+		Finding device file after mapping
+		---------------------------------
+
+		After mapping, the device file can be found by:
+		o  The symlink /sys/class/rnbd-client/ctl/devices/<device_id>@<session_name>
+		points to /sys/block/<dev-name>. The last part of the symlink destination
+		is the same as the device name.  By extracting the last part of the
+		path the path to the device /dev/<dev-name> can be build.
+
+		* /dev/block/$(cat /sys/class/rnbd-client/ctl/devices/<device_id>@<session_name>/dev)
+
+		How to find the <device_id> of the device is described on the next
+		section.
+
+What:		/sys/class/rnbd-client/ctl/devices/
+Date:		Feb 2020
+KernelVersion:	5.7
+Contact:	Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
+Description:	For each device mapped on the client a new symbolic link is created as
+		/sys/class/rnbd-client/ctl/devices/<device_id>@<session_name>, which points
+		to the block device created by rnbd (/sys/block/rnbd<N>/).
+		The <device_id> of each device is created as follows:
+
+		- If the 'device_path' provided during mapping contains slashes ("/"),
+		  they are replaced by exclamation mark ("!") and used as as the
+		  <device_id>. Otherwise, the <device_id> will be the same as the
+		  "device_path" provided.