mirror of
https://github.com/AuxXxilium/linux_dsm_epyc7002.git
synced 2025-04-04 10:38:04 +07:00
block/rnbd: a bit of documentation
README with description of major sysfs entries, sysfs documentation are moved to ABI dir as Bart suggested. Link: https://lore.kernel.org/r/20200511135131.27580-25-danil.kipnis@cloud.ionos.com Signed-off-by: Danil Kipnis <danil.kipnis@cloud.ionos.com> Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com> Reviewed-by: Bart Van Assche <bvanassche@acm.org> Acked-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
This commit is contained in:
Jack Wang
Jason Gunthorpe
parent
bc01885342
commit
aa4d16e44f
46
Documentation/ABI/testing/sysfs-block-rnbd
Normal file
46
Documentation/ABI/testing/sysfs-block-rnbd
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
What: /sys/block/rnbd<N>/rnbd/unmap_device
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: To unmap a volume, "normal" or "force" has to be written to:
|
||||||
|
/sys/block/rnbd<N>/rnbd/unmap_device
|
||||||
|
|
||||||
|
When "normal" is used, the operation will fail with EBUSY if any process
|
||||||
|
is using the device. When "force" is used, the device is also unmapped
|
||||||
|
when device is in use. All I/Os that are in progress will fail.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
# echo "normal" > /sys/block/rnbd0/rnbd/unmap_device
|
||||||
|
|
||||||
|
What: /sys/block/rnbd<N>/rnbd/state
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: The file contains the current state of the block device. The state file
|
||||||
|
returns "open" when the device is successfully mapped from the server
|
||||||
|
and accepting I/O requests. When the connection to the server gets
|
||||||
|
disconnected in case of an error (e.g. link failure), the state file
|
||||||
|
returns "closed" and all I/O requests submitted to it will fail with -EIO.
|
||||||
|
|
||||||
|
What: /sys/block/rnbd<N>/rnbd/session
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: RNBD uses RTRS session to transport the data between client and
|
||||||
|
server. The entry "session" contains the name of the session, that
|
||||||
|
was used to establish the RTRS session. It's the same name that
|
||||||
|
was passed as server parameter to the map_device entry.
|
||||||
|
|
||||||
|
What: /sys/block/rnbd<N>/rnbd/mapping_path
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Contains the path that was passed as "device_path" to the map_device
|
||||||
|
operation.
|
||||||
|
|
||||||
|
What: /sys/block/rnbd<N>/rnbd/access_mode
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Contains the device access mode: ro, rw or migration.
|
||||||
111
Documentation/ABI/testing/sysfs-class-rnbd-client
Normal file
111
Documentation/ABI/testing/sysfs-class-rnbd-client
Normal file
@ -0,0 +1,111 @@
|
|||||||
|
What: /sys/class/rnbd-client
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Provide information about RNBD-client.
|
||||||
|
All sysfs files that are not read-only provide the usage information on read:
|
||||||
|
|
||||||
|
Example:
|
||||||
|
# cat /sys/class/rnbd-client/ctl/map_device
|
||||||
|
|
||||||
|
> Usage: echo "sessname=<name of the rtrs session> path=<[srcaddr,]dstaddr>
|
||||||
|
> [path=<[srcaddr,]dstaddr>] device_path=<full path on remote side>
|
||||||
|
> [access_mode=<ro|rw|migration>] > map_device
|
||||||
|
>
|
||||||
|
> addr ::= [ ip:<ipv4> | ip:<ipv6> | gid:<gid> ]
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-client/ctl/map_device
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Expected format is the following:
|
||||||
|
|
||||||
|
sessname=<name of the rtrs session>
|
||||||
|
path=<[srcaddr,]dstaddr> [path=<[srcaddr,]dstaddr> ...]
|
||||||
|
device_path=<full path on remote side>
|
||||||
|
[access_mode=<ro|rw|migration>]
|
||||||
|
|
||||||
|
Where:
|
||||||
|
|
||||||
|
sessname: accepts a string not bigger than 256 chars, which identifies
|
||||||
|
a given session on the client and on the server.
|
||||||
|
I.e. "clt_hostname-srv_hostname" could be a natural choice.
|
||||||
|
|
||||||
|
path: describes a connection between the client and the server by
|
||||||
|
specifying destination and, when required, the source address.
|
||||||
|
The addresses are to be provided in the following format:
|
||||||
|
|
||||||
|
ip:<IPv6>
|
||||||
|
ip:<IPv4>
|
||||||
|
gid:<GID>
|
||||||
|
|
||||||
|
for example:
|
||||||
|
|
||||||
|
path=ip:10.0.0.66
|
||||||
|
The single addr is treated as the destination.
|
||||||
|
The connection will be established to this server from any client IP address.
|
||||||
|
|
||||||
|
path=ip:10.0.0.66,ip:10.0.1.66
|
||||||
|
First addr is the source address and the second is the destination.
|
||||||
|
|
||||||
|
If multiple "path=" options are specified multiple connection
|
||||||
|
will be established and data will be sent according to
|
||||||
|
the selected multipath policy (see RTRS mp_policy sysfs entry description).
|
||||||
|
|
||||||
|
device_path: Path to the block device on the server side. Path is specified
|
||||||
|
relative to the directory on server side configured in the
|
||||||
|
'dev_search_path' module parameter of the rnbd_server.
|
||||||
|
The rnbd_server prepends the <device_path> received from client
|
||||||
|
with <dev_search_path> and tries to open the
|
||||||
|
<dev_search_path>/<device_path> block device. On success,
|
||||||
|
a /dev/rnbd<N> device file, a /sys/block/rnbd_client/rnbd<N>/
|
||||||
|
directory and an entry in /sys/class/rnbd-client/ctl/devices
|
||||||
|
will be created.
|
||||||
|
|
||||||
|
If 'dev_search_path' contains '%SESSNAME%', then each session can
|
||||||
|
have different devices namespace, e.g. server was configured with
|
||||||
|
the following parameter "dev_search_path=/run/rnbd-devs/%SESSNAME%",
|
||||||
|
client has this string "sessname=blya device_path=sda", then server
|
||||||
|
will try to open: /run/rnbd-devs/blya/sda.
|
||||||
|
|
||||||
|
access_mode: the access_mode parameter specifies if the device is to be
|
||||||
|
mapped as "ro" read-only or "rw" read-write. The server allows
|
||||||
|
a device to be exported in rw mode only once. The "migration"
|
||||||
|
access mode has to be specified if a second mapping in read-write
|
||||||
|
mode is desired.
|
||||||
|
|
||||||
|
By default "rw" is used.
|
||||||
|
|
||||||
|
Exit Codes:
|
||||||
|
|
||||||
|
If the device is already mapped it will fail with EEXIST. If the input
|
||||||
|
has an invalid format it will return EINVAL. If the device path cannot
|
||||||
|
be found on the server, it will fail with ENOENT.
|
||||||
|
|
||||||
|
Finding device file after mapping
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
After mapping, the device file can be found by:
|
||||||
|
o The symlink /sys/class/rnbd-client/ctl/devices/<device_id>
|
||||||
|
points to /sys/block/<dev-name>. The last part of the symlink destination
|
||||||
|
is the same as the device name. By extracting the last part of the
|
||||||
|
path the path to the device /dev/<dev-name> can be build.
|
||||||
|
|
||||||
|
o /dev/block/$(cat /sys/class/rnbd-client/ctl/devices/<device_id>/dev)
|
||||||
|
|
||||||
|
How to find the <device_id> of the device is described on the next
|
||||||
|
section.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-client/ctl/devices/
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: For each device mapped on the client a new symbolic link is created as
|
||||||
|
/sys/class/rnbd-client/ctl/devices/<device_id>, which points
|
||||||
|
to the block device created by rnbd (/sys/block/rnbd<N>/).
|
||||||
|
The <device_id> of each device is created as follows:
|
||||||
|
|
||||||
|
- If the 'device_path' provided during mapping contains slashes ("/"),
|
||||||
|
they are replaced by exclamation mark ("!") and used as as the
|
||||||
|
<device_id>. Otherwise, the <device_id> will be the same as the
|
||||||
|
"device_path" provided.
|
||||||
50
Documentation/ABI/testing/sysfs-class-rnbd-server
Normal file
50
Documentation/ABI/testing/sysfs-class-rnbd-server
Normal file
@ -0,0 +1,50 @@
|
|||||||
|
What: /sys/class/rnbd-server
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: provide information about RNBD-server.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: When a client maps a device, a directory entry with the name of the
|
||||||
|
block device is created under /sys/class/rnbd-server/ctl/devices/.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/devices/<device_name>/block_dev
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Is a symlink to the sysfs entry of the exported device.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
block_dev -> ../../../../class/block/ram0
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/devices/<device_name>/sessions/
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: For each client a particular device is exported to, following directory will be
|
||||||
|
created:
|
||||||
|
|
||||||
|
/sys/class/rnbd-server/ctl/devices/<device_name>/sessions/<session-name>/
|
||||||
|
|
||||||
|
When the device is unmapped by that client, the directory will be removed.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/devices/<device_name>/sessions/<session-name>/read_only
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Contains '1' if device is mapped read-only, otherwise '0'.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/devices/<device_name>/sessions/<session-name>/mapping_path
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Contains the relative device path provided by the user during mapping.
|
||||||
|
|
||||||
|
What: /sys/class/rnbd-server/ctl/devices/<device_name>/sessions/<session-name>/access_mode
|
||||||
|
Date: Feb 2020
|
||||||
|
KernelVersion: 5.7
|
||||||
|
Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
|
||||||
|
Description: Contains the device access mode: ro, rw or migration.
|
||||||
92
drivers/block/rnbd/README
Normal file
92
drivers/block/rnbd/README
Normal file
@ -0,0 +1,92 @@
|
|||||||
|
********************************
|
||||||
|
RDMA Network Block Device (RNBD)
|
||||||
|
********************************
|
||||||
|
|
||||||
|
Introduction
|
||||||
|
------------
|
||||||
|
|
||||||
|
RNBD (RDMA Network Block Device) is a pair of kernel modules
|
||||||
|
(client and server) that allow for remote access of a block device on
|
||||||
|
the server over RTRS protocol using the RDMA (InfiniBand, RoCE, iWARP)
|
||||||
|
transport. After being mapped, the remote block devices can be accessed
|
||||||
|
on the client side as local block devices.
|
||||||
|
|
||||||
|
I/O is transferred between client and server by the RTRS transport
|
||||||
|
modules. The administration of RNBD and RTRS modules is done via
|
||||||
|
sysfs entries.
|
||||||
|
|
||||||
|
Requirements
|
||||||
|
------------
|
||||||
|
|
||||||
|
RTRS kernel modules
|
||||||
|
|
||||||
|
Quick Start
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Server side:
|
||||||
|
# modprobe rnbd_server
|
||||||
|
|
||||||
|
Client side:
|
||||||
|
# modprobe rnbd_client
|
||||||
|
# echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \
|
||||||
|
/sys/devices/virtual/rnbd-client/ctl/map_device
|
||||||
|
|
||||||
|
Where "sessname=" is a session name, a string to identify the session
|
||||||
|
on client and on server sides; "path=" is a destination IP address or
|
||||||
|
a pair of a source and a destination IPs, separated by comma. Multiple
|
||||||
|
"path=" options can be specified in order to use multipath (see RTRS
|
||||||
|
description for details); "device_path=" is the block device to be
|
||||||
|
mapped from the server side. After the session to the server machine is
|
||||||
|
established, the mapped device will appear on the client side under
|
||||||
|
/dev/rnbd<N>.
|
||||||
|
|
||||||
|
|
||||||
|
RNBD-Server Module Parameters
|
||||||
|
=============================
|
||||||
|
|
||||||
|
dev_search_path
|
||||||
|
---------------
|
||||||
|
|
||||||
|
When a device is mapped from the client, the server generates the path
|
||||||
|
to the block device on the server side by concatenating dev_search_path
|
||||||
|
and the "device_path" that was specified in the map_device operation.
|
||||||
|
|
||||||
|
The default dev_search_path is: "/".
|
||||||
|
|
||||||
|
dev_search_path option can also contain %SESSNAME% in order to provide
|
||||||
|
different device namespaces for different sessions. See "device_path"
|
||||||
|
option for details.
|
||||||
|
|
||||||
|
============================
|
||||||
|
Protocol (rnbd/rnbd-proto.h)
|
||||||
|
============================
|
||||||
|
|
||||||
|
1. Before mapping first device from a given server, client sends an
|
||||||
|
RNBD_MSG_SESS_INFO to the server. Server responds with
|
||||||
|
RNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol
|
||||||
|
version for backward compatibility.
|
||||||
|
|
||||||
|
2. Client requests to open a device by sending RNBD_MSG_OPEN message. This
|
||||||
|
contains the path to the device and access mode (read-only or writable).
|
||||||
|
Server responds to the message with RNBD_MSG_OPEN_RSP. This contains
|
||||||
|
a 32 bit device id to be used for IOs and device "geometry" related
|
||||||
|
information: side, max_hw_sectors, etc.
|
||||||
|
|
||||||
|
3. Client attaches RNBD_MSG_IO to each IO message send to a device. This
|
||||||
|
message contains device id, provided by server in his rnbd_msg_open_rsp,
|
||||||
|
sector to be accessed, read-write flags and bi_size.
|
||||||
|
|
||||||
|
4. Client closes a device by sending RNBD_MSG_CLOSE which contains only the
|
||||||
|
device id provided by the server.
|
||||||
|
|
||||||
|
=========================================
|
||||||
|
Contributors List(in alphabetical order)
|
||||||
|
=========================================
|
||||||
|
Danil Kipnis <danil.kipnis@profitbricks.com>
|
||||||
|
Fabian Holler <mail@fholler.de>
|
||||||
|
Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
|
||||||
|
Jack Wang <jinpu.wang@profitbricks.com>
|
||||||
|
Kleber Souza <kleber.souza@profitbricks.com>
|
||||||
|
Lutz Pogrell <lutz.pogrell@cloud.ionos.com>
|
||||||
|
Milind Dumbare <Milind.dumbare@gmail.com>
|
||||||
|
Roman Penyaev <roman.penyaev@profitbricks.com>
|
||||||
Loading…
Reference in New Issue
Block a user