diff options
Diffstat (limited to 'Documentation/admin-guide/thunderbolt.rst')
| -rw-r--r-- | Documentation/admin-guide/thunderbolt.rst | 445 |
1 files changed, 445 insertions, 0 deletions
diff --git a/Documentation/admin-guide/thunderbolt.rst b/Documentation/admin-guide/thunderbolt.rst new file mode 100644 index 000000000..ff25fe853 --- /dev/null +++ b/Documentation/admin-guide/thunderbolt.rst @@ -0,0 +1,445 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================== + USB4 and Thunderbolt +====================== +USB4 is the public specification based on Thunderbolt 3 protocol with +some differences at the register level among other things. Connection +manager is an entity running on the host router (host controller) +responsible for enumerating routers and establishing tunnels. A +connection manager can be implemented either in firmware or software. +Typically PCs come with a firmware connection manager for Thunderbolt 3 +and early USB4 capable systems. Apple systems on the other hand use +software connection manager and the later USB4 compliant devices follow +the suit. + +The Linux Thunderbolt driver supports both and can detect at runtime which +connection manager implementation is to be used. To be on the safe side the +software connection manager in Linux also advertises security level +``user`` which means PCIe tunneling is disabled by default. The +documentation below applies to both implementations with the exception that +the software connection manager only supports ``user`` security level and +is expected to be accompanied with an IOMMU based DMA protection. + +Security levels and how to use them +----------------------------------- +The interface presented here is not meant for end users. Instead there +should be a userspace tool that handles all the low-level details, keeps +a database of the authorized devices and prompts users for new connections. + +More details about the sysfs interface for Thunderbolt devices can be +found in Documentation/ABI/testing/sysfs-bus-thunderbolt. + +Those users who just want to connect any device without any sort of +manual work can add following line to +``/etc/udev/rules.d/99-local.rules``:: + + ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" + +This will authorize all devices automatically when they appear. However, +keep in mind that this bypasses the security levels and makes the system +vulnerable to DMA attacks. + +Starting with Intel Falcon Ridge Thunderbolt controller there are 4 +security levels available. Intel Titan Ridge added one more security level +(usbonly). The reason for these is the fact that the connected devices can +be DMA masters and thus read contents of the host memory without CPU and OS +knowing about it. There are ways to prevent this by setting up an IOMMU but +it is not always available for various reasons. + +Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is +treated as another security level (nopcie). + +The security levels are as follows: + + none + All devices are automatically connected by the firmware. No user + approval is needed. In BIOS settings this is typically called + *Legacy mode*. + + user + User is asked whether the device is allowed to be connected. + Based on the device identification information available through + ``/sys/bus/thunderbolt/devices``, the user then can make the decision. + In BIOS settings this is typically called *Unique ID*. + + secure + User is asked whether the device is allowed to be connected. In + addition to UUID the device (if it supports secure connect) is sent + a challenge that should match the expected one based on a random key + written to the ``key`` sysfs attribute. In BIOS settings this is + typically called *One time saved key*. + + dponly + The firmware automatically creates tunnels for Display Port and + USB. No PCIe tunneling is done. In BIOS settings this is + typically called *Display Port Only*. + + usbonly + The firmware automatically creates tunnels for the USB controller and + Display Port in a dock. All PCIe links downstream of the dock are + removed. + + nopcie + PCIe tunneling is disabled/forbidden from the BIOS. Available in some + USB4 systems. + +The current security level can be read from +``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is +the Thunderbolt domain the host controller manages. There is typically +one domain per Thunderbolt host controller. + +If the security level reads as ``user`` or ``secure`` the connected +device must be authorized by the user before PCIe tunnels are created +(e.g the PCIe device appears). + +Each Thunderbolt device plugged in will appear in sysfs under +``/sys/bus/thunderbolt/devices``. The device directory carries +information that can be used to identify the particular device, +including its name and UUID. + +Authorizing devices when security level is ``user`` or ``secure`` +----------------------------------------------------------------- +When a device is plugged in it will appear in sysfs as follows:: + + /sys/bus/thunderbolt/devices/0-1/authorized - 0 + /sys/bus/thunderbolt/devices/0-1/device - 0x8004 + /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter + /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 + /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. + /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff + +The ``authorized`` attribute reads 0 which means no PCIe tunnels are +created yet. The user can authorize the device by simply entering:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized + +This will create the PCIe tunnels and the device is now connected. + +If the device supports secure connect, and the domain security level is +set to ``secure``, it has an additional attribute ``key`` which can hold +a random 32-byte value used for authorization and challenging the device in +future connects:: + + /sys/bus/thunderbolt/devices/0-3/authorized - 0 + /sys/bus/thunderbolt/devices/0-3/device - 0x305 + /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box + /sys/bus/thunderbolt/devices/0-3/key - + /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 + /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron + /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 + +Notice the key is empty by default. + +If the user does not want to use secure connect they can just ``echo 1`` +to the ``authorized`` attribute and the PCIe tunnels will be created in +the same way as in the ``user`` security level. + +If the user wants to use secure connect, the first time the device is +plugged a key needs to be created and sent to the device:: + + # key=$(openssl rand -hex 32) + # echo $key > /sys/bus/thunderbolt/devices/0-3/key + # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized + +Now the device is connected (PCIe tunnels are created) and in addition +the key is stored on the device NVM. + +Next time the device is plugged in the user can verify (challenge) the +device using the same key:: + + # echo $key > /sys/bus/thunderbolt/devices/0-3/key + # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized + +If the challenge the device returns back matches the one we expect based +on the key, the device is connected and the PCIe tunnels are created. +However, if the challenge fails no tunnels are created and error is +returned to the user. + +If the user still wants to connect the device they can either approve +the device without a key or write a new key and write 1 to the +``authorized`` file to get the new key stored on the device NVM. + +De-authorizing devices +---------------------- +It is possible to de-authorize devices by writing ``0`` to their +``authorized`` attribute. This requires support from the connection +manager implementation and can be checked by reading domain +``deauthorization`` attribute. If it reads ``1`` then the feature is +supported. + +When a device is de-authorized the PCIe tunnel from the parent device +PCIe downstream (or root) port to the device PCIe upstream port is torn +down. This is essentially the same thing as PCIe hot-remove and the PCIe +toplogy in question will not be accessible anymore until the device is +authorized again. If there is storage such as NVMe or similar involved, +there is a risk for data loss if the filesystem on that storage is not +properly shut down. You have been warned! + +DMA protection utilizing IOMMU +------------------------------ +Recent systems from 2018 and forward with Thunderbolt ports may natively +support IOMMU. This means that Thunderbolt security is handled by an IOMMU +so connected devices cannot access memory regions outside of what is +allocated for them by drivers. When Linux is running on such system it +automatically enables IOMMU if not enabled by the user already. These +systems can be identified by reading ``1`` from +``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. + +The driver does not do anything special in this case but because DMA +protection is handled by the IOMMU, security levels (if set) are +redundant. For this reason some systems ship with security level set to +``none``. Other systems have security level set to ``user`` in order to +support downgrade to older OS, so users who want to automatically +authorize devices when IOMMU DMA protection is enabled can use the +following ``udev`` rule:: + + ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" + +Upgrading NVM on Thunderbolt device, host or retimer +---------------------------------------------------- +Since most of the functionality is handled in firmware running on a +host controller or a device, it is important that the firmware can be +upgraded to the latest where possible bugs in it have been fixed. +Typically OEMs provide this firmware from their support site. + +Currently, recommended method of updating firmware is through "fwupd" tool. +It uses LVFS (Linux Vendor Firmware Service) portal by default to get the +latest firmware from hardware vendors and updates connected devices if found +compatible. For details refer to: https://github.com/fwupd/fwupd. + +Before you upgrade firmware on a device, host or retimer, please make +sure it is a suitable upgrade. Failing to do that may render the device +in a state where it cannot be used properly anymore without special +tools! + +Host NVM upgrade on Apple Macs is not supported. + +Fwupd is installed by default. If you don't have it on your system, simply +use your distro package manager to get it. + +To see possible updates through fwupd, you need to plug in a Thunderbolt +device so that the host controller appears. It does not matter which +device is connected (unless you are upgrading NVM on a device - then you +need to connect that particular device). + +Note an OEM-specific method to power the controller up ("force power") may +be available for your system in which case there is no need to plug in a +Thunderbolt device. + +Updating firmware using fwupd is straightforward - refer to official +readme on fwupd github. + +If firmware image is written successfully, the device shortly disappears. +Once it comes back, the driver notices it and initiates a full power +cycle. After a while device appears again and this time it should be +fully functional. + +Device of interest should display new version under "Current version" +and "Update State: Success" in fwupd's interface. + +Upgrading firmware manually +--------------------------------------------------------------- +If possible, use fwupd to updated the firmware. However, if your device OEM +has not uploaded the firmware to LVFS, but it is available for download +from their side, you can use method below to directly upgrade the +firmware. + +Manual firmware update can be done with 'dd' tool. To update firmware +using this method, you need to write it to the non-active parts of NVM +of the host or device. Example on how to update Intel NUC6i7KYK +(Skull Canyon) Thunderbolt controller NVM:: + + # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem + +Once the operation completes we can trigger NVM authentication and +upgrade process as follows:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate + +If no errors are returned, device should behave as described in previous +section. + +We can verify that the new NVM firmware is active by running the following +commands:: + + # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate + 0x0 + # cat /sys/bus/thunderbolt/devices/0-0/nvm_version + 18.0 + +If ``nvm_authenticate`` contains anything other than 0x0 it is the error +code from the last authentication cycle, which means the authentication +of the NVM image failed. + +Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` +depend on the order they are registered in the NVMem subsystem. N in +the name is the identifier added by the NVMem subsystem. + +Upgrading on-board retimer NVM when there is no cable connected +--------------------------------------------------------------- +If the platform supports, it may be possible to upgrade the retimer NVM +firmware even when there is nothing connected to the USB4 +ports. When this is the case the ``usb4_portX`` devices have two special +attributes: ``offline`` and ``rescan``. The way to upgrade the firmware +is to first put the USB4 port into offline mode:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + +This step makes sure the port does not respond to any hotplug events, +and also ensures the retimers are powered on. The next step is to scan +for the retimers:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This enumerates and adds the on-board retimers. Now retimer NVM can be +upgraded in the same way than with cable connected (see previous +section). However, the retimer is not disconnected as we are in offline +mode, so after writing ``1`` to ``nvm_authenticate`` one should wait for +5 or more seconds before running rescan again:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This point if everything went fine, the port can be put back to +functional state again:: + + # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + +Upgrading NVM when host controller is in safe mode +-------------------------------------------------- +If the existing NVM is not properly authenticated (or is missing) the +host controller goes into safe mode which means that the only available +functionality is flashing a new NVM image. When in this mode, reading +``nvm_version`` fails with ``ENODATA`` and the device identification +information is missing. + +To recover from this mode, one needs to flash a valid NVM image to the +host controller in the same way it is done in the previous chapter. + +Tunneling events +---------------- +The driver sends ``KOBJ_CHANGE`` events to userspace when there is a +tunneling change in the ``thunderbolt_domain``. The notification carries +following environment variables:: + + TUNNEL_EVENT=<EVENT> + TUNNEL_DETAILS=0:12 <-> 1:20 (USB3) + +Possible values for ``<EVENT>`` are: + + activated + The tunnel was activated (created). + + changed + There is a change in this tunnel. For example bandwidth allocation was + changed. + + deactivated + The tunnel was torn down. + + low bandwidth + The tunnel is not getting optimal bandwidth. + + insufficient bandwidth + There is not enough bandwidth for the current tunnel requirements. + +The ``TUNNEL_DETAILS`` is only provided if the tunnel is known. For +example, in case of Firmware Connection Manager this is missing or does +not provide full tunnel information. In case of Software Connection Manager +this includes full tunnel details. The format currently matches what the +driver uses when logging. This may change over time. + +Networking over Thunderbolt cable +--------------------------------- +Thunderbolt technology allows software communication between two hosts +connected by a Thunderbolt cable. + +It is possible to tunnel any kind of traffic over a Thunderbolt link but +currently we only support Apple ThunderboltIP protocol. + +If the other host is running Windows or macOS, the only thing you need to +do is to connect a Thunderbolt cable between the two hosts; the +``thunderbolt-net`` driver is loaded automatically. If the other host is +also Linux you should load ``thunderbolt-net`` manually on one host (it +does not matter which one):: + + # modprobe thunderbolt-net + +This triggers module load on the other host automatically. If the driver +is built-in to the kernel image, there is no need to do anything. + +The driver will create one virtual ethernet interface per Thunderbolt +port which are named like ``thunderbolt0`` and so on. From this point +you can either use standard userspace tools like ``ip`` to +configure the interface or let your GUI handle it automatically. + +Streaming data directly over Thunderbolt cable +---------------------------------------------- +In addition to Thunderbolt networking (aka. USB4NET) Linux supports +streaming data directly over a cable as well (aka. USB4STREAM). This is +possible through ``thunderbolt-stream`` driver. + +Similarly to ``thunderbolt-net`` you load the driver first on one end:: + + host1 # modprobe thunderbolt-stream + +Then you configure it via ``ConfigFS``:: + + host1 # cd /sys/kernel/config/thunderbolt/stream + host1 # mkdir -p 0-1.0/data + host1 # cd 0-1.0 + host1 # echo -1 > data/in_hopid + host1 # echo -1 > data/out_hopid + +This information is automatically announced to the other side via +XDomain properties so if you have cable connected the other side knows +that there is a stream named ``data`` available and can configure it for +you automatically:: + + host2 # cd /sys/kernel/config/thunderbolt/stream + host2 # mkdir -p 0-3.0/data + +Here we used auto-configuration but you can configure it manually too. +In that case you need to fill ``in_hopid`` and ``out_hopid`` accordingly. +If you set them to ``-1`` the next available HopID is used which is +typically what we want. + +Once they are configured you can use ``/dev/tbstreamX`` on both sides to +transfer data:: + + host2 # cat /dev/tbstream0 + host1 # dmesg > /dev/tbstream0 + +Once you are done with the stream you can remove them:: + + host2 # cd /sys/kernel/config/thunderbolt/stream + host2 # rmdir -p 0-1.0/data + host1 # cd /sys/kernel/config/thunderbolt/stream + host1 # rmdir -p 0-3.0/data + +Since streams are essentially files you can use any existing application +that supports ``read(2)`` and ``write(2)`` in some form. + +It is possible to have more than one stream and you can have both stream +and ``thunderbolt-net`` in use simultaneously. For example we can create +two streams with name ``control`` and ``data`` like this:: + + host1 # cd /sys/kernel/config/thunderbolt/stream + host1 # mkdir 0-1.0 + host1 # cd 0-1.0 + host1 # mkdir control + host1 # mkdir data + +Then you have ``/dev/tbstream0`` for ``control`` and ``/dev/tbstream1`` +for ``data``. Before you can use them you need to configure them as +shown above for the one stream case. + +Forcing power +------------- +Many OEMs include a method that can be used to force the power of a +Thunderbolt controller to an "On" state even if nothing is connected. +If supported by your machine this will be exposed by the WMI bus with +a sysfs attribute called "force_power", see +Documentation/ABI/testing/sysfs-platform-intel-wmi-thunderbolt for details. + +Note: it's currently not possible to query the force power state of a platform. |