From 034dd340b08be1f2f0477ad16131d609f9dbd53c Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Sun, 30 Aug 2026 09:22:00 -0700 Subject: Merge tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing fixes from Steven Rostedt: - Fix error output of boot instance creation failure Currently if a boot instance creation fails, instead of printing out the name of the instance that failed, it prints "(null)". That is because it prints "cur_str" that had already been processed by strsep(). Print the saved name instead. While at it, print the error code of the failure. - Fix use-after-free for same named historgrams Histograms can be named so that they can be used in multiple events. But if the named histogram has a variable attached, the second event that uses the named histogram which duplicates it and needs to free the original after duplication leaves the old variable in place and still visible. If another histogram uses than variable, it will use the stale one which will try to reference the freed duplicate histogram and crash the kernel. Free the duplicate variables along with the duplicated histogram data. - Check return value of kthread_run() in event self test The events self tests uses a kthread for testing but does not check if it succeeded in creating a kthread. If the kthread creation were to fail, the code will still try to call kthread_stop() on the error returned. - Fix race between reading trace_pipe and updating subbuffer size If a user is reading the trace_pipe file at the same time they update the ring buffer sub-buffer size, can cause the trace_pipe read to read stale data. Add trace_access_lock() around updating the ring buffer sub-buffer size. - Fix eventfs_inode on failure path in creation of the events directory In the creation of the "events" directory, if after allocating the eventfs_inode a failure is detected, it calls cleanup_ei() which calls free_ei(). The free_ei() will test if eventfs_inode being freed has no children. It is a bug if it does. But on the failure case of the creation of the "events" directory, the children lists have not yet been initialized and the free will trigger a warning because list_empty() on an uninitialized list returns false. Move the initialization into init_ei() where it makes more sense and makes sure that a created eventfs_inode has its lists initialized upon creation. - Check return value of kthread_run() in ftrace direct sample code The sample code that shows how to use the ftrace direct calls does not test the return of kthread_run() to see if it succeeds. Return a failure if the kthread_run() doesn't succeed. - Clear user events state on fork in case of alloc failure On fork, the child gets a pointer to the parent's user events state. It makes a copy of it then updates the child's pointer to it. But if the allocation fails, the duplication function leaves the child with a pointer to its parent's descriptor. When the child cleans up its data, it will free the parent's descriptor while the parent is still using it. In the duplication function, set the child's user_event_mm to NULL before testing if the allocation succeeded, and when it exits it will not free the parent's descriptor. - Fix retry exhaustion in simple ring buffer reader swap simple_ring_buffer_swap_reader_page() starts with retry set to 8 and post-decrements it only after a failed link replacement. On the final attempt, a successful replacement leaves retry at zero, while a failed replacement leaves it at -1. But the check for success expects the retry value to be non-zero and exits with an error on zero. This is the opposite result. Fix it. - Fail nicely when the remote swap_reader_page() returns an error Currently, if the swap_reader_page() of a remote buffer fails, it triggers a WARN_ON_ONCE() and continues normally. Instead, have it exit with an error and a pr_warn() print instead of a full WARNING. * tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: ring-buffer: Stop remote reader update when page swap fails tracing: Fix retry exhaustion in simple ring buffer reader swap tracing/user_events: Clear copied tracing state before fork duplication samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-multi-modify samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-modify eventfs: Initialize ei->children and ei->list in init_ei() tracing: Fix use-after-free in trace_pipe read on sub-buffer order change tracing: Fix crash passing ERR_PTR to kthread_stop() tracing: Fix use-after-free with same-name named triggers tracing: Fix logged instance name on creation failure --- .../PCI/endpoint/function/binding/pci-ntb.rst | 38 +++ .../PCI/endpoint/function/binding/pci-test.rst | 26 ++ Documentation/PCI/endpoint/index.rst | 21 ++ Documentation/PCI/endpoint/pci-endpoint-cfs.rst | 138 ++++++++ Documentation/PCI/endpoint/pci-endpoint.rst | 283 +++++++++++++++++ Documentation/PCI/endpoint/pci-ntb-function.rst | 348 +++++++++++++++++++++ Documentation/PCI/endpoint/pci-ntb-howto.rst | 158 ++++++++++ Documentation/PCI/endpoint/pci-nvme-function.rst | 13 + Documentation/PCI/endpoint/pci-test-function.rst | 103 ++++++ Documentation/PCI/endpoint/pci-test-howto.rst | 239 ++++++++++++++ Documentation/PCI/endpoint/pci-vntb-function.rst | 129 ++++++++ Documentation/PCI/endpoint/pci-vntb-howto.rst | 169 ++++++++++ 12 files changed, 1665 insertions(+) create mode 100644 Documentation/PCI/endpoint/function/binding/pci-ntb.rst create mode 100644 Documentation/PCI/endpoint/function/binding/pci-test.rst create mode 100644 Documentation/PCI/endpoint/index.rst create mode 100644 Documentation/PCI/endpoint/pci-endpoint-cfs.rst create mode 100644 Documentation/PCI/endpoint/pci-endpoint.rst create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/PCI/endpoint/pci-test-function.rst create mode 100644 Documentation/PCI/endpoint/pci-test-howto.rst create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst (limited to 'Documentation/PCI/endpoint') diff --git a/Documentation/PCI/endpoint/function/binding/pci-ntb.rst b/Documentation/PCI/endpoint/function/binding/pci-ntb.rst new file mode 100644 index 000000000..40253d3d5 --- /dev/null +++ b/Documentation/PCI/endpoint/function/binding/pci-ntb.rst @@ -0,0 +1,38 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +PCI NTB Endpoint Function +========================== + +1) Create a subdirectory to pci_epf_ntb directory in configfs. + +Standard EPF Configurable Fields: + +================ =========================================================== +vendorid should be 0x104c +deviceid should be 0xb00d for TI's J721E SoC +revid don't care +progif_code don't care +subclass_code should be 0x00 +baseclass_code should be 0x5 +cache_line_size don't care +subsys_vendor_id don't care +subsys_id don't care +interrupt_pin don't care +msi_interrupts don't care +msix_interrupts don't care +================ =========================================================== + +2) Create a subdirectory to directory created in 1 + +NTB EPF specific configurable fields: + +================ =========================================================== +db_count Number of doorbells; default = 4 +mw1 size of memory window1 +mw2 size of memory window2 +mw3 size of memory window3 +mw4 size of memory window4 +num_mws Number of memory windows; max = 4 +spad_count Number of scratchpad registers; default = 64 +================ =========================================================== diff --git a/Documentation/PCI/endpoint/function/binding/pci-test.rst b/Documentation/PCI/endpoint/function/binding/pci-test.rst new file mode 100644 index 000000000..57ee866fb --- /dev/null +++ b/Documentation/PCI/endpoint/function/binding/pci-test.rst @@ -0,0 +1,26 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +PCI Test Endpoint Function +========================== + +name: Should be "pci_epf_test" to bind to the pci_epf_test driver. + +Configurable Fields: + +================ =========================================================== +vendorid should be 0x104c +deviceid should be 0xb500 for DRA74x and 0xb501 for DRA72x +revid don't care +progif_code don't care +subclass_code don't care +baseclass_code should be 0xff +cache_line_size don't care +subsys_vendor_id don't care +subsys_id don't care +interrupt_pin Should be 1 - INTA, 2 - INTB, 3 - INTC, 4 -INTD +msi_interrupts Should be 1 to 32 depending on the number of MSI interrupts + to test +msix_interrupts Should be 1 to 2048 depending on the number of MSI-X + interrupts to test +================ =========================================================== diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst new file mode 100644 index 000000000..dd1f62e73 --- /dev/null +++ b/Documentation/PCI/endpoint/index.rst @@ -0,0 +1,21 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================== +PCI Endpoint Framework +====================== + +.. toctree:: + :maxdepth: 2 + + pci-endpoint + pci-endpoint-cfs + pci-test-function + pci-test-howto + pci-ntb-function + pci-ntb-howto + pci-vntb-function + pci-vntb-howto + pci-nvme-function + + function/binding/pci-test + function/binding/pci-ntb diff --git a/Documentation/PCI/endpoint/pci-endpoint-cfs.rst b/Documentation/PCI/endpoint/pci-endpoint-cfs.rst new file mode 100644 index 000000000..e69c2872c --- /dev/null +++ b/Documentation/PCI/endpoint/pci-endpoint-cfs.rst @@ -0,0 +1,138 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================= +Configuring PCI Endpoint Using CONFIGFS +======================================= + +:Author: Kishon Vijay Abraham I + +The PCI Endpoint Core exposes configfs entry (pci_ep) to configure the +PCI endpoint function and to bind the endpoint function +with the endpoint controller. (For introducing other mechanisms to +configure the PCI Endpoint Function refer to [1]). + +Mounting configfs +================= + +The PCI Endpoint Core layer creates pci_ep directory in the mounted configfs +directory. configfs can be mounted using the following command:: + + mount -t configfs none /sys/kernel/config + +Directory Structure +=================== + +The pci_ep configfs has two directories at its root: controllers and +functions. Every EPC device present in the system will have an entry in +the *controllers* directory and every EPF driver present in the system +will have an entry in the *functions* directory. +:: + + /sys/kernel/config/pci_ep/ + .. controllers/ + .. functions/ + +Creating EPF Device +=================== + +Every registered EPF driver will be listed in controllers directory. The +entries corresponding to EPF driver will be created by the EPF core. +:: + + /sys/kernel/config/pci_ep/functions/ + .. / + ... / + ... / + ... / + .. / + ... / + ... / + +In order to create a of the type probed by , the +user has to create a directory inside . + +Every directory consists of the following entries that can be +used to configure the standard configuration header of the endpoint function. +(These entries are created by the framework when any new is +created) +:: + + .. / + ... / + ... vendorid + ... deviceid + ... revid + ... progif_code + ... subclass_code + ... baseclass_code + ... cache_line_size + ... subsys_vendor_id + ... subsys_id + ... interrupt_pin + ... / + ... primary/ + ... / + ... secondary/ + ... / + +If an EPF device has to be associated with 2 EPCs (like in the case of +Non-transparent bridge), symlink of endpoint controller connected to primary +interface should be added in 'primary' directory and symlink of endpoint +controller connected to secondary interface should be added in 'secondary' +directory. + +The directory can have a list of symbolic links +() to other . These symbolic links should +be created by the user to represent the virtual functions that are bound to +the physical function. In the above directory structure is a +physical function and is a virtual function. An EPF device once +it's linked to another EPF device, cannot be linked to an EPC device. + +EPC Device +========== + +Every registered EPC device will be listed in controllers directory. The +entries corresponding to EPC device will be created by the EPC core. +:: + + /sys/kernel/config/pci_ep/controllers/ + .. / + ... / + ... / + ... start + .. / + ... / + ... / + ... start + +The directory will have a list of symbolic links to +. These symbolic links should be created by the user to +represent the functions present in the endpoint device. Only +that represents a physical function can be linked to an EPC device. + +The directory will also have a *start* field. Once +"1" is written to this field, the endpoint device will be ready to +establish the link with the host. This is usually done after +all the EPF devices are created and linked with the EPC device. +:: + + | controllers/ + | / + | + | start + | functions/ + | / + | / + | vendorid + | deviceid + | revid + | progif_code + | subclass_code + | baseclass_code + | cache_line_size + | subsys_vendor_id + | subsys_id + | interrupt_pin + | function + +[1] Documentation/PCI/endpoint/pci-endpoint.rst diff --git a/Documentation/PCI/endpoint/pci-endpoint.rst b/Documentation/PCI/endpoint/pci-endpoint.rst new file mode 100644 index 000000000..4697377ad --- /dev/null +++ b/Documentation/PCI/endpoint/pci-endpoint.rst @@ -0,0 +1,283 @@ +.. SPDX-License-Identifier: GPL-2.0 + +:Author: Kishon Vijay Abraham I + +This document is a guide to use the PCI Endpoint Framework in order to create +endpoint controller driver, endpoint function driver, and using configfs +interface to bind the function driver to the controller driver. + +Introduction +============ + +Linux has a comprehensive PCI subsystem to support PCI controllers that +operates in Root Complex mode. The subsystem has capability to scan PCI bus, +assign memory resources and IRQ resources, load PCI driver (based on +vendor ID, device ID), support other services like hot-plug, power management, +advanced error reporting and virtual channels. + +However the PCI controller IP integrated in some SoCs is capable of operating +either in Root Complex mode or Endpoint mode. PCI Endpoint Framework will +add endpoint mode support in Linux. This will help to run Linux in an +EP system which can have a wide variety of use cases from testing or +validation, co-processor accelerator, etc. + +PCI Endpoint Core +================= + +The PCI Endpoint Core layer comprises 3 components: the Endpoint Controller +library, the Endpoint Function library, and the configfs layer to bind the +endpoint function with the endpoint controller. + +PCI Endpoint Controller(EPC) Library +------------------------------------ + +The EPC library provides APIs to be used by the controller that can operate +in endpoint mode. It also provides APIs to be used by function driver/library +in order to implement a particular endpoint function. + +APIs for the PCI controller Driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section lists the APIs that the PCI Endpoint core provides to be used +by the PCI controller driver. + +* devm_pci_epc_create()/pci_epc_create() + + The PCI controller driver should implement the following ops: + + * write_header: ops to populate configuration space header + * set_bar: ops to configure the BAR + * clear_bar: ops to reset the BAR + * alloc_addr_space: ops to allocate in PCI controller address space + * free_addr_space: ops to free the allocated address space + * raise_irq: ops to raise a legacy, MSI or MSI-X interrupt + * start: ops to start the PCI link + * stop: ops to stop the PCI link + + The PCI controller driver can then create a new EPC device by invoking + devm_pci_epc_create()/pci_epc_create(). + +* pci_epc_destroy() + + The PCI controller driver can destroy the EPC device created by + pci_epc_create() using pci_epc_destroy(). + +* pci_epc_linkup() + + In order to notify all the function devices that the EPC device to which + they are linked has established a link with the host, the PCI controller + driver should invoke pci_epc_linkup(). + +* pci_epc_mem_init() + + Initialize the pci_epc_mem structure used for allocating EPC addr space. + +* pci_epc_mem_exit() + + Cleanup the pci_epc_mem structure allocated during pci_epc_mem_init(). + + +EPC APIs for the PCI Endpoint Function Driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section lists the APIs that the PCI Endpoint core provides to be used +by the PCI endpoint function driver. + +* pci_epc_write_header() + + The PCI endpoint function driver should use pci_epc_write_header() to + write the standard configuration header to the endpoint controller. + +* pci_epc_set_bar() + + The PCI endpoint function driver should use pci_epc_set_bar() to configure + the Base Address Register in order for the host to assign PCI addr space. + Register space of the function driver is usually configured + using this API. + + Some endpoint controllers also support calling pci_epc_set_bar() again + for the same BAR (without calling pci_epc_clear_bar()) to update inbound + address translations after the host has programmed the BAR base address. + Endpoint function drivers can check this capability via the + dynamic_inbound_mapping EPC feature bit. + + When pci_epf_bar.num_submap is non-zero, the endpoint function driver is + requesting BAR subrange mapping using pci_epf_bar.submap. This requires + the EPC to advertise support via the subrange_mapping EPC feature bit. + + When an EPF driver wants to make use of the inbound subrange mapping + feature, it requires that the BAR base address has been programmed by + the host during enumeration. Thus, it needs to call pci_epc_set_bar() + twice for the same BAR (requires dynamic_inbound_mapping): first with + num_submap set to zero and configuring the BAR size, then after the PCIe + link is up and the host enumerates the endpoint and programs the BAR + base address, again with num_submap set to non-zero value. + + Note that when making use of the inbound subrange mapping feature, the + EPF driver must not call pci_epc_clear_bar() between the two + pci_epc_set_bar() calls, because clearing the BAR can clear/disable the + BAR register or BAR decode on the endpoint while the host still expects + the assigned BAR address to remain valid. + +* pci_epc_clear_bar() + + The PCI endpoint function driver should use pci_epc_clear_bar() to reset + the BAR. + +* pci_epc_raise_irq() + + The PCI endpoint function driver should use pci_epc_raise_irq() to raise + Legacy Interrupt, MSI or MSI-X Interrupt. + +* pci_epc_mem_alloc_addr() + + The PCI endpoint function driver should use pci_epc_mem_alloc_addr(), to + allocate memory address from EPC addr space which is required to access + RC's buffer + +* pci_epc_mem_free_addr() + + The PCI endpoint function driver should use pci_epc_mem_free_addr() to + free the memory space allocated using pci_epc_mem_alloc_addr(). + +* pci_epc_map_addr() + + A PCI endpoint function driver should use pci_epc_map_addr() to map to a RC + PCI address the CPU address of local memory obtained with + pci_epc_mem_alloc_addr(). + +* pci_epc_unmap_addr() + + A PCI endpoint function driver should use pci_epc_unmap_addr() to unmap the + CPU address of local memory mapped to a RC address with pci_epc_map_addr(). + +* pci_epc_mem_map() + + A PCI endpoint controller may impose constraints on the RC PCI addresses that + can be mapped. The function pci_epc_mem_map() allows endpoint function + drivers to allocate and map controller memory while handling such + constraints. This function will determine the size of the memory that must be + allocated with pci_epc_mem_alloc_addr() for successfully mapping a RC PCI + address range. This function will also indicate the size of the PCI address + range that was actually mapped, which can be less than the requested size, as + well as the offset into the allocated memory to use for accessing the mapped + RC PCI address range. + +* pci_epc_mem_unmap() + + A PCI endpoint function driver can use pci_epc_mem_unmap() to unmap and free + controller memory that was allocated and mapped using pci_epc_mem_map(). + + +Other EPC APIs +~~~~~~~~~~~~~~ + +There are other APIs provided by the EPC library. These are used for binding +the EPF device with EPC device. pci-ep-cfs.c can be used as reference for +using these APIs. + +* pci_epc_get() + + Get a reference to the PCI endpoint controller based on the device name of + the controller. + +* pci_epc_put() + + Release the reference to the PCI endpoint controller obtained using + pci_epc_get() + +* pci_epc_add_epf() + + Add a PCI endpoint function to a PCI endpoint controller. A PCIe device + can have up to 8 functions according to the specification. + +* pci_epc_remove_epf() + + Remove the PCI endpoint function from PCI endpoint controller. + +* pci_epc_start() + + The PCI endpoint function driver should invoke pci_epc_start() once it + has configured the endpoint function and wants to start the PCI link. + +* pci_epc_stop() + + The PCI endpoint function driver should invoke pci_epc_stop() to stop + the PCI LINK. + + +PCI Endpoint Function(EPF) Library +---------------------------------- + +The EPF library provides APIs to be used by the function driver and the EPC +library to provide endpoint mode functionality. + +EPF APIs for the PCI Endpoint Function Driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section lists the APIs that the PCI Endpoint core provides to be used +by the PCI endpoint function driver. + +* pci_epf_register_driver() + + The PCI Endpoint Function driver should implement the following ops: + * bind: ops to perform when an EPC device has been bound to EPF device + * unbind: ops to perform when a binding has been lost between an EPC + device and EPF device + * add_cfs: optional ops to create function specific configfs + attributes + + The PCI Function driver can then register the PCI EPF driver by using + pci_epf_register_driver(). + +* pci_epf_unregister_driver() + + The PCI Function driver can unregister the PCI EPF driver by using + pci_epf_unregister_driver(). + +* pci_epf_alloc_space() + + The PCI Function driver can allocate space for a particular BAR using + pci_epf_alloc_space(). + +* pci_epf_free_space() + + The PCI Function driver can free the allocated space + (using pci_epf_alloc_space) by invoking pci_epf_free_space(). + +APIs for the PCI Endpoint Controller Library +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section lists the APIs that the PCI Endpoint core provides to be used +by the PCI endpoint controller library. + +* pci_epf_linkup() + + The PCI endpoint controller library invokes pci_epf_linkup() when the + EPC device has established the connection to the host. + +Other EPF APIs +~~~~~~~~~~~~~~ + +There are other APIs provided by the EPF library. These are used to notify +the function driver when the EPF device is bound to the EPC device. +pci-ep-cfs.c can be used as reference for using these APIs. + +* pci_epf_create() + + Create a new PCI EPF device by passing the name of the PCI EPF device. + This name will be used to bind the EPF device to a EPF driver. + +* pci_epf_destroy() + + Destroy the created PCI EPF device. + +* pci_epf_bind() + + pci_epf_bind() should be invoked when the EPF device has been bound to + an EPC device. + +* pci_epf_unbind() + + pci_epf_unbind() should be invoked when the binding between EPC device + and EPF device is lost. diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst new file mode 100644 index 000000000..3b9d836a4 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst @@ -0,0 +1,348 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NTB Function +================= + +:Author: Kishon Vijay Abraham I + +PCI Non-Transparent Bridges (NTB) allow two host systems to communicate +with each other by exposing each host as a device to the other host. +NTBs typically support the ability to generate interrupts on the remote +machine, expose memory ranges as BARs, and perform DMA. They also support +scratchpads, which are areas of memory within the NTB that are accessible +from both machines. + +PCI NTB Function allows two different systems (or hosts) to communicate +with each other by configuring the endpoint instances in such a way that +transactions from one system are routed to the other system. + +In the below diagram, PCI NTB function configures the SoC with multiple +PCI Endpoint (EP) instances in such a way that transactions from one EP +controller are routed to the other EP controller. Once PCI NTB function +configures the SoC with multiple EP instances, HOST1 and HOST2 can +communicate with each other using SoC as a bridge. + +.. code-block:: text + + +-------------+ +-------------+ + | | | | + | HOST1 | | HOST2 | + | | | | + +------^------+ +------^------+ + | | + | | + +---------|-------------------------------------------------|---------+ + | +------v------+ +------v------+ | + | | | | | | + | | EP | | EP | | + | | CONTROLLER1 | | CONTROLLER2 | | + | | <-----------------------------------> | | + | | | | | | + | | | | | | + | | | SoC With Multiple EP Instances | | | + | | | (Configured using NTB Function) | | | + | +-------------+ +-------------+ | + +---------------------------------------------------------------------+ + +Constructs used for Implementing NTB +==================================== + + 1) Config Region + 2) Self Scratchpad Registers + 3) Peer Scratchpad Registers + 4) Doorbell (DB) Registers + 5) Memory Window (MW) + + +Config Region: +-------------- + +Config Region is a construct that is specific to NTB implemented using NTB +Endpoint Function Driver. The host and endpoint side NTB function driver will +exchange information with each other using this region. Config Region has +Control/Status Registers for configuring the Endpoint Controller. Host can +write into this region for configuring the outbound Address Translation Unit +(ATU) and to indicate the link status. Endpoint can indicate the status of +commands issued by host in this region. Endpoint can also indicate the +scratchpad offset and number of memory windows to the host using this region. + +The format of Config Region is given below. All the fields here are 32 bits. + +.. code-block:: text + + +------------------------+ + | COMMAND | + +------------------------+ + | ARGUMENT | + +------------------------+ + | STATUS | + +------------------------+ + | TOPOLOGY | + +------------------------+ + | ADDRESS (LOWER 32) | + +------------------------+ + | ADDRESS (UPPER 32) | + +------------------------+ + | SIZE | + +------------------------+ + | NO OF MEMORY WINDOW | + +------------------------+ + | MEMORY WINDOW1 OFFSET | + +------------------------+ + | SPAD OFFSET | + +------------------------+ + | SPAD COUNT | + +------------------------+ + | DB ENTRY SIZE | + +------------------------+ + | DB DATA | + +------------------------+ + | : | + +------------------------+ + | : | + +------------------------+ + | DB DATA | + +------------------------+ + + + COMMAND: + + NTB function supports three commands: + + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before + invoking this command, the host should allocate and initialize + MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the + Endpoint). The endpoint on receiving this command will configure + the outbound ATU such that transactions to Doorbell BAR will be routed + to the MSI/MSI-X address programmed by the host. The ARGUMENT + register should be populated with number of DBs to configure (in the + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). + + CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The + host invokes this command after allocating a buffer that can be + accessed by remote host. The allocated address should be programmed + in the ADDRESS register (64 bit), the size should be programmed in + the SIZE register and the memory window index should be programmed + in the ARGUMENT register. The endpoint on receiving this command + will configure the outbound ATU such that transactions to MW BAR + are routed to the address provided by the host. + + CMD_LINK_UP (0x3): Command to indicate an NTB application is + bound to the EP device on the host side. Once the endpoint + receives this command from both the hosts, the endpoint will + raise a LINK_UP event to both the hosts to indicate the host + NTB applications can start communicating with each other. + + ARGUMENT: + + The value of this register is based on the commands issued in + command register. See COMMAND section for more information. + + TOPOLOGY: + + Set to NTB_TOPO_B2B_USD for Primary interface + Set to NTB_TOPO_B2B_DSD for Secondary interface + + ADDRESS/SIZE: + + Address and Size to be used while configuring the memory window. + See "CMD_CONFIGURE_MW" for more info. + + MEMORY WINDOW1 OFFSET: + + Memory Window 1 and Doorbell registers are packed together in the + same BAR. The initial portion of the region will have doorbell + registers and the latter portion of the region is for memory window 1. + This register will specify the offset of the memory window 1. + + NO OF MEMORY WINDOW: + + Specifies the number of memory windows supported by the NTB device. + + SPAD OFFSET: + + Self scratchpad region and config region are packed together in the + same BAR. The initial portion of the region will have config region + and the latter portion of the region is for self scratchpad. This + register will specify the offset of the self scratchpad registers. + + SPAD COUNT: + + Specifies the number of scratchpad registers supported by the NTB + device. + + DB ENTRY SIZE: + + Used to determine the offset within the DB BAR that should be written + in order to raise doorbell. EPF NTB can use either MSI or MSI-X to + ring doorbell (MSI-X support will be added later). MSI uses same + address for all the interrupts and MSI-X can provide different + addresses for different interrupts. The MSI/MSI-X address is provided + by the host and the address it gives is based on the MSI/MSI-X + implementation supported by the host. For instance, ARM platform + using GIC ITS will have the same MSI-X address for all the interrupts. + In order to support all the combinations and use the same mechanism + for both MSI and MSI-X, EPF NTB allocates a separate region in the + Outbound Address Space for each of the interrupts. This region will + be mapped to the MSI/MSI-X address provided by the host. If a host + provides the same address for all the interrupts, all the regions + will be translated to the same address. If a host provides different + addresses, the regions will be translated to different addresses. This + will ensure there is no difference while raising the doorbell. + + DB DATA: + + EPF NTB supports 32 interrupts, so there are 32 DB DATA registers. + This holds the MSI/MSI-X data that has to be written to MSI address + for raising doorbell interrupt. This will be populated by EPF NTB + while invoking CMD_CONFIGURE_DOORBELL. + +Scratchpad Registers: +--------------------- + + Each host has its own register space allocated in the memory of NTB endpoint + controller. They are both readable and writable from both sides of the bridge. + They are used by applications built over NTB and can be used to pass control + and status information between both sides of a device. + + Scratchpad registers has 2 parts + 1) Self Scratchpad: Host's own register space + 2) Peer Scratchpad: Remote host's register space. + +Doorbell Registers: +------------------- + + Doorbell Registers are used by the hosts to interrupt each other. + +Memory Window: +-------------- + + Actual transfer of data between the two hosts will happen using the + memory window. + +Modeling Constructs: +==================== + +There are 5 or more distinct regions (config, self scratchpad, peer +scratchpad, doorbell, one or more memory windows) to be modeled to achieve +NTB functionality. At least one memory window is required while more than +one is permitted. All these regions should be mapped to BARs for hosts to +access these regions. + +If one 32-bit BAR is allocated for each of these regions, the scheme would +look like this: + +====== =============== +BAR NO CONSTRUCTS USED +====== =============== +BAR0 Config Region +BAR1 Self Scratchpad +BAR2 Peer Scratchpad +BAR3 Doorbell +BAR4 Memory Window 1 +BAR5 Memory Window 2 +====== =============== + +However if we allocate a separate BAR for each of the regions, there would not +be enough BARs for all the regions in a platform that supports only 64-bit +BARs. + +In order to be supported by most of the platforms, the regions should be +packed and mapped to BARs in a way that provides NTB functionality and +also makes sure the host doesn't access any region that it is not supposed +to. + +The following scheme is used in EPF NTB Function: + +====== =============================== +BAR NO CONSTRUCTS USED +====== =============================== +BAR0 Config Region + Self Scratchpad +BAR1 Peer Scratchpad +BAR2 Doorbell + Memory Window 1 +BAR3 Memory Window 2 +BAR4 Memory Window 3 +BAR5 Memory Window 4 +====== =============================== + +With this scheme, for the basic NTB functionality 3 BARs should be sufficient. + +Modeling Config/Scratchpad Region: +---------------------------------- + +.. code-block:: text + + +-----------------+------->+------------------+ +-----------------+ + | BAR0 | | CONFIG REGION | | BAR0 | + +-----------------+----+ +------------------+<-------+-----------------+ + | BAR1 | | |SCRATCHPAD REGION | | BAR1 | + +-----------------+ +-->+------------------+<-------+-----------------+ + | BAR2 | Local Memory | BAR2 | + +-----------------+ +-----------------+ + | BAR3 | | BAR3 | + +-----------------+ +-----------------+ + | BAR4 | | BAR4 | + +-----------------+ +-----------------+ + | BAR5 | | BAR5 | + +-----------------+ +-----------------+ + EP CONTROLLER 1 EP CONTROLLER 2 + +Above diagram shows Config region + Scratchpad region for HOST1 (connected to +EP controller 1) allocated in local memory. The HOST1 can access the config +region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. +The peer host (HOST2 connected to EP controller 2) can also access this +scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This +diagram shows the case where Config region and Scratchpad regions are allocated +for HOST1, however the same is applicable for HOST2. + +Modeling Doorbell/Memory Window 1: +---------------------------------- + +.. code-block:: text + + +-----------------+ +----->+----------------+-----------+-----------------+ + | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 | + +-----------------+ | +----------------+ +-----------------+ + | BAR1 | | | Doorbell 2 +---------+ | | + +-----------------+----+ +----------------+ | | | + | BAR2 | | Doorbell 3 +-------+ | +-----------------+ + +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 | + | BAR3 | | | Doorbell 4 +-----+ | +-----------------+ + +-----------------+ | |----------------+ | | | | + | BAR4 | | | | | | +-----------------+ + +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3|| + | BAR5 | | | | | | +-----------------+ + +-----------------+ +----->-----------------+ | | | | + EP CONTROLLER 1 | | | | +-----------------+ + | | | +---->+ MSI-X ADDRESS 4 | + +----------------+ | +-----------------+ + EP CONTROLLER 2 | | | + (OB SPACE) | | | + +-------> MW1 | + | | + | | + +-----------------+ + | | + | | + | | + | | + | | + +-----------------+ + PCI Address Space + (Managed by HOST2) + +Above diagram shows how the doorbell and memory window 1 is mapped so that +HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access +buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and +memory window 1 regions are allocated in EP controller 2 outbound (OB) address +space. Allocating and configuring BARs for doorbell and memory window1 +is done during the initialization phase of NTB endpoint function driver. +Mapping from EP controller 2 OB space to PCI address space is done when HOST2 +sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. + +Modeling Optional Memory Windows: +--------------------------------- + +This is modeled the same was as MW1 but each of the additional memory windows +is mapped to separate BARs. diff --git a/Documentation/PCI/endpoint/pci-ntb-howto.rst b/Documentation/PCI/endpoint/pci-ntb-howto.rst new file mode 100644 index 000000000..4261e7157 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-ntb-howto.rst @@ -0,0 +1,158 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================================== +PCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide +=================================================================== + +:Author: Kishon Vijay Abraham I + +This document is a guide to help users use pci-epf-ntb function driver +and ntb_hw_epf host driver for NTB functionality. The list of steps to +be followed in the host side and EP side is given below. For the hardware +configuration and internals of NTB using configurable endpoints see +Documentation/PCI/endpoint/pci-ntb-function.rst + +Endpoint Device +=============== + +Endpoint Controller Devices +--------------------------- + +For implementing NTB functionality at least two endpoint controller devices +are required. + +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + 2900000.pcie-ep 2910000.pcie-ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + 2900000.pcie-ep 2910000.pcie-ep + + +Endpoint Function Drivers +------------------------- + +To find the list of endpoint function drivers in the system:: + + # ls /sys/bus/pci-epf/drivers + pci_epf_ntb pci_epf_ntb + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/functions + pci_epf_ntb pci_epf_ntb + + +Creating pci-epf-ntb Device +---------------------------- + +PCI endpoint function device can be created using the configfs. To create +pci-epf-ntb device, the following commands can be used:: + + # mount -t configfs none /sys/kernel/config + # cd /sys/kernel/config/pci_ep/ + # mkdir functions/pci_epf_ntb/func1 + +The "mkdir func1" above creates the pci-epf-ntb function device that will +be probed by pci_epf_ntb driver. + +The PCI endpoint framework populates the directory with the following +configurable fields:: + + # ls functions/pci_epf_ntb/func1 + baseclass_code deviceid msi_interrupts pci-epf-ntb.0 + progif_code secondary subsys_id vendorid + cache_line_size interrupt_pin msix_interrupts primary + revid subclass_code subsys_vendor_id + +The PCI endpoint function driver populates these entries with default values +when the device is bound to the driver. The pci-epf-ntb driver populates +vendorid with 0xffff and interrupt_pin with 0x0001:: + + # cat functions/pci_epf_ntb/func1/vendorid + 0xffff + # cat functions/pci_epf_ntb/func1/interrupt_pin + 0x0001 + + +Configuring pci-epf-ntb Device +------------------------------- + +The user can configure the pci-epf-ntb device using its configfs entry. In order +to change the vendorid and the deviceid, the following +commands can be used:: + + # echo 0x104c > functions/pci_epf_ntb/func1/vendorid + # echo 0xb00d > functions/pci_epf_ntb/func1/deviceid + +The PCI endpoint framework also automatically creates a sub-directory in the +function attribute directory. This sub-directory has the same name as the name +of the function device and is populated with the following NTB specific +attributes that can be configured by the user:: + + # ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/ + db_count mw1 mw2 mw3 mw4 num_mws + spad_count + +A sample configuration for NTB function is given below:: + + # echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count + # echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count + # echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1 + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2 + +Binding pci-epf-ntb Device to EP Controller +-------------------------------------------- + +NTB function device should be attached to two PCI endpoint controllers +connected to the two hosts. Use the 'primary' and 'secondary' entries +inside NTB function device to attach one PCI endpoint controller to +primary interface and the other PCI endpoint controller to the secondary +interface:: + + # ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary + # ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary + +Once the above step is completed, both the PCI endpoint controllers are ready to +establish a link with the host. + + +Start the Link +-------------- + +In order for the endpoint device to establish a link with the host, the _start_ +field should be populated with '1'. For NTB, both the PCI endpoint controllers +should establish link with the host:: + + # echo 1 > controllers/2900000.pcie-ep/start + # echo 1 > controllers/2910000.pcie-ep/start + + +RootComplex Device +================== + +lspci Output +------------ + +Note that the devices listed here correspond to the values populated in +"Creating pci-epf-ntb Device" section above:: + + # lspci + 0000:00:00.0 PCI bridge: Texas Instruments Device b00d + 0000:01:00.0 RAM memory: Texas Instruments Device b00d + + +Using ntb_hw_epf Device +----------------------- + +The host side software follows the standard NTB software architecture in Linux. +All the existing client side NTB utilities like NTB Transport Client and NTB +Netdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB +function device. + +For more information on NTB see +:doc:`Non-Transparent Bridge <../../driver-api/ntb>` diff --git a/Documentation/PCI/endpoint/pci-nvme-function.rst b/Documentation/PCI/endpoint/pci-nvme-function.rst new file mode 100644 index 000000000..a68015317 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-nvme-function.rst @@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NVMe Function +================= + +:Author: Damien Le Moal + +The PCI NVMe endpoint function implements a PCI NVMe controller using the NVMe +subsystem target core code. The driver for this function resides with the NVMe +subsystem as drivers/nvme/target/pci-epf.c. + +See Documentation/nvme/nvme-pci-endpoint-target.rst for more details. diff --git a/Documentation/PCI/endpoint/pci-test-function.rst b/Documentation/PCI/endpoint/pci-test-function.rst new file mode 100644 index 000000000..3c8521d7a --- /dev/null +++ b/Documentation/PCI/endpoint/pci-test-function.rst @@ -0,0 +1,103 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI Test Function +================= + +:Author: Kishon Vijay Abraham I + +Traditionally PCI RC has always been validated by using standard +PCI cards like ethernet PCI cards or USB PCI cards or SATA PCI cards. +However with the addition of EP-core in linux kernel, it is possible +to configure a PCI controller that can operate in EP mode to work as +a test device. + +The PCI endpoint test device is a virtual device (defined in software) +used to test the endpoint functionality and serve as a sample driver +for other PCI endpoint devices (to use the EP framework). + +The PCI endpoint test device has the following registers: + + 1) PCI_ENDPOINT_TEST_MAGIC + 2) PCI_ENDPOINT_TEST_COMMAND + 3) PCI_ENDPOINT_TEST_STATUS + 4) PCI_ENDPOINT_TEST_SRC_ADDR + 5) PCI_ENDPOINT_TEST_DST_ADDR + 6) PCI_ENDPOINT_TEST_SIZE + 7) PCI_ENDPOINT_TEST_CHECKSUM + 8) PCI_ENDPOINT_TEST_IRQ_TYPE + 9) PCI_ENDPOINT_TEST_IRQ_NUMBER + +* PCI_ENDPOINT_TEST_MAGIC + +This register will be used to test BAR0. A known pattern will be written +and read back from MAGIC register to verify BAR0. + +* PCI_ENDPOINT_TEST_COMMAND + +This register will be used by the host driver to indicate the function +that the endpoint device must perform. + +======== ================================================================ +Bitfield Description +======== ================================================================ +Bit 0 raise legacy IRQ +Bit 1 raise MSI IRQ +Bit 2 raise MSI-X IRQ +Bit 3 read command (read data from RC buffer) +Bit 4 write command (write data to RC buffer) +Bit 5 copy command (copy data from one RC buffer to another RC buffer) +======== ================================================================ + +* PCI_ENDPOINT_TEST_STATUS + +This register reflects the status of the PCI endpoint device. + +======== ============================== +Bitfield Description +======== ============================== +Bit 0 read success +Bit 1 read fail +Bit 2 write success +Bit 3 write fail +Bit 4 copy success +Bit 5 copy fail +Bit 6 IRQ raised +Bit 7 source address is invalid +Bit 8 destination address is invalid +======== ============================== + +* PCI_ENDPOINT_TEST_SRC_ADDR + +This register contains the source address (RC buffer address) for the +COPY/READ command. + +* PCI_ENDPOINT_TEST_DST_ADDR + +This register contains the destination address (RC buffer address) for +the COPY/WRITE command. + +* PCI_ENDPOINT_TEST_IRQ_TYPE + +This register contains the interrupt type (Legacy/MSI) triggered +for the READ/WRITE/COPY and raise IRQ (Legacy/MSI) commands. + +Possible types: + +====== == +Legacy 0 +MSI 1 +MSI-X 2 +====== == + +* PCI_ENDPOINT_TEST_IRQ_NUMBER + +This register contains the triggered ID interrupt. + +Admissible values: + +====== =========== +Legacy 0 +MSI [1 .. 32] +MSI-X [1 .. 2048] +====== =========== diff --git a/Documentation/PCI/endpoint/pci-test-howto.rst b/Documentation/PCI/endpoint/pci-test-howto.rst new file mode 100644 index 000000000..a822866b1 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-test-howto.rst @@ -0,0 +1,239 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================== +PCI Test User Guide +=================== + +:Author: Kishon Vijay Abraham I + +This document is a guide to help users use pci-epf-test function driver +and pci_endpoint_test host driver for testing PCI. The list of steps to +be followed in the host side and EP side is given below. + +Endpoint Device +=============== + +Endpoint Controller Devices +--------------------------- + +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + 51000000.pcie_ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + 51000000.pcie_ep + + +Endpoint Function Drivers +------------------------- + +To find the list of endpoint function drivers in the system:: + + # ls /sys/bus/pci-epf/drivers + pci_epf_test + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/functions + pci_epf_test + + +Creating pci-epf-test Device +---------------------------- + +PCI endpoint function device can be created using the configfs. To create +pci-epf-test device, the following commands can be used:: + + # mount -t configfs none /sys/kernel/config + # cd /sys/kernel/config/pci_ep/ + # mkdir functions/pci_epf_test/func1 + +The "mkdir func1" above creates the pci-epf-test function device that will +be probed by pci_epf_test driver. + +The PCI endpoint framework populates the directory with the following +configurable fields:: + + # ls functions/pci_epf_test/func1 + baseclass_code interrupt_pin progif_code subsys_id + cache_line_size msi_interrupts revid subsys_vendorid + deviceid msix_interrupts subclass_code vendorid + +The PCI endpoint function driver populates these entries with default values +when the device is bound to the driver. The pci-epf-test driver populates +vendorid with 0xffff and interrupt_pin with 0x0001:: + + # cat functions/pci_epf_test/func1/vendorid + 0xffff + # cat functions/pci_epf_test/func1/interrupt_pin + 0x0001 + + +Configuring pci-epf-test Device +------------------------------- + +The user can configure the pci-epf-test device using configfs entry. In order +to change the vendorid and the number of MSI interrupts used by the function +device, the following commands can be used:: + + # echo 0x104c > functions/pci_epf_test/func1/vendorid + # echo 0xb500 > functions/pci_epf_test/func1/deviceid + # echo 32 > functions/pci_epf_test/func1/msi_interrupts + # echo 2048 > functions/pci_epf_test/func1/msix_interrupts + +By default, pci-epf-test uses the following BAR sizes:: + + # grep . functions/pci_epf_test/func1/pci_epf_test.0/bar?_size + functions/pci_epf_test/func1/pci_epf_test.0/bar0_size:131072 + functions/pci_epf_test/func1/pci_epf_test.0/bar1_size:131072 + functions/pci_epf_test/func1/pci_epf_test.0/bar2_size:131072 + functions/pci_epf_test/func1/pci_epf_test.0/bar3_size:131072 + functions/pci_epf_test/func1/pci_epf_test.0/bar4_size:131072 + functions/pci_epf_test/func1/pci_epf_test.0/bar5_size:1048576 + +The user can override a default value using e.g.:: + # echo 1048576 > functions/pci_epf_test/func1/pci_epf_test.0/bar1_size + +Overriding the default BAR sizes can only be done before binding the +pci-epf-test device to a PCI endpoint controller driver. + +Note: Some endpoint controllers might have fixed-size BARs or reserved BARs; +for such controllers, the corresponding BAR size in configfs will be ignored. + + +Binding pci-epf-test Device to EP Controller +-------------------------------------------- + +In order for the endpoint function device to be useful, it has to be bound to +a PCI endpoint controller driver. Use the configfs to bind the function +device to one of the controller driver present in the system:: + + # ln -s functions/pci_epf_test/func1 controllers/51000000.pcie_ep/ + +Once the above step is completed, the PCI endpoint is ready to establish a link +with the host. + + +Start the Link +-------------- + +In order for the endpoint device to establish a link with the host, the _start_ +field should be populated with '1':: + + # echo 1 > controllers/51000000.pcie_ep/start + + +RootComplex Device +================== + +lspci Output +------------ + +Note that the devices listed here correspond to the value populated in 1.4 +above:: + + 00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) + 01:00.0 Unassigned class [ff00]: Texas Instruments Device b500 + + +Using Endpoint Test function Device +----------------------------------- + +Kselftest added in tools/testing/selftests/pci_endpoint can be used to run all +the default PCI endpoint tests. To build the Kselftest for PCI endpoint +subsystem, the following commands should be used:: + + # cd + # make -C tools/testing/selftests/pci_endpoint + +or if you desire to compile and install in your system:: + + # cd + # make -C tools/testing/selftests/pci_endpoint INSTALL_PATH=/usr/bin install + +The test will be located in /usr/bin/ + +Kselftest Output +~~~~~~~~~~~~~~~~ +:: + + # pci_endpoint_test + TAP version 13 + 1..16 + # Starting 16 tests from 9 test cases. + # RUN pci_ep_bar.BAR0.BAR_TEST ... + # OK pci_ep_bar.BAR0.BAR_TEST + ok 1 pci_ep_bar.BAR0.BAR_TEST + # RUN pci_ep_bar.BAR1.BAR_TEST ... + # OK pci_ep_bar.BAR1.BAR_TEST + ok 2 pci_ep_bar.BAR1.BAR_TEST + # RUN pci_ep_bar.BAR2.BAR_TEST ... + # OK pci_ep_bar.BAR2.BAR_TEST + ok 3 pci_ep_bar.BAR2.BAR_TEST + # RUN pci_ep_bar.BAR3.BAR_TEST ... + # OK pci_ep_bar.BAR3.BAR_TEST + ok 4 pci_ep_bar.BAR3.BAR_TEST + # RUN pci_ep_bar.BAR4.BAR_TEST ... + # OK pci_ep_bar.BAR4.BAR_TEST + ok 5 pci_ep_bar.BAR4.BAR_TEST + # RUN pci_ep_bar.BAR5.BAR_TEST ... + # OK pci_ep_bar.BAR5.BAR_TEST + ok 6 pci_ep_bar.BAR5.BAR_TEST + # RUN pci_ep_basic.CONSECUTIVE_BAR_TEST ... + # OK pci_ep_basic.CONSECUTIVE_BAR_TEST + ok 7 pci_ep_basic.CONSECUTIVE_BAR_TEST + # RUN pci_ep_basic.LEGACY_IRQ_TEST ... + # OK pci_ep_basic.LEGACY_IRQ_TEST + ok 8 pci_ep_basic.LEGACY_IRQ_TEST + # RUN pci_ep_basic.MSI_TEST ... + # OK pci_ep_basic.MSI_TEST + ok 9 pci_ep_basic.MSI_TEST + # RUN pci_ep_basic.MSIX_TEST ... + # OK pci_ep_basic.MSIX_TEST + ok 10 pci_ep_basic.MSIX_TEST + # RUN pci_ep_data_transfer.memcpy.READ_TEST ... + # OK pci_ep_data_transfer.memcpy.READ_TEST + ok 11 pci_ep_data_transfer.memcpy.READ_TEST + # RUN pci_ep_data_transfer.memcpy.WRITE_TEST ... + # OK pci_ep_data_transfer.memcpy.WRITE_TEST + ok 12 pci_ep_data_transfer.memcpy.WRITE_TEST + # RUN pci_ep_data_transfer.memcpy.COPY_TEST ... + # OK pci_ep_data_transfer.memcpy.COPY_TEST + ok 13 pci_ep_data_transfer.memcpy.COPY_TEST + # RUN pci_ep_data_transfer.dma.READ_TEST ... + # OK pci_ep_data_transfer.dma.READ_TEST + ok 14 pci_ep_data_transfer.dma.READ_TEST + # RUN pci_ep_data_transfer.dma.WRITE_TEST ... + # OK pci_ep_data_transfer.dma.WRITE_TEST + ok 15 pci_ep_data_transfer.dma.WRITE_TEST + # RUN pci_ep_data_transfer.dma.COPY_TEST ... + # OK pci_ep_data_transfer.dma.COPY_TEST + ok 16 pci_ep_data_transfer.dma.COPY_TEST + # PASSED: 16 / 16 tests passed. + # Totals: pass:16 fail:0 xfail:0 xpass:0 skip:0 error:0 + + +Testcase 16 (pci_ep_data_transfer.dma.COPY_TEST) will fail for most of the DMA +capable endpoint controllers due to the absence of the MEMCPY over DMA. For such +controllers, it is advisable to skip this testcase using this +command:: + + # pci_endpoint_test -f pci_ep_bar -f pci_ep_basic -v memcpy -T COPY_TEST -v dma + +Kselftest EP Doorbell +~~~~~~~~~~~~~~~~~~~~~ + +If the Endpoint MSI controller is used for the doorbell usecase, run below +command for testing it: + + # pci_endpoint_test -f pcie_ep_doorbell + + # Starting 1 tests from 1 test cases. + # RUN pcie_ep_doorbell.DOORBELL_TEST ... + # OK pcie_ep_doorbell.DOORBELL_TEST + ok 1 pcie_ep_doorbell.DOORBELL_TEST + # PASSED: 1 / 1 tests passed. + # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 diff --git a/Documentation/PCI/endpoint/pci-vntb-function.rst b/Documentation/PCI/endpoint/pci-vntb-function.rst new file mode 100644 index 000000000..0c51f53ab --- /dev/null +++ b/Documentation/PCI/endpoint/pci-vntb-function.rst @@ -0,0 +1,129 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI vNTB Function +================= + +:Author: Frank Li + +The difference between PCI NTB function and PCI vNTB function is + +PCI NTB function need at two endpoint instances and connect HOST1 +and HOST2. + +PCI vNTB function only use one host and one endpoint(EP), use NTB +connect EP and PCI host + +.. code-block:: text + + + +------------+ +---------------------------------------+ + | | | | + +------------+ | +--------------+ + | NTB | | | NTB | + | NetDev | | | NetDev | + +------------+ | +--------------+ + | NTB | | | NTB | + | Transfer | | | Transfer | + +------------+ | +--------------+ + | | | | | + | PCI NTB | | | | + | EPF | | | | + | Driver | | | PCI Virtual | + | | +---------------+ | NTB Driver | + | | | PCI EP NTB |<------>| | + | | | FN Driver | | | + +------------+ +---------------+ +--------------+ + | | | | | | + | PCI BUS | <-----> | PCI EP BUS | | Virtual PCI | + | | PCI | | | BUS | + +------------+ +---------------+--------+--------------+ + PCI RC PCI EP + +Constructs used for Implementing vNTB +===================================== + + 1) Config Region + 2) Self Scratchpad Registers + 3) Peer Scratchpad Registers + 4) Doorbell (DB) Registers + 5) Memory Window (MW) + + +Config Region: +-------------- + +It is same as PCI NTB Function driver + +Scratchpad Registers: +--------------------- + +It is appended after Config region. + +.. code-block:: text + + + +--------------------------------------------------+ Base + | | + | | + | | + | Common Config Register | + | | + | | + | | + +-----------------------+--------------------------+ Base + span_offset + | | | + | Peer Span Space | Span Space | + | | | + | | | + +-----------------------+--------------------------+ Base + span_offset + | | | + span_count * 4 + | | | + | Span Space | Peer Span Space | + | | | + +-----------------------+--------------------------+ + Virtual PCI Pcie Endpoint + NTB Driver NTB Driver + + +Doorbell Registers: +------------------- + + Doorbell Registers are used by the hosts to interrupt each other. + +Memory Window: +-------------- + + Actual transfer of data between the two hosts will happen using the + memory window. + +Modeling Constructs: +==================== + +32-bit BARs. + +====== =============== +BAR NO CONSTRUCTS USED +====== =============== +BAR0 Config Region +BAR1 Doorbell +BAR2 Memory Window 1 +BAR3 Memory Window 2 +BAR4 Memory Window 3 +BAR5 Memory Window 4 +====== =============== + +64-bit BARs. + +====== =============================== +BAR NO CONSTRUCTS USED +====== =============================== +BAR0 Config Region + Scratchpad +BAR1 +BAR2 Doorbell +BAR3 +BAR4 Memory Window 1 +BAR5 +====== =============================== + + diff --git a/Documentation/PCI/endpoint/pci-vntb-howto.rst b/Documentation/PCI/endpoint/pci-vntb-howto.rst new file mode 100644 index 000000000..3679f5c30 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-vntb-howto.rst @@ -0,0 +1,169 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================================== +PCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide +=================================================================== + +:Author: Frank Li + +This document is a guide to help users use pci-epf-vntb function driver +and ntb_hw_epf host driver for NTB functionality. The list of steps to +be followed in the host side and EP side is given below. For the hardware +configuration and internals of NTB using configurable endpoints see +Documentation/PCI/endpoint/pci-vntb-function.rst + +Endpoint Device +=============== + +Endpoint Controller Devices +--------------------------- + +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + 5f010000.pcie_ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + 5f010000.pcie_ep + +Endpoint Function Drivers +------------------------- + +To find the list of endpoint function drivers in the system:: + + # ls /sys/bus/pci-epf/drivers + pci_epf_ntb pci_epf_test pci_epf_vntb + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/functions + pci_epf_ntb pci_epf_test pci_epf_vntb + + +Creating pci-epf-vntb Device +---------------------------- + +PCI endpoint function device can be created using the configfs. To create +pci-epf-vntb device, the following commands can be used:: + + # mount -t configfs none /sys/kernel/config + # cd /sys/kernel/config/pci_ep/ + # mkdir functions/pci_epf_vntb/func1 + +The "mkdir func1" above creates the pci-epf-vntb function device that will +be probed by pci_epf_vntb driver. + +The PCI endpoint framework populates the directory with the following +configurable fields:: + + # ls functions/pci_epf_vntb/func1 + baseclass_code deviceid msi_interrupts pci-epf-vntb.0 + progif_code secondary subsys_id vendorid + cache_line_size interrupt_pin msix_interrupts primary + revid subclass_code subsys_vendor_id + +The PCI endpoint function driver populates these entries with default values +when the device is bound to the driver. The pci-epf-vntb driver populates +vendorid with 0xffff and interrupt_pin with 0x0001:: + + # cat functions/pci_epf_vntb/func1/vendorid + 0xffff + # cat functions/pci_epf_vntb/func1/interrupt_pin + 0x0001 + + +Configuring pci-epf-vntb Device +------------------------------- + +The user can configure the pci-epf-vntb device using its configfs entry. In order +to change the vendorid and the deviceid, the following +commands can be used:: + + # echo 0x1957 > functions/pci_epf_vntb/func1/vendorid + # echo 0x0809 > functions/pci_epf_vntb/func1/deviceid + +The PCI endpoint framework also automatically creates a sub-directory in the +function attribute directory. This sub-directory has the same name as the name +of the function device and is populated with the following NTB specific +attributes that can be configured by the user:: + + # ls functions/pci_epf_vntb/func1/pci_epf_vntb.0/ + ctrl_bar db_count mw1_bar mw2_bar mw3_bar mw4_bar spad_count + db_bar mw1 mw2 mw3 mw4 num_mws vbus_number + vntb_vid vntb_pid + +A sample configuration for NTB function is given below:: + + # echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count + # echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count + # echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws + # echo 0x100000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1 + +By default, each construct is assigned a BAR, as needed and in order. +Should a specific BAR setup be required by the platform, BAR may be assigned +to each construct using the related ``XYZ_bar`` entry. + +A sample configuration for virtual NTB driver for virtual PCI bus:: + + # echo 0x1957 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid + # echo 0x080A > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid + # echo 0x10 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number + +Binding pci-epf-vntb Device to EP Controller +-------------------------------------------- + +NTB function device should be attached to PCI endpoint controllers +connected to the host. + + # ln -s controllers/5f010000.pcie_ep functions/pci_epf_vntb/func1/primary + +Once the above step is completed, the PCI endpoint controllers are ready to +establish a link with the host. + + +Start the Link +-------------- + +In order for the endpoint device to establish a link with the host, the _start_ +field should be populated with '1'. For NTB, both the PCI endpoint controllers +should establish link with the host (imx8 don't need this steps):: + + # echo 1 > controllers/5f010000.pcie_ep/start + +RootComplex Device +================== + +lspci Output at Host side +------------------------- + +Note that the devices listed here correspond to the values populated in +"Creating pci-epf-vntb Device" section above:: + + # lspci + 00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) + 01:00.0 RAM memory: Freescale Semiconductor Inc Device 0809 + +Endpoint Device / Virtual PCI bus +================================= + +lspci Output at EP Side / Virtual PCI bus +----------------------------------------- + +Note that the devices listed here correspond to the values populated in +"Creating pci-epf-vntb Device" section above:: + + # lspci + 10:00.0 Unassigned class [ffff]: Dawicontrol Computersysteme GmbH Device 1234 (rev ff) + +Using ntb_hw_epf Device +----------------------- + +The host side software follows the standard NTB software architecture in Linux. +All the existing client side NTB utilities like NTB Transport Client and NTB +Netdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB +function device. + +For more information on NTB see +:doc:`Non-Transparent Bridge <../../driver-api/ntb>` -- cgit v1.3.1