349 lines
15 KiB
ReStructuredText
349 lines
15 KiB
ReStructuredText
.. SPDX-License-Identifier: GPL-2.0
|
|
|
|
=================
|
|
PCI NTB Function
|
|
=================
|
|
|
|
:Author: Kishon Vijay Abraham I <kishon@ti.com>
|
|
|
|
PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
|
|
with each other by exposing each host as a device to the other host.
|
|
NTBs typically support the ability to generate interrupts on the remote
|
|
machine, expose memory ranges as BARs, and perform DMA. They also support
|
|
scratchpads, which are areas of memory within the NTB that are accessible
|
|
from both machines.
|
|
|
|
PCI NTB Function allows two different systems (or hosts) to communicate
|
|
with each other by configuring the endpoint instances in such a way that
|
|
transactions from one system are routed to the other system.
|
|
|
|
In the below diagram, PCI NTB function configures the SoC with multiple
|
|
PCI Endpoint (EP) instances in such a way that transactions from one EP
|
|
controller are routed to the other EP controller. Once PCI NTB function
|
|
configures the SoC with multiple EP instances, HOST1 and HOST2 can
|
|
communicate with each other using SoC as a bridge.
|
|
|
|
.. code-block:: text
|
|
|
|
+-------------+ +-------------+
|
|
| | | |
|
|
| HOST1 | | HOST2 |
|
|
| | | |
|
|
+------^------+ +------^------+
|
|
| |
|
|
| |
|
|
+---------|-------------------------------------------------|---------+
|
|
| +------v------+ +------v------+ |
|
|
| | | | | |
|
|
| | EP | | EP | |
|
|
| | CONTROLLER1 | | CONTROLLER2 | |
|
|
| | <-----------------------------------> | |
|
|
| | | | | |
|
|
| | | | | |
|
|
| | | SoC With Multiple EP Instances | | |
|
|
| | | (Configured using NTB Function) | | |
|
|
| +-------------+ +-------------+ |
|
|
+---------------------------------------------------------------------+
|
|
|
|
Constructs used for Implementing NTB
|
|
====================================
|
|
|
|
1) Config Region
|
|
2) Self Scratchpad Registers
|
|
3) Peer Scratchpad Registers
|
|
4) Doorbell (DB) Registers
|
|
5) Memory Window (MW)
|
|
|
|
|
|
Config Region:
|
|
--------------
|
|
|
|
Config Region is a construct that is specific to NTB implemented using NTB
|
|
Endpoint Function Driver. The host and endpoint side NTB function driver will
|
|
exchange information with each other using this region. Config Region has
|
|
Control/Status Registers for configuring the Endpoint Controller. Host can
|
|
write into this region for configuring the outbound Address Translation Unit
|
|
(ATU) and to indicate the link status. Endpoint can indicate the status of
|
|
commands issued by host in this region. Endpoint can also indicate the
|
|
scratchpad offset and number of memory windows to the host using this region.
|
|
|
|
The format of Config Region is given below. All the fields here are 32 bits.
|
|
|
|
.. code-block:: text
|
|
|
|
+------------------------+
|
|
| COMMAND |
|
|
+------------------------+
|
|
| ARGUMENT |
|
|
+------------------------+
|
|
| STATUS |
|
|
+------------------------+
|
|
| TOPOLOGY |
|
|
+------------------------+
|
|
| ADDRESS (LOWER 32) |
|
|
+------------------------+
|
|
| ADDRESS (UPPER 32) |
|
|
+------------------------+
|
|
| SIZE |
|
|
+------------------------+
|
|
| NO OF MEMORY WINDOW |
|
|
+------------------------+
|
|
| MEMORY WINDOW1 OFFSET |
|
|
+------------------------+
|
|
| SPAD OFFSET |
|
|
+------------------------+
|
|
| SPAD COUNT |
|
|
+------------------------+
|
|
| DB ENTRY SIZE |
|
|
+------------------------+
|
|
| DB DATA |
|
|
+------------------------+
|
|
| : |
|
|
+------------------------+
|
|
| : |
|
|
+------------------------+
|
|
| DB DATA |
|
|
+------------------------+
|
|
|
|
|
|
COMMAND:
|
|
|
|
NTB function supports three commands:
|
|
|
|
CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
|
|
invoking this command, the host should allocate and initialize
|
|
MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
|
|
Endpoint). The endpoint on receiving this command will configure
|
|
the outbound ATU such that transactions to Doorbell BAR will be routed
|
|
to the MSI/MSI-X address programmed by the host. The ARGUMENT
|
|
register should be populated with number of DBs to configure (in the
|
|
lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
|
|
|
|
CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
|
|
host invokes this command after allocating a buffer that can be
|
|
accessed by remote host. The allocated address should be programmed
|
|
in the ADDRESS register (64 bit), the size should be programmed in
|
|
the SIZE register and the memory window index should be programmed
|
|
in the ARGUMENT register. The endpoint on receiving this command
|
|
will configure the outbound ATU such that transactions to MW BAR
|
|
are routed to the address provided by the host.
|
|
|
|
CMD_LINK_UP (0x3): Command to indicate an NTB application is
|
|
bound to the EP device on the host side. Once the endpoint
|
|
receives this command from both the hosts, the endpoint will
|
|
raise a LINK_UP event to both the hosts to indicate the host
|
|
NTB applications can start communicating with each other.
|
|
|
|
ARGUMENT:
|
|
|
|
The value of this register is based on the commands issued in
|
|
command register. See COMMAND section for more information.
|
|
|
|
TOPOLOGY:
|
|
|
|
Set to NTB_TOPO_B2B_USD for Primary interface
|
|
Set to NTB_TOPO_B2B_DSD for Secondary interface
|
|
|
|
ADDRESS/SIZE:
|
|
|
|
Address and Size to be used while configuring the memory window.
|
|
See "CMD_CONFIGURE_MW" for more info.
|
|
|
|
MEMORY WINDOW1 OFFSET:
|
|
|
|
Memory Window 1 and Doorbell registers are packed together in the
|
|
same BAR. The initial portion of the region will have doorbell
|
|
registers and the latter portion of the region is for memory window 1.
|
|
This register will specify the offset of the memory window 1.
|
|
|
|
NO OF MEMORY WINDOW:
|
|
|
|
Specifies the number of memory windows supported by the NTB device.
|
|
|
|
SPAD OFFSET:
|
|
|
|
Self scratchpad region and config region are packed together in the
|
|
same BAR. The initial portion of the region will have config region
|
|
and the latter portion of the region is for self scratchpad. This
|
|
register will specify the offset of the self scratchpad registers.
|
|
|
|
SPAD COUNT:
|
|
|
|
Specifies the number of scratchpad registers supported by the NTB
|
|
device.
|
|
|
|
DB ENTRY SIZE:
|
|
|
|
Used to determine the offset within the DB BAR that should be written
|
|
in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
|
|
ring doorbell (MSI-X support will be added later). MSI uses same
|
|
address for all the interrupts and MSI-X can provide different
|
|
addresses for different interrupts. The MSI/MSI-X address is provided
|
|
by the host and the address it gives is based on the MSI/MSI-X
|
|
implementation supported by the host. For instance, ARM platform
|
|
using GIC ITS will have the same MSI-X address for all the interrupts.
|
|
In order to support all the combinations and use the same mechanism
|
|
for both MSI and MSI-X, EPF NTB allocates a separate region in the
|
|
Outbound Address Space for each of the interrupts. This region will
|
|
be mapped to the MSI/MSI-X address provided by the host. If a host
|
|
provides the same address for all the interrupts, all the regions
|
|
will be translated to the same address. If a host provides different
|
|
addresses, the regions will be translated to different addresses. This
|
|
will ensure there is no difference while raising the doorbell.
|
|
|
|
DB DATA:
|
|
|
|
EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
|
|
This holds the MSI/MSI-X data that has to be written to MSI address
|
|
for raising doorbell interrupt. This will be populated by EPF NTB
|
|
while invoking CMD_CONFIGURE_DOORBELL.
|
|
|
|
Scratchpad Registers:
|
|
---------------------
|
|
|
|
Each host has its own register space allocated in the memory of NTB endpoint
|
|
controller. They are both readable and writable from both sides of the bridge.
|
|
They are used by applications built over NTB and can be used to pass control
|
|
and status information between both sides of a device.
|
|
|
|
Scratchpad registers has 2 parts
|
|
1) Self Scratchpad: Host's own register space
|
|
2) Peer Scratchpad: Remote host's register space.
|
|
|
|
Doorbell Registers:
|
|
-------------------
|
|
|
|
Doorbell Registers are used by the hosts to interrupt each other.
|
|
|
|
Memory Window:
|
|
--------------
|
|
|
|
Actual transfer of data between the two hosts will happen using the
|
|
memory window.
|
|
|
|
Modeling Constructs:
|
|
====================
|
|
|
|
There are 5 or more distinct regions (config, self scratchpad, peer
|
|
scratchpad, doorbell, one or more memory windows) to be modeled to achieve
|
|
NTB functionality. At least one memory window is required while more than
|
|
one is permitted. All these regions should be mapped to BARs for hosts to
|
|
access these regions.
|
|
|
|
If one 32-bit BAR is allocated for each of these regions, the scheme would
|
|
look like this:
|
|
|
|
====== ===============
|
|
BAR NO CONSTRUCTS USED
|
|
====== ===============
|
|
BAR0 Config Region
|
|
BAR1 Self Scratchpad
|
|
BAR2 Peer Scratchpad
|
|
BAR3 Doorbell
|
|
BAR4 Memory Window 1
|
|
BAR5 Memory Window 2
|
|
====== ===============
|
|
|
|
However if we allocate a separate BAR for each of the regions, there would not
|
|
be enough BARs for all the regions in a platform that supports only 64-bit
|
|
BARs.
|
|
|
|
In order to be supported by most of the platforms, the regions should be
|
|
packed and mapped to BARs in a way that provides NTB functionality and
|
|
also makes sure the host doesn't access any region that it is not supposed
|
|
to.
|
|
|
|
The following scheme is used in EPF NTB Function:
|
|
|
|
====== ===============================
|
|
BAR NO CONSTRUCTS USED
|
|
====== ===============================
|
|
BAR0 Config Region + Self Scratchpad
|
|
BAR1 Peer Scratchpad
|
|
BAR2 Doorbell + Memory Window 1
|
|
BAR3 Memory Window 2
|
|
BAR4 Memory Window 3
|
|
BAR5 Memory Window 4
|
|
====== ===============================
|
|
|
|
With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
|
|
|
|
Modeling Config/Scratchpad Region:
|
|
----------------------------------
|
|
|
|
.. code-block:: text
|
|
|
|
+-----------------+------->+------------------+ +-----------------+
|
|
| BAR0 | | CONFIG REGION | | BAR0 |
|
|
+-----------------+----+ +------------------+<-------+-----------------+
|
|
| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
|
|
+-----------------+ +-->+------------------+<-------+-----------------+
|
|
| BAR2 | Local Memory | BAR2 |
|
|
+-----------------+ +-----------------+
|
|
| BAR3 | | BAR3 |
|
|
+-----------------+ +-----------------+
|
|
| BAR4 | | BAR4 |
|
|
+-----------------+ +-----------------+
|
|
| BAR5 | | BAR5 |
|
|
+-----------------+ +-----------------+
|
|
EP CONTROLLER 1 EP CONTROLLER 2
|
|
|
|
Above diagram shows Config region + Scratchpad region for HOST1 (connected to
|
|
EP controller 1) allocated in local memory. The HOST1 can access the config
|
|
region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
|
|
The peer host (HOST2 connected to EP controller 2) can also access this
|
|
scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
|
|
diagram shows the case where Config region and Scratchpad regions are allocated
|
|
for HOST1, however the same is applicable for HOST2.
|
|
|
|
Modeling Doorbell/Memory Window 1:
|
|
----------------------------------
|
|
|
|
.. code-block:: text
|
|
|
|
+-----------------+ +----->+----------------+-----------+-----------------+
|
|
| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
|
|
+-----------------+ | +----------------+ +-----------------+
|
|
| BAR1 | | | Doorbell 2 +---------+ | |
|
|
+-----------------+----+ +----------------+ | | |
|
|
| BAR2 | | Doorbell 3 +-------+ | +-----------------+
|
|
+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
|
|
| BAR3 | | | Doorbell 4 +-----+ | +-----------------+
|
|
+-----------------+ | |----------------+ | | | |
|
|
| BAR4 | | | | | | +-----------------+
|
|
+-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
|
|
| BAR5 | | | | | | +-----------------+
|
|
+-----------------+ +----->-----------------+ | | | |
|
|
EP CONTROLLER 1 | | | | +-----------------+
|
|
| | | +---->+ MSI-X ADDRESS 4 |
|
|
+----------------+ | +-----------------+
|
|
EP CONTROLLER 2 | | |
|
|
(OB SPACE) | | |
|
|
+-------> MW1 |
|
|
| |
|
|
| |
|
|
+-----------------+
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
+-----------------+
|
|
PCI Address Space
|
|
(Managed by HOST2)
|
|
|
|
Above diagram shows how the doorbell and memory window 1 is mapped so that
|
|
HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
|
|
buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
|
|
memory window 1 regions are allocated in EP controller 2 outbound (OB) address
|
|
space. Allocating and configuring BARs for doorbell and memory window1
|
|
is done during the initialization phase of NTB endpoint function driver.
|
|
Mapping from EP controller 2 OB space to PCI address space is done when HOST2
|
|
sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
|
|
|
|
Modeling Optional Memory Windows:
|
|
---------------------------------
|
|
|
|
This is modeled the same was as MW1 but each of the additional memory windows
|
|
is mapped to separate BARs.
|