PCIe has an MSI-X interrupt 'capabillity' which consists of
a number (n) interrupt desctiptors and an associated Pending
Bit Array where each bit in PBA has a corresponding 128-bit
desctiptor. A descriptor contains a 64-bit address, a 32-bit
message, and a 32-bit vector control word.
There are 2-levels of enablement, one at the MSI-X configura-
tion control register and one in each interrupt descriptor at
vector control bit[31].
As the device raises an interrupt, it sets a bit in PBA.
When MSI-X is enabled and a bit in PBA is set (1) and the
vector control bit[31] is enabled, the device sends a
write of the message to the address in the descriptor,
and clears the bit in PBA.
I am assuming that the MSI-X enable bit is used to throttle
a device so that it sends bursts of interrupts to optimize
the caching behavior of the cores handling the interrupts.
run applications->handle k interrupts->run applications.
A home machine would not use this featrue as the interrupt
load is small, but a GB server might more control over when.
But does anybody know ??
a) device dommand to interrupt descriptor mapping {
Thre is no mention of the mapping of commands to the device
and to these interrupt descriptors. Can anyone supply input
or pointers to this mapping.
A single device (such as a SATA drive) might have a queue of
outstanding commands that it services in whatever order it
thinks best. Many of these commands want to inform some core
when the command is complete (or cannot be completed). To do
this, device sends a stored interrupt messages to the stored
service port.
}
I don't really NEED to know this mapping, but knowing would
significantly enhance my understanding of what is supposed
to be going on, and thus avoid making crippling errors.
b) address space of interrupt service port {
The address in the interrupt descriptor points at a service
port (APIC). Since a service port is "not like memory"*, I
want to mandate this aqddress be in MMI/O space, and since
My 66000 has a full 64-bit address space for MMI/O there is
no burden on the size of MMI/O space--it is already as big
as possible on a 64-bit machine. Plus, MMI/O space has the
property of being sequentially consistent whereas DRAM is
only cache consistent.
Most current architectures just partition a hunk of the
physical address space as MMI/O address space.
(*) memory has the property that a read will return the last
bit pattern written, a service port does not.
I assume that service port addresses map to different cores
(or local APICs of a core).
I want to directly support the
notion of a virtual core so while a 'chip' might have a large
number of physical cores, one would want a pool of thousands+
of virtual cores. I want said service ports to support raising
interrupt directly to a physical or virtual core.
}
Apparently, the message part of the MSI-X interrupt can be
interpreted any way that both SW and HW agree.
This works
for already defined architectures, and doing it like one
or more others, makes an OS port significantly easier.
However what these messages contain is difficult to find
via Google.
So, it seems to me, that the combination of the 64-bit address
and the 32-bit message must provide::
a) which level of the system to interrupt
{Secure Monitor, HyperVisor, SuperVisor, Application}
b) which core should handle the interrupt
{physical[0..k], virtual[l..m]}
c) what priority level is the interrupt.
{There are 64 unique priority levels}
d) something about why the interrupt was raised
{what remains of the meassage}
I suspect that (a) and (b) are parts of the address while (c)
and (d) are part of the message. Although nothing prevents
(c) from being part of the address.
Once MSI-X is sorted out MSI becomes a subset.
HostBridge has a service port that provides INT[A,B,C,D] to
MSI-X translation, so only MSI-X message are used system-
wide.
------------------------------------------------------------
It seems to me that the interrupt address needs translation
via I/O MMU, but which of the 4 levels provides the trans-
lation Root pointers ??
Am I allowed to use bits in Vector Control to provide this ??
But if I put it there then there is cross privilege leakage !
c) interupt latency {
When "what is running on a core" is timesliced by a HyperVisor,
a core that launched a command to a device may not be running
at the instant the interrupt arrives back.
It seems to me, that the HyperVisor would want to perform ISR
processing of the interrupt (low latency) and then schedule
the softIRQs to the <sleeping> core so when it regains control
the pending I/O stack of "stuff" is proprly cleaned up.
So, shold all initerrupt simple go to HyperVisor and let HV
sort it all out? Or can the <sleeping> virtual core just deal
with it when it is given a next time slice ??
mitchalsup@aol.com (MitchAlsup1) writes:
When MSI-X is enabled and a bit in PBA is set (1) and the
vector control bit[31] is enabled, the device sends a
write of the message to the address in the descriptor,
and clears the bit in PBA.
Note that if the interrupt condition is asserted after the
global enable in the MSI-X capability and the vector enable
have both been set to allow delivery, the message will be sent to
the root complex and PBA will not be updated. (P is for
pending, and once the message is sent, it's no longer
pending). PBA is only updated when the interrupt is masked
(either function-wide in the capability or per-vector).
mitchalsup@aol.com (MitchAlsup1) writes:
PCIe has an MSI-X interrupt 'capabillity' which consists of
a number (n) interrupt desctiptors and an associated Pending
Bit Array where each bit in PBA has a corresponding 128-bit
desctiptor. A descriptor contains a 64-bit address, a 32-bit
message, and a 32-bit vector control word.
There are 2-levels of enablement, one at the MSI-X configura-
tion control register and one in each interrupt descriptor at
vector control bit[31].
As the device raises an interrupt, it sets a bit in PBA.
When MSI-X is enabled and a bit in PBA is set (1) and the
vector control bit[31] is enabled, the device sends a
write of the message to the address in the descriptor,
and clears the bit in PBA.
Note that if the interrupt condition is asserted after the
global enable in the MSI-X capability and the vector enable
have both been set to allow delivery, the message will be sent to
the root complex and PBA will not be updated. (P is for
pending, and once the message is sent, it's no longer
pending). PBA is only updated when the interrupt is masked
(either function-wide in the capability or per-vector).
I am assuming that the MSI-X enable bit is used to throttle
In my experience the MSI-X function enable and vector enables
are not modified during runtime, rather the device has control
registers which allow masking of the interrupt (e.g.
for AHCI, the MSI message will only be sent if the port
PxIE (Port n Interrupt Enable) bit corresponding to a
PxIS (Port n Interrupt Status) bit is set).
Granted, AHCI specifies MSI, not MSI-X, but every MSI-X
device I've worked with operates the same way, with
device specific interrupt enables for a particular vector.
a device so that it sends bursts of interrupts to optimize
the caching behavior of the cores handling the interrupts.
run applications->handle k interrupts->run applications.
A home machine would not use this featrue as the interrupt
load is small, but a GB server might more control over when.
But does anybody know ??
Yes, we use MSI-X extensively. See above.
There are a number of mechanisms used for interrupt moderation,
but all generally are independent of the PCI message delivery.
(e.g. RSS spreads interrupts across multiple target cores,
or the Intel 10Ge network adapters interrupt moderation feature).
a) device dommand to interrupt descriptor mapping {
Thre is no mention of the mapping of commands to the device
and to these interrupt descriptors. Can anyone supply input
or pointers to this mapping.
Once the message leaves the device, is received by the
root complex port and is forwarded across the host bridge
to the system fabric, it's completely under control of
the host. On x86, the TLP for the upstream message is
received and forwarded to the specified address (which is
the IOAPIC on Intel and the GIC ITS on Arm64).
The interrupt controller may further mask the interrupt if
desired or if the interrupt priority is lower than the
current running priority.
A single device (such as a SATA drive) might have a queue of
outstanding commands that it services in whatever order it
thinks best. Many of these commands want to inform some core
when the command is complete (or cannot be completed). To do
this, device sends a stored interrupt messages to the stored
service port.
Each SATA port has an PxIS and PxIE register. The SATA (AHCI)
controller
MSI configuration can provide one vector per port - the main
difference between MSI and MSI-X is that the interrupt numbers
for MSI must be consecutive and there is only one address;
while for MSI-X each vector has an unique address and a programmable
data (interrupt number) field. The interpretation of the data
of the MSI-X or MSI upstream write is up to the interrupt controller
and may be virtualized in the interrupt controller.
Note that support for MSI in AHCI is optional (in which case the
legacy level sensitive PCI INTA/B/C/D signals are used).
The AHCI standard specification (ahci_1_3.pdf) is available publically.
}
I don't really NEED to know this mapping, but knowing would
significantly enhance my understanding of what is supposed
to be going on, and thus avoid making crippling errors.
b) address space of interrupt service port {
The address in the interrupt descriptor points at a service
port (APIC). Since a service port is "not like memory"*, I
want to mandate this aqddress be in MMI/O space, and since
My 66000 has a full 64-bit address space for MMI/O there is
no burden on the size of MMI/O space--it is already as big
as possible on a 64-bit machine. Plus, MMI/O space has the
property of being sequentially consistent whereas DRAM is
only cache consistent.
From the standpoint of the PCIexpress root port, the upstream write
generated by the device to send the MSI message to the host
looks just like any other inbound DMA from the device to the
host. It is the responsibility of the host bridge and interconnect to
route the message the appropriate destination (which generally
is an interrupt controller, but just as legally could be a
DRAM address which software polls periodically).
Most current architectures just partition a hunk of the
physical address space as MMI/O address space.
The address field in the MSI-X vector (or MSI-X capability)
is opaque to hardware below the PCIe root port.
Our chips recognize the interrupt controller range of
addresses in the inbound message at the host bridge
and route the message to the interrupt translation service;
the destinations in the interrupt controller are simply
control and status registers in the MMIO space. The
ARM64 interrupt controller supports multiple destinations
with different semantics (SPI and xSPI have one target
register and LPI has a different target register the address
of which is programmed into the MSI-X Vector address field).
(*) memory has the property that a read will return the last
bit pattern written, a service port does not.
I assume that service port addresses map to different cores
(or local APICs of a core).
The IOAPIC handles the message and has configuration registers
that determine which lAPIC should be signalled.
The GIC has configuration tables in memory that can remap
the interrupt to a different vector (e.g. for a guest VM).
I want to directly support the
notion of a virtual core so while a 'chip' might have a large
number of physical cores, one would want a pool of thousands+
of virtual cores. I want said service ports to support raising
interrupt directly to a physical or virtual core.
Take a look at IHI0069 (https://developer.arm.com/documentation/ihi0069/latest/)
}
Apparently, the message part of the MSI-X interrupt can be
interpreted any way that both SW and HW agree.
Yes.
This works
for already defined architectures, and doing it like one
or more others, makes an OS port significantly easier.
However what these messages contain is difficult to find
via Google.
The message is a 32-bit field and it is fully interpreted by
the interrupt controller (The GIC can be configured to support
from 16 to 32-bits data payload in an upstream MSI-X write;
the interpretation of the data is host specific).
On intel and ARM systems, the firmware knows the grungy details
and simply passes the desired payload value to the kernel
via the device tree(linux) or ACPI tables (for windows/linux).
So, it seems to me, that the combination of the 64-bit address
and the 32-bit message must provide::
a) which level of the system to interrupt
{Secure Monitor, HyperVisor, SuperVisor, Application}
No. That's completely a function of the interrupt controller
and how the hardware handles the data payload.
b) which core should handle the interrupt
{physical[0..k], virtual[l..m]}
Again, a function of the interrupt controller.
c) what priority level is the interrupt.
{There are 64 unique priority levels}
Yep, a function of the interrupt controller.
d) something about why the interrupt was raised
The interrupt itself causes the operating system
device driver interrupt function to be invoked. The
device-specific interrupt handler determines both
why the interrupt was raised (e.g. via the PxIS
register in the AHCI/SATA controller) and takes
the appropriate action.
On ARM64, it is common for the data field for
the MSI-X interrupts to number starting at zero
on every device, and they're mapped to a system-wide
unique value by the interrupt controller (e.g.
the GICv4 ITS).
If interrupt remapping hardware is
not available then unique data payloads for each
device need to be used.
Note that like any other inbound DMA, the address
in the MSI-X TLP that gets sent to the host bridge is subject
to translation by an IOMMU before getting to the
interrupt controller (or by the device itself if it
supports PCI-e Address Translation Services (ATS)).
{what remains of the meassage}
I suspect that (a) and (b) are parts of the address while (c)
and (d) are part of the message. Although nothing prevents
(c) from being part of the address.
Once MSI-X is sorted out MSI becomes a subset.
HostBridge has a service port that provides INT[A,B,C,D] to
MSI-X translation, so only MSI-X message are used system-
wide.
Note that INTA/B/C/D are level-sensitive. This requires
TWO MSI-X vectors - one that targets an "interrupt set"
register and the other targets and "interrupt clear"
register.
------------------------------------------------------------
It seems to me that the interrupt address needs translation
via I/O MMU, but which of the 4 levels provides the trans-
lation Root pointers ??
On Intel the IOMMU translation tables are not shared with the
AP.
The PCI address (aka Stream ID) is passed to the interrupt
controller and IOMMU and used as an index to determine the
page table root pointer.
The stream id format is
<2:0> PCI function number
<7:3> PCI device number
<15:8> PCI bus number
<xx:16> PCI segment (root complex) number.
This allows each device capable of inbound DMA to identify
themselves uniquely to the interrupt controller and IOMMU.
Both intel and AMD use this convention.
Am I allowed to use bits in Vector Control to provide this ??
But if I put it there then there is cross privilege leakage !
No, you're not allowed to do anything not explicitly allowed
in the PCI express specification. Remember, an MSI-X write
generated by the device is indistinguishable from any other
upstream DMA request initiated by the device.
c) interupt latency {
When "what is running on a core" is timesliced by a HyperVisor,
a core that launched a command to a device may not be running
at the instant the interrupt arrives back.
See again the document referenced above. The interrupt controller
is aware that the guest is not currently scheduled and maintains
a virutal pending state (and can optionally signal the hypervisor
that the guest should be scheduled ASAP).
Most of this is done completely by the hardware, without any
intervention by the hypervisor for the vast majority of
interrupts.
It seems to me, that the HyperVisor would want to perform ISR
processing of the interrupt (low latency) and then schedule
the softIRQs to the <sleeping> core so when it regains control
the pending I/O stack of "stuff" is proprly cleaned up.
So, shold all initerrupt simple go to HyperVisor and let HV
sort it all out? Or can the <sleeping> virtual core just deal
with it when it is given a next time slice ??
The original GIC did something like this (the HV took all
interrupts and there was a hardware mechanism to inject them
into a guest as if they were a hardware interrupt). But
it was too much overhead going through the hypervisor, especially
when the endpoint device support the SRIOV capability. So the
GIC supports handling virtual interrupt delivery completely
in hardware unless the guest is not currently resident on any
virtual CPU.
One thing to note::
The PCIe root complex may send out an address, but when the
MMI/O receiver of said address then uses bits in the address
for routing purposes or interpretation purposes, it matters
not that the sender did not know those bits were being used
thusly--those bits actually do carry specific meaning--just
not to the sender.
Scott Lurndal wrote:
It seems to me that the interrupt address needs translation
via I/O MMU, but which of the 4 levels provides the trans-
lation Root pointers ??
On Intel the IOMMU translation tables are not shared with the
AP.
I have seen in the past 3 days AP being used to point at a
random device out on the PCIe tree and of the unprivileged
application layer. Both ends of the spectrum. Which is your
usage ?
The PCI address (aka Stream ID) is passed to the interrupt
controller and IOMMU and used as an index to determine the
page table root pointer.
The stream id format is
<2:0> PCI function number
<7:3> PCI device number
<15:8> PCI bus number
<xx:16> PCI segment (root complex) number.
I use ChipID for the last field in case each chip has its own
PCIe tree. {Except that the bits are placed elsewhere in the
address.}
But (now with the new CXL) instead of allocating 200+ pins
to DRAM those pins can be allocated to PCIe links; making any
chip much less dependent on which DRAM technology, which chip-
to-chip repeaters,... So, the thought is all I/O is PCIe + CXL;
and about the only other pins chip gets are RESET and ClockIn.
Bunches of these pins can be 'configured' into standard width
PCIe links (at least until one runs out of pins.)
Given that one has a PCIe root complex with around 256-pins
available, does one need multiple roots of such a wide tree ?
This allows each device capable of inbound DMA to identify
themselves uniquely to the interrupt controller and IOMMU.
Both intel and AMD use this convention.
Am I allowed to use bits in Vector Control to provide this ??
But if I put it there then there is cross privilege leakage !
No, you're not allowed to do anything not explicitly allowed
in the PCI express specification. Remember, an MSI-X write
generated by the device is indistinguishable from any other
upstream DMA request initiated by the device.
Why did PCI committee specify a 32-bit container and define the
use on only 1 bit ?? Or are more bits defined but I just haven't
run into any literature concerning those ?
c) interupt latency {
When "what is running on a core" is timesliced by a HyperVisor,
a core that launched a command to a device may not be running
at the instant the interrupt arrives back.
See again the document referenced above. The interrupt controller
is aware that the guest is not currently scheduled and maintains
a virutal pending state (and can optionally signal the hypervisor
that the guest should be scheduled ASAP).
Are you using the word 'signal' as LINUX signal delivery, or as
a proxy for interrupt of some form, or perhaps as an SVC to HV
of some form ?
Scott Lurndal wrote:
Once the message leaves the device, is received by the
root complex port and is forwarded across the host bridge
to the system fabric, it's completely under control of
the host. On x86, the TLP for the upstream message is
received and forwarded to the specified address (which is
the IOAPIC on Intel and the GIC ITS on Arm64).
The interrupt controller may further mask the interrupt if
desired or if the interrupt priority is lower than the
current running priority.
{note to self:: that is why its a local APIC--it has to be close
enough to see the core's priority.}
Question:: Down below you talk of the various interrupt control-
lers routing an interrupt <finally> to a core. What happens if the
core has changed its priority by the time the interrupt signal
arrives, but before it can change the state of the tables in the
interrupt controller that routed said interrupt here ?
A single device (such as a SATA drive) might have a queue of
outstanding commands that it services in whatever order it
thinks best. Many of these commands want to inform some core
when the command is complete (or cannot be completed). To do
this, device sends a stored interrupt messages to the stored
service port.
Each SATA port has an PxIS and PxIE register. The SATA (AHCI)
controller
MSI configuration can provide one vector per port - the main
difference between MSI and MSI-X is that the interrupt numbers
for MSI must be consecutive and there is only one address;
while for MSI-X each vector has an unique address and a programmable
data (interrupt number) field. The interpretation of the data
of the MSI-X or MSI upstream write is up to the interrupt controller
and may be virtualized in the interrupt controller.
I see (below) that you (they) migrated all the stuff I though might
be either in the address or data to the "other side" of HostBridge.
Fair enough.
For what reason are there multiple addresses ?
From the standpoint of the PCIexpress root port, the upstream write
generated by the device to send the MSI message to the host
looks just like any other inbound DMA from the device to the
host. It is the responsibility of the host bridge and interconnect to
route the message the appropriate destination (which generally
is an interrupt controller, but just as legally could be a
DRAM address which software polls periodically).
So the message arrive at the top of the PCIe tree is RAW, then
the address gets translated by I/O MMU, and both translated
address and RAW data are passed forward to its fate.
What I am trying to do is to figure out a means to route the
message to a virtual core's interrupt table such that:: if that
virtual core happens to be running on any physical core, that
the physical core sees the interrupt without delay, and if
the virtual core is not running, the event is properly logged
so when the virtual core runs on a physical core that those
ISRs are performed before any lower priority work is performed.
{and make this work for any number of physical cores and any
number of virtual cores; where cores can sharing interrupt
tables. For example, Guest OS[k] thinks that is has 13 cores
and shares its interrupt table across 5 of them, but HyperVisor
remains free to time slice Guest OS[k] cores any way it likes.}
(*) memory has the property that a read will return the last
bit pattern written, a service port does not.
I assume that service port addresses map to different cores
(or local APICs of a core).
The IOAPIC handles the message and has configuration registers
that determine which lAPIC should be signalled.
The GIC has configuration tables in memory that can remap
the interrupt to a different vector (e.g. for a guest VM).
GIC = Global Interrupt Controller ?
It seems to me that the interrupt address needs translation
via I/O MMU, but which of the 4 levels provides the trans-
lation Root pointers ??
On Intel the IOMMU translation tables are not shared with the
AP.
I have seen in the past 3 days AP being used to point at a
random device out on the PCIe tree and of the unprivileged
application layer. Both ends of the spectrum. Which is your
usage ?
The PCI address (aka Stream ID) is passed to the interrupt
controller and IOMMU and used as an index to determine the
page table root pointer.
The stream id format is
<2:0> PCI function number
<7:3> PCI device number
<15:8> PCI bus number
<xx:16> PCI segment (root complex) number.
I use ChipID for the last field in case each chip has its own
PCIe tree. {Except that the bits are placed elsewhere in the
address.}
But (now with the new CXL) instead of allocating 200+ pins
to DRAM those pins can be allocated to PCIe links; making any
chip much less dependent on which DRAM technology, which chip-
to-chip repeaters,... So, the thought is all I/O is PCIe + CXL;
and about the only other pins chip gets are RESET and ClockIn.
Bunches of these pins can be 'configured' into standard width
PCIe links (at least until one runs out of pins.)
Given that one has a PCIe root complex with around 256-pins
available, does one need multiple roots of such a wide tree ?
This allows each device capable of inbound DMA to identify
themselves uniquely to the interrupt controller and IOMMU.
Both intel and AMD use this convention.
Am I allowed to use bits in Vector Control to provide this ??
But if I put it there then there is cross privilege leakage !
No, you're not allowed to do anything not explicitly allowed
in the PCI express specification. Remember, an MSI-X write
generated by the device is indistinguishable from any other
upstream DMA request initiated by the device.
Why did PCI committee specify a 32-bit container and define the
use on only 1 bit ?? Or are more bits defined but I just haven't
run into any literature concerning those ?
c) interupt latency {
When "what is running on a core" is timesliced by a HyperVisor,
a core that launched a command to a device may not be running
at the instant the interrupt arrives back.
See again the document referenced above. The interrupt controller
is aware that the guest is not currently scheduled and maintains
a virutal pending state (and can optionally signal the hypervisor
that the guest should be scheduled ASAP).
Are you using the word 'signal' as LINUX signal delivery, or as
a proxy for interrupt of some form, or perhaps as an SVC to HV
of some form ?
Most of this is done completely by the hardware, without any
intervention by the hypervisor for the vast majority of
interrupts.
That is the goal.
It seems to me, that the HyperVisor would want to perform ISR
processing of the interrupt (low latency) and then schedule
the softIRQs to the <sleeping> core so when it regains control
the pending I/O stack of "stuff" is proprly cleaned up.
So, shold all initerrupt simple go to HyperVisor and let HV
sort it all out? Or can the <sleeping> virtual core just deal
with it when it is given a next time slice ??
The original GIC did something like this (the HV took all
interrupts and there was a hardware mechanism to inject them
into a guest as if they were a hardware interrupt). But
it was too much overhead going through the hypervisor, especially
when the endpoint device support the SRIOV capability. So the
GIC supports handling virtual interrupt delivery completely
in hardware unless the guest is not currently resident on any
virtual CPU.
Leave HV out of the loop unless something drastic happens.
I/O completion and I/O aborts are not that drastic.
Once again, I thank you greatly for your long and informative
post.
mitchalsup@aol.com (MitchAlsup1) writes:
Scott Lurndal wrote:
The PCI address (aka Stream ID) is passed to the interrupt
controller and IOMMU and used as an index to determine the
page table root pointer.
The stream id format is
<2:0> PCI function number
<7:3> PCI device number
<15:8> PCI bus number
<xx:16> PCI segment (root complex) number.
I use ChipID for the last field in case each chip has its own
PCIe tree. {Except that the bits are placed elsewhere in the
address.}
Each root complex needs to be an unique segment. A single
SRIOV endpoint can consume the entire 8-bit bus space and
the 8-bit dev/function space. In this context, a root complex
can be considered a PCI express controller with one or more
root ports. Each root port should be considered an unique
'segment'.
This is for device discovery, which uses the PCI express
"Extended Configuration Access Method" (aka ECAM) to scan
the PCI configuration spaces of all PCI ports.
But (now with the new CXL) instead of allocating 200+ pins
to DRAM those pins can be allocated to PCIe links; making any
chip much less dependent on which DRAM technology, which chip-
to-chip repeaters,... So, the thought is all I/O is PCIe + CXL;
and about the only other pins chip gets are RESET and ClockIn.
Note that bridging to PCI signalling will increase latency
somewhat, even with PCIe gen 6.
Bunches of these pins can be 'configured' into standard width
PCIe links (at least until one runs out of pins.)
Given that one has a PCIe root complex with around 256-pins
available, does one need multiple roots of such a wide tree ?
You basically need a root per device to accommodate SRIOV
devices (like enterprise grade network adapters, high-end
NVMe devices, etc).
mitchalsup@aol.com (MitchAlsup1) writes:
Scott Lurndal wrote:
Question:: Down below you talk of the various interrupt control-
lers routing an interrupt <finally> to a core. What happens if the
core has changed its priority by the time the interrupt signal
arrives, but before it can change the state of the tables in the
interrupt controller that routed said interrupt here ?
Speaking for the ARM64 systems that I'm most recently
familiar with, the concept of priority is associated with
an interrupt (up to 8-bits worth of priority - an implementation
of the GIC is allowed to support as few as three bits).
The interrupt controller is distributed logic; there is a
component called the 'distributor' and another component called
the 'redistributor'. The former is global to the system and
the latter is a per-CPU component. The distributor also contains
a subsystem called the interrupt translation subsystem (ITS) which
supports interrupt virtualization.
The redistributor, being part of the core, handles the delivery
of an interrupt to the core (specifically asserting either the FIQ
or IRQ signals that cause entry to the IRQ or FIQ exception
handlers). The redistributor tracks the current running priority
(which is directly associated with the priority of the current
active interrupt, when not processing an interrupt, the current
running priority is called the IDLE priority and doesn't block
delivery of any interrupts). The redistributor communicates changes to
the RPR to the distributor, which will hold any interrupt that
is not eligable for delivery (for any reason, including lack
of priority). There is no way for software to change the
RPR - it only tracks the priority of the currently executing
interrupt.
+-------------------------+
| PCI Device |
+-------------------------+
| MSI-X message (address: GITS_TRANSLATER control
register)
| (payload: Interrupt number (0 to N))
v (sideband: streamid) +-------------------------+
| Interrupt Translation | (DRAM tables: Device, Collection)
| Service | Lookup streamid in device table.
| | DT refers to Interrupt Translation Table
| | Translate inbound payload based on ITT to
an LPI
| | Collection table identifies target core +-------------------------+
| Internal message from ITS to redistributor for target
v
+-------------------------+
| Redistributor | (DRAM table: LPI properties)
| | Lookup LPI properties, contains priority
and enable bit
| | If not enabled or priority too low,
| | store in LPI pending table (also DRAM)
[*]
| | If enabled, unmasked at the CPU interface
| | and priority higher than RPR, assert FIQ
| | or IRQ signals to core. +-------------------------+
| IRQ/FIQ signals
v
+-------------------------+
| Core | Check PSTATE IRQ and FIQ mask bits
| | IRQ/FIQ can be routed to EL1, EL2 or EL3
| | by per-core control bits. Update
| | core state appropriately and enter ISR +-------------------------+
[*] as core RPR and signal masks change, the ITS re-evaluates pending
[**] LPI properties and pending bits are generally cached in the
redistributor for performance.
+-------------------------+
| PCI Device |
+-------------------------+
| MSI-X message (address: GITS_TRANSLATER control
register)
| (payload: Interrupt number (0 to N))
v (sideband: streamid) +-------------------------+
| HostBridge Translation | IOMMU tables: B:D,F->Originating Context
| Service | Originating Context supplies Root pointers
| | and interrupt table address
| In LLC | HostBridge DRAM accesses are performed
through LLC
| | HostBridge MMI/O accesses routed out into
Chip
+-------------------------+
| MMI/O message from HTS to virtual context stack DRAM
address
|
| If core interrupt table matches MMI/O address ? SNARF message
| the message contains pending priority interrupt
bits.
v
+-------------------------+
| Core | If there is a interrupt at higher priority
| | than I am currently running ? begin
interrupt
| | negotiation (core continues to run
instructions)
| | If negotiation is successful ? Claim
interrupt
| | and context switch to Interrupt
Dispatcher.
+-------------------------+
For what reason are there multiple addresses ?
A system may have multiple interrupt controllers. In the
case of the ARM64 systems, there may be a case where some
interrupts should be considered level sensitive, in which
case they must use SPI type interrupts which have a different
target register for the MSI-X address field when compared with
LPI type interrupts.
Recall that the PCI spec must accomodate a wide range of system implementations (including Z-series).
mitchalsup@aol.com (MitchAlsup1) writes:
One thing to note::
The PCIe root complex may send out an address, but when the
MMI/O receiver of said address then uses bits in the address
for routing purposes or interpretation purposes, it matters
not that the sender did not know those bits were being used
thusly--those bits actually do carry specific meaning--just
not to the sender.
Said routing does not necessarily mean that the bits in
the addresses are interpreted in any specific way, if the
routing is done by a set of programmed range registers
(i.e. range X to X+Y is routed to destination Z,e.g. the
interrupt controller).
Scott Lurndal wrote:
Again allow me to express my gratitute on the quality of your posts !
A couple of dumb questions to illustrate how much more I need to
learn::
mitchalsup@aol.com (MitchAlsup1) writes:
Scott Lurndal wrote:
The PCI address (aka Stream ID) is passed to the interrupt
controller and IOMMU and used as an index to determine the
page table root pointer.
The stream id format is
<2:0> PCI function number
<7:3> PCI device number
<15:8> PCI bus number
<xx:16> PCI segment (root complex) number.
I use ChipID for the last field in case each chip has its own
PCIe tree. {Except that the bits are placed elsewhere in the
address.}
Each root complex needs to be an unique segment. A single
SRIOV endpoint can consume the entire 8-bit bus space and
the 8-bit dev/function space. In this context, a root complex
can be considered a PCI express controller with one or more
root ports. Each root port should be considered an unique
'segment'.
This is for device discovery, which uses the PCI express
"Extended Configuration Access Method" (aka ECAM) to scan
the PCI configuration spaces of all PCI ports.
Within a 'Chip' there are k cores, 1 last level cache, and
1 HostBridge with (say) 256 pins at its disposal. Said
pins can be handed out in powers of 2 of 4-pins each
so multiple PCIe trees of differing widths emanate from
the 256-PCIe-pins.
I guess you are calling each point of emanation a root.
I just bundle them under 1 HostBridge, and consider how
the "handing out" is done to be a HostBridge problem.
But As seen on the on-chip interconnect there is one
HostBridge which accesses all devices attached to this
Chip. Basically, I see on-chip-interconnect with one
HostBridge knowing that the pins will be allocated
"efficiently" for the attached devices.
You basically need a root per device to accommodate SRIOV
devices (like enterprise grade network adapters, high-end
NVMe devices, etc).
As noted above: I knew more bits than B:D,F were needed,
but not which and where. And if a single SR-IOV device
consumes a whole B:D,F space sobeit. ECAM alignment
identifies those bits and the routings.
I guess reading m post backwards I did not pose any questions.
My thanks again.
Scott Lurndal wrote:
mitchalsup@aol.com (MitchAlsup1) writes:
One thing to note::
The PCIe root complex may send out an address, but when the
MMI/O receiver of said address then uses bits in the address
for routing purposes or interpretation purposes, it matters
not that the sender did not know those bits were being used
thusly--those bits actually do carry specific meaning--just
not to the sender.
Said routing does not necessarily mean that the bits in
the addresses are interpreted in any specific way, if the
routing is done by a set of programmed range registers
(i.e. range X to X+Y is routed to destination Z,e.g. the
interrupt controller).
I was asking the contrapositive::
Is a system architecture allowed to define certain bits of
the translated address to be used as either routing or
indexing of a table that provides routing information.
Not as seen by request originator or request target, but
by the middle-men of transport ?
You basically need a root per device to accommodate SRIOV
devices (like enterprise grade network adapters, high-end
NVMe devices, etc).
As noted above: I knew more bits than B:D,F were needed,
but not which and where.
And if a single SR-IOV device
consumes a whole B:D,F space sobeit. ECAM alignment
identifies those bits and the routings.
Ah, I should have read this one backwards :-)
mitchalsup@aol.com (MitchAlsup1) writes:
Scott Lurndal wrote:
mitchalsup@aol.com (MitchAlsup1) writes:
One thing to note::
The PCIe root complex may send out an address, but when the
MMI/O receiver of said address then uses bits in the address
for routing purposes or interpretation purposes, it matters
not that the sender did not know those bits were being used >>>>thusly--those bits actually do carry specific meaning--just
not to the sender.
Said routing does not necessarily mean that the bits in
the addresses are interpreted in any specific way, if the
routing is done by a set of programmed range registers
(i.e. range X to X+Y is routed to destination Z,e.g. the
interrupt controller).
I was asking the contrapositive::
Is a system architecture allowed to define certain bits of
the translated address to be used as either routing or
indexing of a table that provides routing information.
Not as seen by request originator or request target, but
by the middle-men of transport ?
From the standpoint of the PCI specification, the host
side is completely unspecified. You could, for example,
use bits <63:60> to specify the socket, or chiplet that
the address should be routed to. Other bits may encode
the PCI controller #, interrupt controller, IOMMU, etc.
mitchalsup@aol.com (MitchAlsup1) writes:
A single device (such as a SATA drive) might have a queue of
outstanding commands that it services in whatever order it
thinks best. Many of these commands want to inform some core
when the command is complete (or cannot be completed). To do
this, device sends a stored interrupt messages to the stored
service port.
Each SATA port has an PxIS and PxIE register. The SATA (AHCI) controller MSI configuration can provide one vector per port - the main
difference between MSI and MSI-X is that the interrupt numbers
for MSI must be consecutive and there is only one address;
while for MSI-X each vector has an unique address and a programmable
data (interrupt number) field. The interpretation of the data
of the MSI-X or MSI upstream write is up to the interrupt controller
and may be virtualized in the interrupt controller.
Note that support for MSI in AHCI is optional (in which case the
legacy level sensitive PCI INTA/B/C/D signals are used).
The AHCI standard specification (ahci_1_3.pdf) is available publically.
Scott Lurndal wrote:
mitchalsup@aol.com (MitchAlsup1) writes:
A single device (such as a SATA drive) might have a queue of
outstanding commands that it services in whatever order it
thinks best. Many of these commands want to inform some core
when the command is complete (or cannot be completed). To do
this, device sends a stored interrupt messages to the stored
service port.
Each SATA port has an PxIS and PxIE register. The SATA (AHCI) controller >> MSI configuration can provide one vector per port - the main
difference between MSI and MSI-X is that the interrupt numbers
for MSI must be consecutive and there is only one address;
while for MSI-X each vector has an unique address and a programmable
data (interrupt number) field. The interpretation of the data
of the MSI-X or MSI upstream write is up to the interrupt controller
and may be virtualized in the interrupt controller.
Note that support for MSI in AHCI is optional (in which case the
legacy level sensitive PCI INTA/B/C/D signals are used).
The AHCI standard specification (ahci_1_3.pdf) is available publically.
What happens to SATA tagged command queuing with SRIOV?
The tag mapping would seem to interact with virtual interrupts.
SATA allows up to I think 8 commands queued at once, each with its own
tag number, which can be performed in any order. That tag indicates
which DMA mapping scatter gather set to use and is used to identify
which IO's are complete. A single interrupt can indicate multiple
tags are complete.
In native (non-virtualized) use the device driver assigns a free tag
number to an IO, sets up the DMA scatter/gather list for that tag,
and on completion interrupt, for each done tag it tears down the
DMA scatter/gather list, frees that tag number,
and completes the associated individual IO.
How would that work for SATA on SRIOV?
Scott,Mitch,
Can you foresee any problem is I define the positions in the
physical address space as: <ECAM: My 66000-style>::
<1:0> must be 00 for word access
<2:7> standard MMI/O register
<12:8> extended MMI/O register
<25:13> growth space for registers or for functions
<32:26> PCIe Device, Function
<40:32> PCIe Bus
<56:41> PCIe Segment
<63:57> Chip
I am expecting the code touching the MMI/O register to have
a virtual address pointer to B;D,F and use the 16-bit immediate
field of the LD or ST as the register specifier:
ST #command7,[Rdevice,#registername]
--------------------------------------------------------------
I am expecting to use the Chip field to route requests between
chips. It is plausible that physical device sends an interrupt
from its PCIe segment across one-or-more chips before arriving
at the interrupt service port in a particular chips last level
cache. Other than latency its all part of a large coherent DRAM
space.
Is that plausible ? desirable ? or are there reasons to keep
interrupt processing "more local" to the chip hosting the PCIe
root complexes ?? {in any event, that is all under SW control.}
mitchalsup@aol.com (MitchAlsup1) writes:
Scott,
Note that the ECAM must support 8, 16, 32-bit (optionally 64-bit)
single-copy atomic accesses to all configuration space registers.
That depends on how your interrupt controller is designed. If
you have a multi-socket/multi-chiplet configuration where all
chiplets are identical, and each has its own interrupt controller
(allowing single chiplet implementations), then you'll probably
want to use your CHIP bits in the address to route to the interrupt controller on the closest chiplet just to reduce interrupt
latency.
The interrupt controllers will likely need to cooperate
at the hardware level to maintain a single OS-visible "interrupt
space" where each controller handles a subset of the interrupt
number space.
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
Scott Lurndal wrote:
mitchalsup@aol.com (MitchAlsup1) writes:
Scott,
Note that the ECAM must support 8, 16, 32-bit (optionally 64-bit)
single-copy atomic accesses to all configuration space registers.
MMI/O is sequentially consistent while Config is Strongly ordered.
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
On Thu, 27 Jun 2024 01:47:49 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
It sounds as a politically correct way of saying "default memory
ordering of ARMv8.1-A and later".
I.e. weaker than x86-64 and SPARC TSO, but stronger than Itanium.
Probably stronger than POWER, but I am not sure if POWER ever had memory ordering model formalized.
Michael S wrote:
On Thu, 27 Jun 2024 01:47:49 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
It sounds as a politically correct way of saying "default memory
ordering of ARMv8.1-A and later".
I.e. weaker than x86-64 and SPARC TSO, but stronger than Itanium.
Probably stronger than POWER, but I am not sure if POWER ever had memory
ordering model formalized.
Multi-copy atomic is ARM's name for a write-update coherence protocol
as it allows each cache to have its own copy of a single memory
location.
Single-copy atomic is their name for a write-invalidate protocol
as it ensures that there is one value for each memory location.
Originally ARM's weak cache coherence protocol spec, like Alpha,
did not explicitly exclude multi-copy atomic so software designers had
to consider all the extra race conditions a write-update implementation
might allow. But this was wasted extra effort because no one implements
a write-update protocol, just write-invalidate.
Eventually ARM specified that it was single-copy atomic
(write-invalidate).
mitchalsup@aol.com (MitchAlsup1) writes:
Scott Lurndal wrote:
mitchalsup@aol.com (MitchAlsup1) writes:
Scott,
Note that the ECAM must support 8, 16, 32-bit (optionally 64-bit)
single-copy atomic accesses to all configuration space registers.
MMI/O is sequentially consistent while Config is Strongly ordered.
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
It's a term-of-art in the ARM architecture document (DDI0487).
A memory access instruction that is single-copy atomic has the
following properties:
1. For a pair of overlapping single-copy atomic store instructions,
all
of the overlapping writes generated by one of the stores are
Coherence-after the corresponding overlapping writes generated
by the other store.
2. For a single-copy atomic load instruction L1 that overlaps a single-copy
atomic store instruction S2, if one of the overlapping reads
generated
by L1 Reads-from one of the overlapping writes generated by S2,
then none
of the overlapping writes generated by S2 are Coherence-after the
corresponding overlapping reads generated
by L1.
EricP wrote:
Michael S wrote:
On Thu, 27 Jun 2024 01:47:49 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
It sounds as a politically correct way of saying "default memory
ordering of ARMv8.1-A and later".
I.e. weaker than x86-64 and SPARC TSO, but stronger than Itanium.
Probably stronger than POWER, but I am not sure if POWER ever had memory >>> ordering model formalized.
Multi-copy atomic is ARM's name for a write-update coherence protocol
as it allows each cache to have its own copy of a single memory
location.
Sounds like SNARFing
Originally ARM's weak cache coherence protocol spec, like Alpha,
did not explicitly exclude multi-copy atomic so software designers had
to consider all the extra race conditions a write-update implementation
might allow. But this was wasted extra effort because no one implements
a write-update protocol, just write-invalidate.
Eventually ARM specified that it was single-copy atomic
(write-invalidate).
Seems to me that if one is sequentially consistent, then one is also multi-copy ATOMIC.
MitchAlsup1 wrote:
Seems to me that if one is sequentially consistent, then one is also
multi-copy ATOMIC.
Yes, store atomicity to each locations would be implied by SC
otherwise how could all nodes agree on the order of all updates.
MitchAlsup1 wrote:
Sounds like SNARFing
Write-update depends on broadcasting all writes if that's what snarf
means.
EricP wrote:
MitchAlsup1 wrote:
Sounds like SNARFing
Write-update depends on broadcasting all writes if that's what snarf
means.
General cache coherency policies broadcast a cores address to all
caches in the system, and if that cache contains that same cache
line, it responds with a SHARED back to requestor, or it invalidates
the line. We call this SNOOPing. It works well.
SNARF is a term whereby the owner of data broadcasts the data and
its address, and any cache containing that line will write the
data payload into its cache 9rather than invalidating and then
going back and fetching it anew. For certain kinds of data struct
SNARF is significantly more efficient than Invalidate-Refetch.
A single message around the system performs all the needed updates,
instead of 1 invalidate and K fetches.
SNARF is almost exclusively used as side-band signals hiding under
the cache coherent Interconnect command set.
SNARF is almost never available to software. It is more like micro- >Architecture talking to other microArchitecture.
Also note: µA-to-µA is rarely of line size and often uses physical
address bits not available through MMU tables.
On Fri, 28 Jun 2024 20:26:42 +0000, mitchalsup@aol.com (MitchAlsup1)
wrote:
EricP wrote:
MitchAlsup1 wrote:
Sounds like SNARFing
Write-update depends on broadcasting all writes if that's what snarf
means.
General cache coherency policies broadcast a cores address to all
caches in the system, and if that cache contains that same cache
line, it responds with a SHARED back to requestor, or it invalidates
the line. We call this SNOOPing. It works well.
SNARF is a term whereby the owner of data broadcasts the data and
its address, and any cache containing that line will write the
data payload into its cache 9rather than invalidating and then
going back and fetching it anew. For certain kinds of data struct
SNARF is significantly more efficient than Invalidate-Refetch.
A single message around the system performs all the needed updates,
instead of 1 invalidate and K fetches.
SNARF is almost exclusively used as side-band signals hiding under
the cache coherent Interconnect command set.
SNARF is almost never available to software. It is more like micro- >>Architecture talking to other microArchitecture.
Also note: µA-to-µA is rarely of line size and often uses physical >>address bits not available through MMU tables.
Stupid question: why is it called "snarf"?
IIRC, Snoopy (Peanuts) "scarfed" his food. I don't recall ever seeing
Snarf (Thundercats) actually eat.
Michael S wrote:
On Thu, 27 Jun 2024 01:47:49 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:
Exactly what are you intending to mean from "single-copy atomic
accesses" ??
It sounds as a politically correct way of saying "default memory
ordering of ARMv8.1-A and later".
I.e. weaker than x86-64 and SPARC TSO, but stronger than Itanium.
Probably stronger than POWER, but I am not sure if POWER ever had memory
ordering model formalized.
Multi-copy atomic is ARM's name for a write-update coherence protocol
as it allows each cache to have its own copy of a single memory location.
Single-copy atomic is their name for a write-invalidate protocol
as it ensures that there is one value for each memory location.
Originally ARM's weak cache coherence protocol spec, like Alpha,
did not explicitly exclude multi-copy atomic so software designers had
to consider all the extra race conditions a write-update implementation
might allow. But this was wasted extra effort because no one implements
a write-update protocol, just write-invalidate.
Eventually ARM specified that it was single-copy atomic (write-invalidate).
On Page Request Services (PRS)
Device performs an Address Translation Request (ATS) to a page
which is not currently present in memory. So the I/O MMU sends
it a PTE which contains the not-present information.
But the system operates with nested paging. one manipulated
by Guest OS and the other manipulated by HyperVisor.
Yet device merely got "not-present"
So when device requests the page be brought in, how does the
I/O MMU know whether to interrupt Guest OS or to interrupt
Hypervisor to bring in the page and restart the command ??
On page 43 of:: >https://www.cs.uml.edu/~bill/cs520/slides_15E_PCI_Express_IOV.pdf
it states: "Must not indicate an invalidation has completed
until all outstanding Read Requests that reference the
associated translation have retired"
"Must insure that the invalidation completion indication to RC
will arrive at the RC after previously posted writes that use
the stale address."
and
"...If transactions are in a queue waiting to be sent, It is
not necessary for the device to expunge requests from the
queue even if those transaction[s] use an address that is
being invalidated."
The first 2 seem to be PCIe ordering requirements between
EP and RC.
The 3rd seems to say if EP used a translation while it was
valid, then its invalidation does not prevent requests
using the now stale translation.
So, a SATA device could receive a command to read a page
into memory. SATA EP requests ATS for the translation of
the given virtual address to the physical page. Then the
EP creates a queue of write requests filling in the addr
while waiting on data. Once said queue has been filled,
and before the data comes off the disk, an invalidation
arrives and is ACKed. The data is still allowed to write
into memory.
{{But any new command to the SATA device would not be
allowed to use the translation.}}
Is this a reasonable interpretation of that page?
In article <922220c8593353c7ed0fda9e656d359d@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:
On page 43 of:: >>https://www.cs.uml.edu/~bill/cs520/slides_15E_PCI_Express_IOV.pdf
it states: "Must not indicate an invalidation has completed
until all outstanding Read Requests that reference the
associated translation have retired"
"Must insure that the invalidation completion indication to RC
will arrive at the RC after previously posted writes that use
the stale address."
and
"...If transactions are in a queue waiting to be sent, It is
not necessary for the device to expunge requests from the
queue even if those transaction[s] use an address that is
being invalidated."
The first 2 seem to be PCIe ordering requirements between
EP and RC.
The 3rd seems to say if EP used a translation while it was
valid, then its invalidation does not prevent requests
using the now stale translation.
So, a SATA device could receive a command to read a page
into memory. SATA EP requests ATS for the translation of
the given virtual address to the physical page. Then the
EP creates a queue of write requests filling in the addr
while waiting on data. Once said queue has been filled,
and before the data comes off the disk, an invalidation
arrives and is ACKed. The data is still allowed to write
into memory.
{{But any new command to the SATA device would not be
allowed to use the translation.}}
Is this a reasonable interpretation of that page?
No, it's saying that the EP can keep using a stale translation UNTIL it >returns the ACK for an invalidation. It does not need to toss those >requests--it just needs to delay the ACK. Or it could toss the requests,
and then send the ACK faster, but it's optional if it wants to toss requests.
kegs@provalid.com (Kent Dickey) writes:
In article <922220c8593353c7ed0fda9e656d359d@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:
On page 43 of:: >>>https://www.cs.uml.edu/~bill/cs520/slides_15E_PCI_Express_IOV.pdf
it states: "Must not indicate an invalidation has completed
until all outstanding Read Requests that reference the
associated translation have retired"
"Must insure that the invalidation completion indication to RC
will arrive at the RC after previously posted writes that use
the stale address."
and
"...If transactions are in a queue waiting to be sent, It is
not necessary for the device to expunge requests from the
queue even if those transaction[s] use an address that is
being invalidated."
The first 2 seem to be PCIe ordering requirements between
EP and RC.
The 3rd seems to say if EP used a translation while it was
valid, then its invalidation does not prevent requests
using the now stale translation.
So, a SATA device could receive a command to read a page
into memory. SATA EP requests ATS for the translation of
the given virtual address to the physical page. Then the
EP creates a queue of write requests filling in the addr
while waiting on data. Once said queue has been filled,
and before the data comes off the disk, an invalidation
arrives and is ACKed. The data is still allowed to write
into memory.
{{But any new command to the SATA device would not be
allowed to use the translation.}}
Is this a reasonable interpretation of that page?
No, it's saying that the EP can keep using a stale translation UNTIL it >>returns the ACK for an invalidation. It does not need to toss those >>requests--it just needs to delay the ACK. Or it could toss the requests, >>and then send the ACK faster, but it's optional if it wants to toss
requests.
Indeed. And I'd suggest that the official PCI Express
specification is a better source than a set of slides.
From the spec:
a. A Function is required not to indicate the invalidation has completed until
all outstanding Read Requests or Translation Requests that reference
the
associated translated address have been retired or nullified.
b. A Function is required to ensure that the Invalidate Completion
indication
to the RC will arrive at the RC after any previously posted writes
that use
the "stale" address.
On Wed, 10 Jul 2024 19:02:12 +0000, Scott Lurndal wrote:
Indeed. And I'd suggest that the official PCI Express
specification is a better source than a set of slides.
From the spec:
I do not have access through the PCIe paywall.
On Wed, 10 Jul 2024 17:59:46 +0000, MitchAlsup1 wrote:
On page 34 of:: >https://www.cs.uml.edu/~bill/cs520/slides_15E_PCI_Express_IOV.pdf
They/He uses the notation VP# (virtual Plane number)
Is that what we have been calling the PCIe "Segment" ?? from ECAM
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 546 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 42:29:36 |
| Calls: | 10,392 |
| Files: | 14,064 |
| Messages: | 6,417,214 |