This contains notes about using a AMD Alveo™ U200 Data Center Accelerator Card (Active)
The card was second hand and has been installed in slot 2 on a HP Z6 G4 workstation, with the 8-pin aux power cable connected. This is a PCle3 x16 slot connected to the CPU. The slot 2 BIOS settings were the following, with Hot Plug enabled since had previously used other FPGA boards in the slot:
Slot 2 PCI Express x16
Disable
*Enable
Slot 2 Option Rom Download
Disable
*Enable
Slot 2 Limit PCIe Speed
*Auto
Gen1 (2.5Gbps)
Gen2 (5Gbps)
Gen3 (8Gbps)
Slot 2 Bifurcation
*Auto
x8x8
x4x4x4x4
Slot 2 Intel VROC NVMe Raid
*Disable
Enable
Slot 2 Hot Plug
Disable
*Enable
Slot 2 Hot Plug Buses
0
*8
16
32
64
128
Slot 2 Resizable Bars
*Disable
Enable
PCIe Training Reset is also enabled in the BIOS, which was previously enabled when investigating failures to enumerate as a PCIe endpoint for some other FPGA designs.
The first micro USB cable tried didn't fit fully and couldn't detect the JTAG port. Alveo Elongated USB Cable is described as:
U200/U250 Micro-USB connector location requires elongated USB for proper connection on Active cards due to blocking from heat spreader. Elongated USB cables can be used for both active and passive cards, but is a must have for active cards.
Found a different micro USB cable, which while didn't appeared to be the length of exposed onnector but a thinner plastic shell. That allowed the JTAG port to be detected as:
Bus 003 Device 008: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
The target part is xcu200-fsgd2104-2-e. From Alveo and Kria the corresponding standalone FPGA is a XCVU9P. Some documentation such as Virtex UltraScale+ Devices Available GT Quads from the UltraScale+ Devices Integrated Block for PCI Express Product Guide (PG213) doesn't list the U200 so have to search for the corresponding standalone FPGA.
After initially fitting the card booted into Windows 11 Pro for Workstations Version 24H2.
Device Manager shows a PCI Serial Port in slot 2, but doesn't find a compatible driver.
It is shown as a Gen 3 x16 device:
PS C:\Users\mr_halfword> C:\Users\mr_halfword\Git-projects\fpga_sio\multiple_boards\report_pcie_links.ps1
Name ExpressSpecVersion MaxLinkSpeed MaxLinkWidth CurrentLinkSpeed CurrentLinkWidth
---- ------------------ ------------ ------------ ---------------- ----------------
Standard NVM Express Controller 2 3 4 3 4
Mellanox ConnectX-3 PRO VPI (MT04103) Network Adapter 2 3 8 3 8
Intel(R) Ethernet Connection X722 for 10GBASE-T 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 10GBASE-T #2 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 1GbE 2 1 1 1 1
High Definition Audio Controller 2 3 16 1 4
NVIDIA Quadro K620 2 3 16 1 4
PCI Serial Port 2 3 16 3 16
Mellanox ConnectX-4 Adapter 2 3 16 3 16
Booted into openSUSE Leap 15.5. dump_pci_info_pciutils shows the following for the serial port PCIe endpoint
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> dump_info/dump_pci_info_pciutils 1c9d
domain=0000 bus=31 dev=00 func=00 rev=00
vendor_id=1c9d (Vendor 1c9d) device_id=0101 (Device 0101) subvendor_id=1c9d subdevice_id=0007
iommu_group=81
physical_slot=2-2
control: I/O- Mem+ BusMaster- ParErr+ SERR+ DisINTx-
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
bar[0] base_addr=97400000 size=200000 is_IO=0 is_prefetchable=0 is_64=1
Capabilities: [40] Power Management
Capabilities: [48] Message Signaled Interrupts
Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port # 0 ASPM not supported
L0s Exit Latency More than 4 μs
L1 Exit Latency More than 64 μs
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
domain=0000 bus=30 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=51
driver=pcieport
physical_slot=2
control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 5 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #2 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management
There is no driver bound. The PCI vendor 1c9d isn't known to the PCI libraries. The PCI SIG Member Companies search reports this is for http://www.illumina.com/. Their website say they provide innovative sequencing and array technologies for medical research. Can't see any obvious products which make use of a U200.
So far haven't attempted to modify the QSPI contents.
quad_spi_flasher reports:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> xilinx_quad_spi/quad_spi_flasher
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Enabled bus master for 0000:31:00.0
Displaying information for SPI flash using U200_dma_stream_crc64 design in PCI device 0000:31:00.0 IOMMU group 22
FIFO depth=256
Flash device : Micron MT25QU01G
Manufacturer ID=0x20 Memory Interface Type=0xbb Density=0x21
Flash Size Bytes=134217728 Page Size Bytes=256 Num Address Bytes=4
Successfully parsed bitstream of length 40454200 bytes with 1555146 configuration packets with 3 SLRs
Read 40468480 bytes from SPI flash starting at address 0
SLR[0] Sync word at byte index 0x50
Type 1 packet opcode NOP
Type 1 packet opcode write register BSPI words 0000066C
Type 1 packet opcode write register CMD command BSPI_READ
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register TIMER words 47FF0000
Type 1 packet opcode write register WBSTAR words 04000000
Type 1 packet opcode write register CMD command IPROG
Type 1 packet opcode NOP
Type 1 packet opcode write register CMD command RCRC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register FAR words 00000000
Type 1 packet opcode write register RBCRC_SW words 00000000
Type 1 packet opcode write register COR0 words 384235E5
Type 1 packet opcode write register COR1 words 00400000
Type 1 packet opcode write register IDCODE XCU200
Type 1 packet opcode write register CMD command FALL_EDGE
Type 1 packet opcode write register CMD command SWITCH
Type 1 packet opcode NOP
Type 1 packet opcode write register MASK words 00001001
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00001000
Type 1 packet opcode NOP (8 consecutive)
Configuration data writes consisting of:
250975 NOPs
20624 FAR writes
1271 WCFG commands
1271 FDRI writes with a total of 209994 words
604 MFW commands
604 NULL commands
19353 MFWR writes with a total of 135471 words
224 Type 2 packets with a total of 4786710 words
Type 1 packet opcode write register CRC words 2571C4B7
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command GRESTORE
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DGHIGH_LFRM
Type 1 packet opcode NOP (20 consecutive)
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00000000
Type 1 packet opcode write register CMD command START
Type 1 packet opcode NOP
Type 1 packet opcode write register FAR words 07FC0000
Type 1 packet opcode write register MASK words 00001101
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register CRC words 2C43DECC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DESYNC
Type 1 packet opcode NOP (393 consecutive)
SLR[1] Sync word at byte index 0x14CAFEC
Type 1 packet opcode NOP
Type 1 packet opcode write register BSPI words 0000066C
Type 1 packet opcode write register CMD command BSPI_READ
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register TIMER words 00000000
Type 1 packet opcode write register WBSTAR words 04000000
Type 1 packet opcode write register CMD command IPROG
Type 1 packet opcode NOP
Type 1 packet opcode write register CMD command RCRC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register FAR words 00000000
Type 1 packet opcode write register RBCRC_SW words 00000000
Type 1 packet opcode write register COR0 words 384235E5
Type 1 packet opcode write register COR1 words 00400000
Type 1 packet opcode write register IDCODE unknown (0x4b22093)
Type 1 packet opcode write register CMD command SWITCH
Type 1 packet opcode NOP
Type 1 packet opcode write register MASK words 00001001
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00001000
Type 1 packet opcode NOP (8 consecutive)
Configuration data writes consisting of:
495983 NOPs
62331 FAR writes
1968 WCFG commands
1968 FDRI writes with a total of 287835 words
1491 MFW commands
1491 NULL commands
60363 MFWR writes with a total of 422541 words
57 Type 2 packets with a total of 958830 words
Type 1 packet opcode write register CRC words 51E20486
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command GRESTORE
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DGHIGH_LFRM
Type 1 packet opcode NOP (20 consecutive)
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00000000
Type 1 packet opcode write register CMD command START
Type 1 packet opcode NOP
Type 1 packet opcode write register FAR words 07FC0000
Type 1 packet opcode write register MASK words 00001101
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register CRC words 2C43DECC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DESYNC
Type 1 packet opcode NOP (393 consecutive)
SLR[2] Sync word at byte index 0x1DCE47C
Type 1 packet opcode NOP
Type 1 packet opcode write register BSPI words 0000066C
Type 1 packet opcode write register CMD command BSPI_READ
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register TIMER words 00000000
Type 1 packet opcode write register WBSTAR words 04000000
Type 1 packet opcode write register CMD command IPROG
Type 1 packet opcode NOP
Type 1 packet opcode write register CMD command RCRC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register FAR words 00000000
Type 1 packet opcode write register RBCRC_SW words 00000000
Type 1 packet opcode write register COR0 words 384235E5
Type 1 packet opcode write register COR1 words 00400000
Type 1 packet opcode write register IDCODE unknown (0x4b24093)
Type 1 packet opcode write register CMD command SWITCH
Type 1 packet opcode NOP
Type 1 packet opcode write register MASK words 00001001
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00001000
Type 1 packet opcode NOP (8 consecutive)
Configuration data writes consisting of:
501004 NOPs
63312 FAR writes
2042 WCFG commands
2042 FDRI writes with a total of 300204 words
1514 MFW commands
1514 NULL commands
61270 MFWR writes with a total of 428890 words
57 Type 2 packets with a total of 868992 words
Type 1 packet opcode write register CRC words C77D23DF
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command GRESTORE
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DGHIGH_LFRM
Type 1 packet opcode NOP (20 consecutive)
Type 1 packet opcode write register MASK words 00001000
Type 1 packet opcode write register CTL1 words 00000000
Type 1 packet opcode write register CMD command START
Type 1 packet opcode NOP
Type 1 packet opcode write register FAR words 07FC0000
Type 1 packet opcode write register MASK words 00001101
Type 1 packet opcode write register CTL0 words 00001101
Type 1 packet opcode write register CRC words 2C43DECC
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register CMD command DESYNC
Type 1 packet opcode NOP (399 consecutive)
Type 1 packet opcode write register CMD command START
Type 1 packet opcode NOP
Type 1 packet opcode write register CMD command DESYNC
Type 1 packet opcode NOP (10 consecutive)
Type 1 packet opcode write register CMD command START
Type 1 packet opcode NOP
Type 1 packet opcode write register CMD command DESYNC
Type 1 packet opcode NOP (404 consecutive)
Above bitstream appears to be golden image. Looking for following multiboot image at offset 0x4000000
Error parsing bitstream: Unknown packet type 6 in header word C3834B67 at index 388
Read 32768 bytes from SPI flash starting at address 67108864
Sync word at byte index 0x50
Type 1 packet opcode NOP
Type 1 packet opcode write register BSPI words 0000066C
Type 1 packet opcode write register CMD command BSPI_READ
Type 1 packet opcode NOP (2 consecutive)
Type 1 packet opcode write register MASK words 80000040
Type 1 packet opcode write register CTL0 words 80000040
Type 1 packet opcode write register CBC words B7AB46BB AB46BB8F 46BB8F1B BB8F1B5A
Type 1 packet opcode NOP (61 consecutive)
The bitstream at address 0, which contains a IPROG and a WBSTAR of 0x04000000 can be parsed OK.
However the parsing of the bitstream at 0x4000000 aborts due to an unknown packet type. Prior to the aborted parsing:
- Writes to MASK and CTRL0 which set:
- FUSE_KEY (bit 31) 1: eFUSE to select the AES source
- DEC (bit 6) 1: Decryptor enabled
- Writes 128 bits to CBC
I.e. the bitstream which failed to parse appears to be encrypted.
The EFUSE register settings reported by the Vivado Hardware Manager using JTAG:
REGISTER.EFUSE.FUSE_DNA <snip>
REGISTER.EFUSE.DNA_PORT <snip>
REGISTER.EFUSE.FUSE_CNTL 1447C1
REGISTER.EFUSE.FUSE_USER 18111901
REGISTER.EFUSE.FUSE_SECURITY 40
REGISTER.EFUSE.FUSE_KEY Unreadable
REGISTER.EFUSE.FUSE_RSA FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
REGISTER.EFUSE.FUSE_USER_128 00000000000000000000000000000000
Where the FUSE_KEY is reported as Unreadable.
The following bits are set in FUSE_CNTL:
- R_DIS_KEY - disables the CRC check that verifies the AES key and programming of the AES key.
- R_DIS_RSA - disables reading of the authentication key FUSE_RSA register through the JTAG interface.
- W_DIS_KEY - disables programming of the FUSE_KEY AES encryption key register through the JTAG interface and disables the CRC check that verifies the AES encryption key.
- W_DIS_USER - disables programming of the FUSE_USER user code register through the JTAG interface.
- W_DIS_SEC - disables programming of the FUSE_SEC register through the JTAG interface.
- Reserved bits 10, 14, 18, 20
FUSE_SECURITY has the FUSE_BKS_ENABLE bit set, which enables key obfuscation.
See also Using Encryption and Authentication to Secure an UltraScale/UltraScale+ FPGA Bitstream
This was the initial design created, to prove could create a bitstream with the no-cost Vivado license, before brought the card.
Programmed the bitstream while openSUSE Leap 15.5 was booted, and bind_xilinx_devices_to_vfio.sh was able to bind to the loaded bitstream PCIe endpoint. I.e. hot plug worked.
display_identified_pcie_fpga_designs reports zero for the User access build timestamp:
~/fpga_sio/software_tests/eclipse_project/bin/release> identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Enabled bus master for 0000:31:00.0
Design AS02MC04_enum:
PCI device 0000:31:00.0 rev 00 IOMMU group 22 physical slot 2-2
DMA bridge bar 1 memory size 0x1000
Channel ID addr_alignment len_granularity num_address_bits
H2C 0 1 1 64
C2H 0 1 1 64
User access build timestamp : 00000000 - 00/00/2000 00:00:00
Based upon the output of parse_bitstream_file failed to enable the user access build timestamp.
After changing the design to insert the user access build timestamp and allocate a different identity:
identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Enabled bus master for 0000:31:00.0
Design U200_enum:
PCI device 0000:31:00.0 rev 00 IOMMU group 22 physical slot 2-2
DMA bridge bar 1 memory size 0x1000
Channel ID addr_alignment len_granularity num_address_bits
H2C 0 1 1 64
C2H 0 1 1 64
User access build timestamp : 34B37034 - 06/09/2025 23:00:52
dump_info/dump_pci_info_pciutils
domain=0000 bus=31 dev=00 func=00 rev=00
vendor_id=10ee (Xilinx Corporation) device_id=903f (Device 903f) subvendor_id=0002 subdevice_id=001c
iommu_group=22
driver=vfio-pci
physical_slot=2-2
control: I/O- Mem+ BusMaster- ParErr- SERR+ DisINTx-
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
bar[0] base_addr=97410000 size=1000 is_IO=0 is_prefetchable=0 is_64=0
bar[1] base_addr=97400000 size=10000 is_IO=0 is_prefetchable=0 is_64=0
Capabilities: [40] Power Management
Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 0 ASPM not supported
L0s Exit Latency More than 4 μs
L1 Exit Latency More than 64 μs
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
domain=0000 bus=30 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=51
driver=pcieport
physical_slot=2
control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 5 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #2 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management
In a Block Design tried to create a x8x8 design, with two DMA/Bridge Subsystem for PCI Express IP which share the same Utility Buffer for the PCIe refclk:
The design synthesised, but got errors during implementation with conflicts over placement.
The first error was:
[DRC REQP-1963] connects_too_many_BUFG_GT_SYNC_loads: The IBUFDS_GTE4 U200_enum_i/util_ds_buf_0/U0/USE_IBUFDS_GTE4.GEN_IBUFDS_GTE4[0].IBUFDS_GTE4_I (ODIV2 pin) is driving more than one BUFG_GT_SYNC load, which is an unroutable situation. Optimization may not have been able to merge BUFG_GT_SYNC cells because of differing control pin connections.
There are also 4 BUF_GT critical warnings:
[DRC BFGTL-1] bad_BUFG_GT_muxing: Invalid CE or CLR connectivity for BUFG_GT cells U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/bufg_gt_sysclk and U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/bufg_gt_sysclk. The CE and CLR pins of BUFG_GT cells that share a common clock input should be driven by the same net. The CE pin with net U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/sync_sc_ce of the first cell is not the same as the net on that pin for the other cell, which is U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/sync_sc_ce. These nets must be the same because the input clock net U200_enum_i/util_ds_buf_0/U0/IBUF_DS_ODIV2[0] is the same for both cells. The design must be modified (note: optimization may resolve this issue).
[DRC BFGTL-1] bad_BUFG_GT_muxing: Invalid CE or CLR connectivity for BUFG_GT cells U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/bufg_gt_sysclk and U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/bufg_gt_sysclk. The CE and CLR pins of BUFG_GT cells that share a common clock input should be driven by the same net. The CLR pin with net U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/sync_sc_clr of the first cell is not the same as the net on that pin for the other cell, which is U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/sync_sc_clr. These nets must be the same because the input clock net U200_enum_i/util_ds_buf_0/U0/IBUF_DS_ODIV2[0] is the same for both cells. The design must be modified (note: optimization may resolve this issue).
[DRC BFGTL-1] bad_BUFG_GT_muxing: Invalid CE or CLR connectivity for BUFG_GT cells U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/bufg_gt_sysclk and U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/bufg_gt_sysclk. The CE and CLR pins of BUFG_GT cells that share a common clock input should be driven by the same net. The CE pin with net U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/sync_sc_ce of the first cell is not the same as the net on that pin for the other cell, which is U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/sync_sc_ce. These nets must be the same because the input clock net U200_enum_i/util_ds_buf_0/U0/IBUF_DS_ODIV2[0] is the same for both cells. The design must be modified (note: optimization may resolve this issue).
[DRC BFGTL-1] bad_BUFG_GT_muxing: Invalid CE or CLR connectivity for BUFG_GT cells U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/bufg_gt_sysclk and U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/bufg_gt_sysclk. The CE and CLR pins of BUFG_GT cells that share a common clock input should be driven by the same net. The CLR pin with net U200_enum_i/xdma_0/inst/pcie4_ip_i/inst/sync_sc_clr of the first cell is not the same as the net on that pin for the other cell, which is U200_enum_i/xdma_1/inst/pcie4_ip_i/inst/sync_sc_clr. These nets must be the same because the input clock net U200_enum_i/util_ds_buf_0/U0/IBUF_DS_ODIV2[0] is the same for both cells. The design must be modified (note: optimization may resolve this issue).
The U200_enum/U200_enum.gen/sources_1/bd/U200_enum/ip/U200_enum_xdma_0_0/ip_0/source/U200_enum_xdma_0_0_pcie4_ip_pcie4_uscale_core_top.v and U200_enum/U200_enum.gen/sources_1/bd/U200_enum/ip/U200_enum_xdma_1_0/ip_0/source/U200_enum_xdma_1_0_pcie4_ip_pcie4_uscale_core_top.v files both have:
wire sync_sc_ce;
wire sync_sc_clr;
BUFG_GT bufg_gt_sysclk (.CE (sync_sc_ce), .CEMASK (1'd0), .CLR (sync_sc_clr), .CLRMASK (1'd0), .DIV (3'd0), .I (sys_clk), .O (sys_clk_bufg));
BUFG_GT_SYNC sync_sys_clk (.CESYNC(sync_sc_ce), .CLRSYNC(sync_sc_clr), .CE(gt_gtpowergood[0]), .CLK(sys_clk), .CLR(1'b0));
assign sys_clk_ce_out = gt_gtpowergood[0];BUFG_GT and BUFG_GT_SYNC from the UltraScale Architecture Clocking Resources User Guide (UG572) has an overview of the above components.
lp with BUFDS_GTE4 in Ultrascale+. describes issues with device constaints.
For the placement options for the DMA/Bridge Subsystem for PCI Express IP:
- PCIe Block Location of
X1Y2allows a max width of x16 with QUADs 224 to 227 - PCIe Block Location of
X1Y4allows a max width of x8 with QUADs 230 or 231
The same placement options exist for the UltraScale+ Integrated Block (PCIE4) for PCI Express (1.3) IP.
The PCIe constaints for the U200 use:
- QUADs 224 to 227 for the PCIe lanes
- QUAD 226 for the PCIe refclk
TABLE: Virtex UltraScale+ Devices Available GT Quads (XCVU9P) shows for a XCVU9P in the FSGD2104 package:
| PCIE Blocks | Quads with Max Link Width X16 Support | Quads with Max Link Width X8 Support | Quads with Max Link Width X4 Support |
|---|---|---|---|
| X1Y2 | GTY_Quad_228, GTY_Quad_227 | GTY_Quad_226, GTY_Quad_225 | GTY_Quad_224 |
| X1Y4 | GTY_Quad_233, GTY_Quad_232 | GTY_Quad_231, GTY_Quad_230 | GTY_Quad_229 |
UltraScale+ Device Packaging and Pinouts Product Specification User Guide (UG575) Figure 1-122: XCVU9P Banks in FSGD2104 Package shows the "PCIE4 X1Y2 (tandem)" and "PCIE4 X1Y4" blocks are different sides of a SLR Crossing GT Locations in the UltraScale+ Devices Integrated Block for PCI Express Product Guide (PG213) contains:
A GT Quad is comprised of four GT lanes. When selecting GT Quads for the PCIe IP, AMD recommends that you use the GT Quad most adjacent to the PCIe hard block. While this is not required, it improves place, route, and timing for the design.
- Link widths of x1, x2, and x4 require one bonded GT Quad and should not split lanes between two GT Quads.
- A link width of x8 requires two adjacent GT Quads that are bonded and are in the same SLR.
- A link width of x16 requires four adjacent GT Quads that are bonded and are in the same SLR.
The SYSMON, Configuration, PCIe, Interlaken, and 100GE Integrated Blocks section in UG575 has:
Note: Do not connect the integrated block for PCIe to transceiver channels through an SLR crossing. For further details, refer to the Placement Rules section of the UltraScale Devices Gen3 Integrated Block for PCI Express Product Guide (PG156) and UltraScale+ Devices Integrated Block for PCI Express Product Guide (PG213). Blocks with an additional (Tandem) label support Tandem configuration.
The U200 has:
- QUADs 224 to 227 connected to PCIe lanes
- QAUD 230 connected to QSFP1
- QUAD 231 connected to QSFP0
Perhaps that is why the DMA/Bridge Subsystem for PCI Express IP selection of the PCIe Block Location and QUADs is showing less options for number of blocks and QUADs compared to a XCVU9P in the FSGD2104 package.
In the Alveo Product Details in the Alveo U200 and U250 Data Center Accelerator Cards Data Sheet (DS962):
- Figure U200/U250 Block Diagram has the text:
PCIe x 16 or PCIe x8 (2)
- However, the Alveo U200/U250 Accelerator Card Product Details table has: PCIe Interface: Gen3 x16
Table Alveo U200/U250 Features in Alveo U200 and U250 Accelerator Cards User Guide (UG1289) has:
Gen1, 2, or 3 up to x16 and Dual Gen4 x8 compatible
The VU9P doesn't have a "PCIe Gen3 x16/Gen4 x8" block, so the mention of "Dual Gen4 x8 compatible" doesn't seem correct.
Created a Vivado 2025.1 project using the xcvu9p-fsgd2104-2-e part. The DMA/Bridge Subsystem for PCI Express IP allows the following placement options:
| PCIe Block Location | Max lane width | Possible GT Quads |
|---|---|---|
| X0Y1 | x16 | 120 to 123 |
| Y0Y3 | x16 | 124 to 127 |
| X0Y5 | x4 | 131 |
| X1Y2 | x16 | 224 to 228 |
| X1Y4 | x16 | 229 to 232 |
Where each GT Quad can only be selected for use by one PCIe block. I.e. can't see a way for 16 lane PCIe interface to be connected to allow either x16 or x8x8 operation.
Load U200_enum/gen3_x16_normal_order. Enumerates as Current speed 8 GT/s Width x16 in physical_slot=2-2
Load U200_enum/gen3_x8_upper_normal_order. Does not enumerate.
Load U200_enum/gen3_x8_lower_normal_order. Enumerates as Current speed 8 GT/s Width x8 in physical_slot=2-2. The root port width is x16.
Rebooted into Windows and changed the slot 2 bifurcation from Auto to x8x8:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 2 Bifurcation","x8x8"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/10/05" Time="20:38:40" UTC="1">
<SETTING changeStatus="pass" name="Slot 2 Bifurcation" returnCode="0">
<OLDVALUE><![CDATA[Auto]]></OLDVALUE>
<VALUE><![CDATA[x8x8]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
Rebooted into Linux. The U200_enum/gen3_x8_lower_normal_order. Enumerates as Current speed 8 GT/s Width x8, but no physical_slot set. The PCIe root port width was x8.
Load U200_enum/gen3_x8_upper_normal_order. Enumerated and allowed bind_xilinx_devices_to_vfio.sh to bind to it. However, the PC rebooted when attempted to run Load U200_enum/gen3_x8_upper_normal_order.
During the reboot the BIOS reported a suprise link down and timeout (didn't write down the details).
Once has re-booted into Linux U200_enum/gen3_x8_upper_normal_order had enumerated as Current speed 8 GT/s Width x8 on physical_slot=2-3. The PCIe root port is x8 width on physical_slot=2.
Attempted to load U200_enum/gen3_x8_lower_normal_order and U200_enum/gen3_x16_normal_order but they failed to enumerate. Loaded U200_enum/gen3_x8_upper_normal_order which did enumerate.
On the PCIe root ports:
- physical_slot=2-1 has
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- - physical_slot=2 has
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
As well as U200_enum/gen3_x8_lower_normal_order and U200_enum/gen3_x16_normal_order failing to enumerate then loaded by JTAG, the as-delivered endpoint in configuration flash failed to enumerate a power-up. Returned Slot 2 Bifurcation to Auto:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 2 Bifurcation","Auto"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/10/12" Time="11:11:29" UTC="1">
<SETTING changeStatus="pass" name="Slot 2 Bifurcation" returnCode="0">
<OLDVALUE><![CDATA[x8x8]]></OLDVALUE>
<VALUE><![CDATA[Auto]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
FT4232HQ USB-JTAG/UART Interface from the User Guide contains:
The FT4232HQ Quad USB-UART provides a UART connection through the micro-AB USB connector. The FPGA UART TX/RX (two-wire) connection is made through the FT4232HQ BD port. Channel BD implements a 2-wire level-shifted TX/RX UART connection to the FPGA.
Using How to find all serial devices (ttyS, ttyUSB, ..) on Linux without opening them? the serial ports are:
$ ls -l /dev/serial/by-id/
total 0
lrwxrwxrwx 1 root root 13 Sep 27 12:08 usb-FTDI_FT2232H_device_FTVIZUNN-if00-port0 -> ../../ttyUSB0
lrwxrwxrwx 1 root root 13 Sep 27 12:08 usb-FTDI_FT2232H_device_FTVIZUNN-if01-port0 -> ../../ttyUSB1
lrwxrwxrwx 1 root root 13 Sep 27 19:31 usb-Xilinx_A-U200-A32G_22100105Z01C-if00-port0 -> ../../ttyUSB2
lrwxrwxrwx 1 root root 13 Sep 27 19:31 usb-Xilinx_A-U200-A32G_22100105Z01C-if01-port0 -> ../../ttyUSB3
lrwxrwxrwx 1 root root 13 Sep 27 19:31 usb-Xilinx_A-U200-A32G_22100105Z01C-if02-port0 -> ../../ttyUSB4
lrwxrwxrwx 1 root root 13 Sep 27 19:31 usb-Xilinx_A-U200-A32G_22100105Z01C-if03-port0 -> ../../ttyUSB5
In the above assume ttyUSB3 on if01 is the FT4232HQ BD port.
The Additional Links of References in the User Guide gives the part number as Micron MT25QU01GBB8E12-0SIT (although think there is a missing B and the part number should be MT25QU01GBBB8E12-0SIT to match the Micron documentation).
The Micron Search Results for "MT25QU01GBB" says have to register.
Found the Datasheet is on the Digikey site.
Decoding the part number from the datasheet:
| Description | Part number field | Meaning |
|---|---|---|
| Micron Technology | MT | |
| Part Family | 25Q | SPI NOR |
| Voltage | U | 1.7–2.0V |
| Density | 01G | 1Gb (128MB) |
| Stack | B | 2 die/1 S# |
| Device Generation | B | 2nd generation |
| Die Revision | B | Rev. B |
| Pin Configuration Option | 8 | RESET# and HOLD# pin |
| Sector size | E | 64KB sectors, 4KB and 32KB subsectors |
| Package Codes | 12 | 24-ball T-PBGA, 05/6 x 8mm (5 x 5 array) |
| Presumed option separator | - | |
| Security Features | 0 | Standard default security |
| Special Options | S | Standard |
| Operating Temperature | IT | –40°C to +85°C |
| Production Status | Blank | Production |
The datasheet found in Dikikey has the following in Serial Flash Discovery Parameter Data:
See Micron TN-25-06: Serial Flash Discovery Parameters for MT25Q Family for serial Flash discovery parameter data.
Was unable to find TN-25-06 on a site other than Micron, so had to register with Micron to get it.
Also downloaded the MT25QU01GBBB datasheet from Micron, which was the later Rev E compared to the Rev D found on Digikey.
This uses the U200_ibert_100G_ether design, containing the Card Management Solution Subsystem (CMS Subsystem) IP.
With no QSFP+ modules plugged in:
identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Design U200_ibert_100G_ether:
PCI device 0000:31:00.0 rev 00 IOMMU group 22 physical slot 2-2
User access build timestamp : CD32AC96 - 25/10/2025 10:50:22
Quad SPI registers at bar 0 offset 0x44000
SYSMON registers at bar 0 offset 0x40000
CMS software profile U200/U250
CMS firmware version 1.2.25 (0x0C010219)
Card S/N : 22100105Z01C
MAC address 0 : 00:0A:35:06:A4:AA
MAC address 1 : 00:0A:35:06:A4:AB
MAC address 2 : FF:FF:FF:FF:FF:FF
MAC address 3 : FF:FF:FF:FF:FF:FF
Card revision : 1.0
Card name : AU200A32G
Satellite version : 3.1
Total power available: 225W
Fan presence : P
Config mode : Master_SPI_x4
QSFP 0 : Interrupt Clear
QSFP 0 : Module not Present
QSFP 0 : Module Selected
QSFP 0 : High Power Mode
QSFP 0 : Reset Clear
QSFP 1 : Interrupt Clear
QSFP 1 : Module not Present
QSFP 1 : Module Selected
QSFP 1 : High Power Mode
QSFP 1 : Reset Clear
xilinx_sensors/display_sensor_values
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Displaying SYSMON values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 22:
SYSMON instance 0 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vccint Vccaux Vbram Cal Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0xEC8A
Channel Measurement Min Max
Temp 38.5878C 36.5982C 41.0746C
Vccint 0.8467V 0.8438V 0.8496V
Vccaux 1.8223V 1.8193V 1.8311V
Vbram 0.8525V 0.8496V 0.8555V
Vuser0 1.8135V 1.8076V 1.8164V
Vuser1 0.8965V 0.8906V 0.8965V
Vuser2 0.8525V 0.8496V 0.8555V
Vuser3 1.1982V 1.1924V 1.2041V
SYSMON instance 1 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x02CE
Channel Measurement Min Max
Temp 36.5982C 35.1061C 39.0851C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8525V
Vuser0 1.1982V 1.1953V 1.2041V
Vuser1 0.8525V 0.8496V 0.8555V
Vuser2 1.1982V 1.1924V 1.2041V
Vuser3 0.8525V 0.8496V 0.8555V
SYSMON instance 2 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x8D9C
Channel Measurement Min Max
Temp 38.0904C 36.1009C 40.5773C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8525V
Vuser0 0.8525V 0.8496V 0.8555V
Vuser1 1.1895V 1.1865V 1.1924V
Vuser2 1.8252V 1.8223V 1.8311V
Vuser3 0.8906V 0.8877V 0.8936V
Displaying CMS values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 22:
Sensor Max Average Instantaneous
3V3_AUX 3.312V 1.656V 3.312V
3V3_PEX 3.349V 1.674V 3.349V
12V_AUX 12.341V 6.170V 12.341V
12V_AUX_I_IN 0.610A 0.305A 0.610A
12V_PEX 12.181V 6.090V 12.181V
12V_SW 12.287V 6.143V 12.287V
12VPEX_I_IN 1.369A 0.684A 1.369A
CAGE_TEMP0 0C 0C 0C
CAGE_TEMP1 0C 0C 0C
DDR4_VPP_BTM 2.500V 1.250V 2.500V
DDR4_VPP_TOP 2.500V 1.250V 2.500V
DIMM_TEMP0 27C 13C 27C
DIMM_TEMP1 29C 14C 29C
DIMM_TEMP2 35C 17C 35C
DIMM_TEMP3 32C 16C 32C
FAN_SPEED 1063RPM 531RPM 1063RPM
FAN_TEMP 42C 21C 42C
FPGA_TEMP 38C 19C 38C
MGT0V9AVCC 0.902V 0.451V 0.902V
MGTAVTT 1.207V 0.603V 1.207V
SE98_TEMP0 42C 21C 42C
SE98_TEMP1 30C 15C 30C
SE98_TEMP2 46C 23C 46C
SYS_5V5 5.505V 2.752V 5.505V
VCC0V85 0.857V 0.428V 0.857V
VCC1V2_BTM 0.000V 0.000V 0.000V
VCC1V2_TOP 1.207V 0.603V 1.207V
VCC1V8 1.834V 0.917V 1.834V
VCCINT 0.850V 0.425V 0.850V
VCCINT_I 8.124A 4.062A 8.124A
VCCINT_TEMP 0C 0C 0C
12V_AUX_POWER 7.528W 1.882W 7.528W
12V_PEX_POWER 16.676W 4.166W 16.676W
Power Good
After plugging in a pair of INNOLIGHT TR-FC13T-NFB QSFP modules connected with a LC fibre, and with IBERT running successfully:
identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Design U200_ibert_100G_ether:
PCI device 0000:31:00.0 rev 00 IOMMU group 22 physical slot 2-2
User access build timestamp : CD32AC96 - 25/10/2025 10:50:22
Quad SPI registers at bar 0 offset 0x44000
SYSMON registers at bar 0 offset 0x40000
CMS software profile U200/U250
CMS firmware version 1.2.25 (0x0C010219)
Card S/N : 22100105Z01C
MAC address 0 : 00:0A:35:06:A4:AA
MAC address 1 : 00:0A:35:06:A4:AB
MAC address 2 : FF:FF:FF:FF:FF:FF
MAC address 3 : FF:FF:FF:FF:FF:FF
Card revision : 1.0
Card name : AU200A32G
Satellite version : 3.1
Total power available: 225W
Fan presence : P
Config mode : Master_SPI_x4
QSFP 0 : Interrupt Set
QSFP 0 : Module Present
QSFP 0 : Module Selected
QSFP 0 : High Power Mode
QSFP 0 : Reset Clear
QSFP 1 : Interrupt Set
QSFP 1 : Module Present
QSFP 1 : Module Selected
QSFP 1 : High Power Mode
QSFP 1 : Reset Clear
xilinx_sensors/display_sensor_values
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Displaying SYSMON values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 22:
SYSMON instance 0 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vccint Vccaux Vbram Cal Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0xEC8A
Channel Measurement Min Max
Temp 39.5825C 37.0956C 41.5720C
Vccint 0.8496V 0.8438V 0.8525V
Vccaux 1.8281V 1.8193V 1.8281V
Vbram 0.8525V 0.8496V 0.8555V
Vuser0 1.8105V 1.8076V 1.8164V
Vuser1 0.8936V 0.8906V 0.8965V
Vuser2 0.8525V 0.8496V 0.8555V
Vuser3 1.1982V 1.1924V 1.2041V
SYSMON instance 1 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x02CE
Channel Measurement Min Max
Temp 37.5930C 35.6035C 40.0799C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8525V
Vuser0 1.1982V 1.1924V 1.2041V
Vuser1 0.8496V 0.8496V 0.8555V
Vuser2 1.1982V 1.1953V 1.2041V
Vuser3 0.8525V 0.8496V 0.8555V
SYSMON instance 2 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x8D9C
Channel Measurement Min Max
Temp 40.0799C 36.5982C 41.5720C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8525V
Vuser0 0.8525V 0.8496V 0.8555V
Vuser1 1.1895V 1.1865V 1.1953V
Vuser2 1.8252V 1.8223V 1.8311V
Vuser3 0.8906V 0.8877V 0.8936V
Displaying CMS values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 22:
Sensor Max Average Instantaneous
3V3_AUX 3.327V 1.663V 3.327V
3V3_PEX 3.335V 1.667V 3.335V
12V_AUX 12.337V 6.168V 12.337V
12V_AUX_I_IN 0.668A 0.334A 0.668A
12V_PEX 12.124V 6.062V 12.124V
12V_SW 12.292V 6.146V 12.292V
12VPEX_I_IN 1.951A 0.975A 1.951A
CAGE_TEMP0 0C 0C 0C
CAGE_TEMP1 0C 0C 0C
DDR4_VPP_BTM 2.500V 1.250V 2.500V
DDR4_VPP_TOP 2.500V 1.250V 2.500V
DIMM_TEMP0 28C 14C 28C
DIMM_TEMP1 29C 14C 29C
DIMM_TEMP2 36C 18C 36C
DIMM_TEMP3 32C 16C 32C
FAN_SPEED 1063RPM 531RPM 1063RPM
FAN_TEMP 42C 21C 42C
FPGA_TEMP 39C 19C 39C
MGT0V9AVCC 0.904V 0.452V 0.904V
MGTAVTT 1.207V 0.603V 1.207V
SE98_TEMP0 43C 21C 43C
SE98_TEMP1 30C 15C 30C
SE98_TEMP2 48C 24C 48C
SYS_5V5 5.512V 2.756V 5.512V
VCC0V85 0.858V 0.429V 0.858V
VCC1V2_BTM 0.000V 0.000V 0.000V
VCC1V2_TOP 1.206V 0.603V 1.206V
VCC1V8 1.836V 0.918V 1.836V
VCCINT 0.851V 0.425V 0.851V
VCCINT_I 9.252A 4.626A 9.252A
VCCINT_TEMP 0C 0C 0C
12V_AUX_POWER 8.241W 2.060W 8.241W
12V_PEX_POWER 23.654W 5.910W 23.654W
Power Good
Notes:
- VCCINT_TEMP reads as zero.
- CAGE_TEMP0 and CAGE_TEMP1 read as zero even when the INNOLIGHT TR-FC13T-NFB were plugged in. The Module temperature was read when fitted to a ConnectX-4.
- Total power with no QSFP modules fitted was 24.2W. When two INNOLIGHT TR-FC13T-NFB plugged in increased to 31.9W.
75174 - Alveo U200 / U250 - Satellite Controller Release Notes shows that:
- Version 3.1 is before the first Manufacturing version
- Version 4.6.21 is the latest
- Changes between versions 3.1 and 4.6.21 include:
- Fixed issue where, in some cases, zero was incorrectly returned for some sensor values.
- New: QSFP temperature reporting on in-band
- Fixed: Fixed in-band temperature reporting
- New: Added support for QSFP management
73654 - Alveo - Custom Flow - CMS IP - Upgrading Satellite Controller Firmware via CMS Firmware describes how to update the CMS firmware.
Since PG348 describes the protocol for performing a firmware upgrade, could in theory add support to xilinx_cms.c for a firmware update. However, if had a bug in the firmware update could potentially brick the Satellite Controller Firmware which could make the U200 unusable - not sure if the Satellite Controller is needed to sequence powering up the FPGA. 000034261 - Alveo SC - Interruption during in-band SC firmware update may corrupt the card does indicate a failed firmware update may require the card not powering up, resulting in needing to RMA the card to AMD. See also [Alveo U50] How to recover Satellite Controller firmware after unsuccessful upgrade?:
We wanted to upgrade Satellite Controller firmware to version 5.2.6 using the CMS IP, but due to a software bug the last section of the firmware was not written to the controller's flash. After that, the connection between SC and CMS stopped working.
Therefore, will try the loadsc linked from the above AMD article.
loadsc_v2.3.zip contains:
- A Linux executable
- Source code
With the U200_ibert_100G_ether FPGA design loaded attempting to run with the -r option to report the current versions fails with being unable to open /dev/mem:
linux@DESKTOP-BVUMP11:~/Downloads/loadsc_v2.3/bin> sudo ./loadsc -d 0000:31:00.0 -r
------------------------------------------------------------------------------------
CMS Satellite Controller Firmware Download Tool v2.3
------------------------------------------------------------------------------------
>> Aquiring BAR address of target card
> PCIe DBDF 0000:31:00.0
> PCIe BAR/Region 0
> PCIe Vendor ID 0x10EE
[ERROR] opening /dev/mem. May need sudo permissions.
Exiting tool
dmesg indicates the issue is due to kernel lockdown due to secure boot:
[10136.274526] Lockdown: loadsc: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
In Windows disabled secure boot:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Configure Legacy Support and Secure Boot","Legacy Support Disable and Secure Boot Disable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/11/11" Time="13:39:15" UTC="0">
<SETTING changeStatus="pass" name="Configure Legacy Support and Secure Boot" returnCode="0">
<OLDVALUE><![CDATA[Legacy Support Disable and Secure Boot Enable]]></OLDVALUE>
<VALUE><![CDATA[Legacy Support Disable and Secure Boot Disable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
Power cycled the PC after disabling secure boot. On powering up the BIOS prompted for confirmation to disable secure boot.
With secure boot loadsc get further, but fails reporting can't find the CMS subsystem:
linux@DESKTOP-BVUMP11:~/Downloads/loadsc_v2.3/bin> sudo ./loadsc -d 0000:31:00.0 -r -v
------------------------------------------------------------------------------------
CMS Satellite Controller Firmware Download Tool v2.3
------------------------------------------------------------------------------------
>> Aquiring BAR address of target card
> PCIe DBDF 0000:31:00.0
> PCIe BAR/Region 0
> PCIe Vendor ID 0x10EE
<< Aquiring BAR address of target card (success)
< PCIe Device ID 0x903F
< PCIe BAR Address 0x97400000
>> Bring CMS Microblaze out of reset and establish comms with satellite controller
> CMS Subsystem Base Address 0x00000000
[ERROR] CMS Subsystem was not found at offset 0x00000000 of PCIe BAR Address 0x97400000
Exiting tool
No communication with Satellite Controller has an example of the loadsc successfully reporting version information.
Modified the xilinx_cms.c code to display the CMS build information based upon the loadsc code:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 81
Design U200_ibert_100G_ether:
PCI device 0000:31:00.0 rev 00 IOMMU group 81 physical slot 2-2
User access build timestamp : CD32AC96 - 25/10/2025 10:50:22
Quad SPI registers at bar 0 offset 0x44000
SYSMON registers at bar 0 offset 0x40000
CMS software profile U200/U250
CMS subsystem ID 2
CMS Hardware Version Vivado 2025.1
CMS build info Major 4 Minor 0
CMS build info patch core revision 0x0000000E
CMS build info perforce CL 0x005DAB35
CMS build info scratch 0x00000000
CMS firmware version 1.2.25 (0x0C010219)
Card S/N : 22100105Z01C
MAC address 0 : 00:0A:35:06:A4:AA
MAC address 1 : 00:0A:35:06:A4:AB
MAC address 2 : FF:FF:FF:FF:FF:FF
MAC address 3 : FF:FF:FF:FF:FF:FF
Card revision : 1.0
Card name : AU200A32G
Satellite version : 3.1
Total power available: 225W
Fan presence : P
Config mode : Master_SPI_x4
QSFP 0 : Interrupt Clear
QSFP 0 : Module not Present
QSFP 0 : Module Selected
QSFP 0 : High Power Mode
QSFP 0 : Reset Clear
QSFP 1 : Interrupt Clear
QSFP 1 : Module not Present
QSFP 1 : Module Selected
QSFP 1 : High Power Mode
QSFP 1 : Reset Clear
Where:
- CMS subsystem ID of 2 is the value the loadsc code is checking for to detect the CMS subsystem.
- CMS Hardware Version matches the Vivado version used to build the FPGA.
- CMS build info Major/Minor matches the PG348 version.
Compling the loadsc from source, after editing compile.bash to add -g for debugging information and stepping loadsc in the debugger showed:
- Mapping the expected addresses in
/dev/mem. - But the read of
BUILD_INFO_VIV_ID_VERSIONreturned all ones, which explains the failure.
The 5.14.21-150500.55.62-default Kernel has been compiled with CONFIG_STRICT_DEVMEM enabled:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> grep DEVMEM /boot/config-`uname -r`
CONFIG_DEVMEM=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
CONFIG_STRICT_DEVMEM=y
CONFIG_IO_STRICT_DEVMEM=y
Tried booting with manally adding both intel_iommu=on and iomem=relaxed to the command line. That still resulted in loadsc reading all ones.
When booted with intel_iommu=on and iomem=relaxed found the following sequence was repeatible:
- Bind the vfio-pci module to the U200_ibert_100G_ether endpoint.
- Run display_identified_pcie_fpga_designs in the debugger and step-over the
identify_pcie_fpga_designscall. This opens the endpoint using VFIO, but hasn't yet accessed the endpoint registers. - While display_identified_pcie_fpga_designs paused in the debugger, run
loadsc -rand was able to identify the CMS subsystem. - Resume display_identified_pcie_fpga_designs, which successfully accesses the CMS and displays the CMS information.
Comments in set_pci_resource_user_permission.sh note that for a PCIe endpoint with the memory space not enabled:
- Reads return all ones.
- Writes are ignored.
Looking at the loadsc source code can't see anything which ensure memory space is enabled on the endpoint.
Booted with secure boot disabled, and no manual edits to the command line arguments. Resulting arguments:
> cat /proc/cmdline
BOOT_IMAGE=(hd0)/boot/x86_64/loader/linux splash=silent quiet systemd.show_status=yes root=live:CDLABEL=openSUSE_Leap_15.5_GNOME_Live rd.live.image rd.live.overlay.persistent rd.live.overlay.cowfs=ext4
No driver bound to the endpoint, and memory access disabled:
> dump_info/dump_pci_info_pciutils
domain=0000 bus=31 dev=00 func=00 rev=00
vendor_id=10ee (Xilinx Corporation) device_id=903f (Device 903f) subvendor_id=0002 subdevice_id=001f
physical_slot=2-2
control: I/O- Mem- BusMaster- ParErr- SERR+ DisINTx-
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
bar[0] base_addr=97400000 size=80000 is_IO=0 is_prefetchable=0 is_64=1
Capabilities: [40] Power Management
Capabilities: [48] Message Signaled Interrupts
Capabilities: [60] MSI-X
Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port # 0 ASPM not supported
L0s Exit Latency More than 4 μs
L1 Exit Latency More than 64 μs
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Run script which enables memory access:
> (PATH=$PATH:/sbin ~/fpga_sio/software_tests/eclipse_project/set_pci_resource_user_permission.sh 10ee)
Giving user permission to PCI resources for 0000:31:00.0 10ee:903f [0002:001f]
The pre-compiled loadsc can now read back the CMS build information:
linux@DESKTOP-BVUMP11:~/Downloads/loadsc_v2.3/bin> sudo ./loadsc -d 0000:31:00.0 -r -v
------------------------------------------------------------------------------------
CMS Satellite Controller Firmware Download Tool v2.3
------------------------------------------------------------------------------------
>> Aquiring BAR address of target card
> PCIe DBDF 0000:31:00.0
> PCIe BAR/Region 0
> PCIe Vendor ID 0x10EE
<< Aquiring BAR address of target card (success)
< PCIe Device ID 0x903F
< PCIe BAR Address 0x97400000
>> Bring CMS Microblaze out of reset and establish comms with satellite controller
> CMS Subsystem Base Address 0x00000000
DBG: Microblaze is in reset state
DBG: Establishing comms with satellite controller
DBG: Establishing comms with satellite controller (success)
<< Bring CMS Microblaze out of reset and establish comms with satellite controller (success)
DBG: CMS Satellite Comms Version 2
< CMS Firmware Version 1.2.25 (0x0C010219)
< CMS Hardware Version Vivado 2025.1 DBG: v4.0
< SC Firmware Version 3.1
------------------------------------------------------------------------------------
The current Satellite Controller firmware version is 3.1
------------------------------------------------------------------------------------
Downloaded the firmware files on Alveo - Custom Flow - Latest CMS IP and Satellite Controller Firmware. Needed to login to an AMD account before could downloadl.
Performed the firmware update, which was reported as successful:
linux@DESKTOP-BVUMP11:~/Downloads/loadsc_v2.3/bin> sudo ./loadsc -f ~/Downloads/S3-Files\ 2025-11-02\ 08-01-52\ AM/SC_U200_U250_4_6_21.txt -d 0000:31:00.0 -v
------------------------------------------------------------------------------------
CMS Satellite Controller Firmware Download Tool v2.3
------------------------------------------------------------------------------------
>> Aquiring BAR address of target card
> PCIe DBDF 0000:31:00.0
> PCIe BAR/Region 0
> PCIe Vendor ID 0x10EE
<< Aquiring BAR address of target card (success)
< PCIe Device ID 0x903F
< PCIe BAR Address 0x97400000
>> Bring CMS Microblaze out of reset and establish comms with satellite controller
> CMS Subsystem Base Address 0x00000000
>> Reading SC image file "/home/linux/Downloads/S3-Files 2025-11-02 08-01-52 AM/SC_U200_U250_4_6_21.txt"
DBG: SC Image contains 4 records
DBG: Record 0 contains 00000324 octets and has start address 00000000
DBG: Record 1 contains 00175308 octets and has start address 00000200
DBG: Record 2 contains 00007581 octets and has start address 0002aed0
DBG: Record 3 contains 00001280 octets and has start address 0002cc70
<< Reading SC image file "/home/linux/Downloads/S3-Files 2025-11-02 08-01-52 AM/SC_U200_U250_4_6_21.txt" (success)
DBG: Microblaze is already out of reset
<< Bring CMS Microblaze out of reset and establish comms with satellite controller (success)
DBG: CMS Satellite Comms Version 2
< CMS Firmware Version 1.2.25 (0x0C010219)
< CMS Hardware Version Vivado 2025.1 DBG: v4.0
< SC Firmware Version 3.1
>> DownloadSequence_Initial (start)
<< DownloadSequence_Initial (success)
>> DownloadSequence_EraseFirmware (start)
<< DownloadSequence_EraseFirmware (success)
>> DownloadSequence_SendImage (start)
>> Image downloading: .....
<< DownloadSequence_SendImage (success)
>> DownloadSequence_JumpToResetVector (start)
<< DownloadSequence_JumpToResetVector (success)
------------------------------------------------------------------------------------
Satellite Controller firmware has been succesfully updated from 3.1 to 4.6.21
------------------------------------------------------------------------------------
Read back, and reports the updated version:
linux@DESKTOP-BVUMP11:~/Downloads/loadsc_v2.3/bin> sudo ./loadsc -d 0000:31:00.0 -r -v
------------------------------------------------------------------------------------
CMS Satellite Controller Firmware Download Tool v2.3
------------------------------------------------------------------------------------
>> Aquiring BAR address of target card
> PCIe DBDF 0000:31:00.0
> PCIe BAR/Region 0
> PCIe Vendor ID 0x10EE
<< Aquiring BAR address of target card (success)
< PCIe Device ID 0x903F
< PCIe BAR Address 0x97400000
>> Bring CMS Microblaze out of reset and establish comms with satellite controller
> CMS Subsystem Base Address 0x00000000
DBG: Microblaze is already out of reset
<< Bring CMS Microblaze out of reset and establish comms with satellite controller (success)
DBG: CMS Satellite Comms Version 9
< CMS Firmware Version 1.2.25 (0x0C010219)
< CMS Hardware Version Vivado 2025.1 DBG: v4.0
< SC Firmware Version 4.6.21
------------------------------------------------------------------------------------
The current Satellite Controller firmware version is 4.6.21
------------------------------------------------------------------------------------
When booted without the intel_iommu=on argument, attempted to bind the vfio-pci module. Expected to use NOIOMMU mode, but resulted in the error:
[18926.634606] vfio: unknown parameter 'enable_unsafe_noiommu_mode' ignored
[18926.634755] VFIO - User Level meta-driver version: 0.3
[18926.770789] vfio-pci: probe of 0000:31:00.0 failed with error -22
I.e. looks like the vfio-pci module has been built without NOIOMMU mode support.
Rebooted the PC, and this time manually edit the arguments to contain intel_iommu=on. Running with INNOLIGHT TR-FC13T-NFB QSFP modules plugged in:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 81
Design U200_ibert_100G_ether:
PCI device 0000:31:00.0 rev 00 IOMMU group 81 physical slot 2-2
User access build timestamp : CD32AC96 - 25/10/2025 10:50:22
Quad SPI registers at bar 0 offset 0x44000
SYSMON registers at bar 0 offset 0x40000
CMS software profile U200/U250
CMS subsystem ID 2
CMS Hardware Version Vivado 2025.1
CMS build info Major 4 Minor 0
CMS build info patch core revision 0x0000000E
CMS build info perforce CL 0x005DAB35
CMS build info scratch 0x00000000
CMS firmware version 1.2.25 (0x0C010219)
Card S/N : 22100105Z01C
MAC address 0 : 00:0A:35:06:A4:AA
MAC address 1 : 00:0A:35:06:A4:AB
Card revision : 1.0
Card name : AU200A32G
Satellite version : 4.6.21
Total power available: 225W
Fan presence : P
Config mode : Master_SPI_x4
QSFP 0 : Interrupt Set
QSFP 0 : Module Present
QSFP 0 : Module Selected
QSFP 0 : High Power Mode
QSFP 0 : Reset Clear
QSFP 1 : Interrupt Set
QSFP 1 : Module Present
QSFP 1 : Module Selected
QSFP 1 : High Power Mode
QSFP 1 : Reset Clear
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> xilinx_sensors/display_sensor_values
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 81
Displaying SYSMON values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 81:
SYSMON instance 0 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vccint Vccaux Vbram Cal Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0xEC8A
Channel Measurement Min Max
Temp 41.0746C 37.5930C 42.0694C
Vccint 0.8467V 0.8438V 0.8496V
Vccaux 1.8252V 1.8193V 1.8281V
Vbram 0.8525V 0.8496V 0.8555V
Vuser0 1.8105V 1.8076V 1.8164V
Vuser1 0.8936V 0.8906V 0.8965V
Vuser2 0.8525V 0.8496V 0.8555V
Vuser3 1.2012V 1.1924V 1.2012V
SYSMON instance 1 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x02CE
Channel Measurement Min Max
Temp 39.0851C 36.1009C 41.0746C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8555V
Vuser0 1.1982V 1.1924V 1.2041V
Vuser1 0.8525V 0.8496V 0.8555V
Vuser2 1.2012V 1.1953V 1.2041V
Vuser3 0.8525V 0.8496V 0.8555V
SYSMON instance 2 samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp Vuser0 Vuser1 Vuser2 Vuser3
Analog Bus configuration 0x8D9C
Channel Measurement Min Max
Temp 39.0851C 37.5930C 41.5720C
Vccint 0.8496V
Vccaux 1.8281V
Vbram 0.8525V
Vuser0 0.8525V 0.8496V 0.8555V
Vuser1 1.1924V 1.1865V 1.1924V
Vuser2 1.8281V 1.8223V 1.8311V
Vuser3 0.8906V 0.8877V 0.8936V
Displaying CMS values for design U200_ibert_100G_ether in PCI device 0000:31:00.0 IOMMU group 81:
Sensor Max Average Instantaneous
3V3_AUX 3.325V 1.662V 3.325V
3V3_PEX 3.333V 1.666V 3.333V
12V_AUX 12.344V 6.172V 12.344V
12V_AUX_I_IN 0.647A 0.323A 0.647A
12V_PEX 12.109V 6.054V 12.109V
12V_SW 12.294V 6.147V 12.294V
12VPEX_I_IN 1.898A 0.949A 1.898A
CAGE_TEMP0 47C 23C 47C
CAGE_TEMP1 47C 23C 47C
DDR4_VPP_BTM 2.500V 1.250V 2.500V
DDR4_VPP_TOP 2.500V 1.250V 2.500V
DIMM_TEMP0 28C 14C 28C
DIMM_TEMP1 29C 14C 29C
DIMM_TEMP2 35C 17C 35C
DIMM_TEMP3 32C 16C 32C
FAN_SPEED 1062RPM 531RPM 1062RPM
FAN_TEMP 41C 20C 41C
FPGA_TEMP 39C 19C 39C
MGT0V9AVCC 0.903V 0.451V 0.903V
MGTAVTT 1.207V 0.603V 1.207V
SE98_TEMP0 41C 20C 41C
SE98_TEMP1 30C 15C 30C
SE98_TEMP2 44C 22C 44C
SYS_5V5 5.512V 2.756V 5.512V
VCC0V85 0.859V 0.429V 0.859V
VCC1V2_BTM 1.205V 0.602V 1.205V
VCC1V2_TOP 1.203V 0.601V 1.203V
VCC1V8 1.835V 0.917V 1.835V
VCCINT 0.850V 0.425V 0.850V
VCCINT_I 9.156A 4.578A 9.156A
VCCINT_TEMP 44C 22C 44C
12V_AUX_POWER 7.987W 1.994W 7.987W
12V_PEX_POWER 22.983W 5.745W 22.983W
Power Good
Compared to the original Satellite Controller Firmware v3.1:
- The reported firmware version is 4.6.21, which is the latest.
- Only two MAC addresses are reported. The spurious MAC address 2 and MAC address 3 with all FF's are no longer reported.
- The CAGE_TEMP0 and CAGE_TEMP1 are now non-zero.
Left the power off at the mains for 10 minutes, since the Satellite Controller is powered by the standby supply. Powered on and checked the Satellite Controller Firmware was still v4.6.21 and could communicate with it.
Re-enabled secure boot from Windows:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Configure Legacy Support and Secure Boot","Legacy Support Disable and Secure Boot Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/11/12" Time="16:54:40" UTC="0">
<SETTING changeStatus="pass" name="Configure Legacy Support and Secure Boot" returnCode="0">
<OLDVALUE><![CDATA[Legacy Support Disable and Secure Boot Disable]]></OLDVALUE>
<VALUE><![CDATA[Legacy Support Disable and Secure Boot Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
The Figure 4: Alveo U200/U250 Clocking in the Alveo™ U200 and U250 Accelerator Cards User Guide shows the MGTREFCLK1 on QUAD 231 (QSFP0) and QUAD 230 (QSFP1) are selectable via GPIOs connected to SI5335A-B06201-GM devices.
The constraints in https://www.amd.com/bin/public/amdOpenDownload?filename=alveo-u250-xdc_20210909.zip have:
# Input Clocks and Controls for QSFP28 Port 0
#
# MGT_SI570_CLOCK0 -> MGT Ref Clock 0 156.25MHz Default (User re-programmable)
# QSFP0_CLOCK -> MGT Ref Clock 1 User selectable by QSFP0_FS
#
set_property PACKAGE_PIN M10 [get_ports MGT_SI570_CLOCK0_N]; # Bank 231 Net "MGT_SI570_CLOCK0_C_N" - MGTREFCLK0N_231
set_property PACKAGE_PIN M11 [get_ports MGT_SI570_CLOCK0_P]; # Bank 231 Net "MGT_SI570_CLOCK0_C_P" - MGTREFCLK0P_231
set_property PACKAGE_PIN K10 [get_ports QSFP0_CLOCK_N ]; # Bank 231 Net "QSFP0_CLOCK_N" - MGTREFCLK1N_231
set_property PACKAGE_PIN K11 [get_ports QSFP0_CLOCK_P ]; # Bank 231 Net "QSFP0_CLOCK_P" - MGTREFCLK1P_231
#
# QSFP0 Clock Control Signals
# FS[1:0] <-- Clock Select Pin FS[1:0] = 1X -> 161.132812 MHz 1.8V LVDS (default when FPGA pin Hi-Z due to 10K pullups)
# FS[1:0] = 01 -> 156.250000 MHz 1.8V LVDS
# RESET <-- Device Reset - Asserting this pin (driving high) is required to change FS1,FS0 pin setting.
#
set_property -dict {PACKAGE_PIN AT20 IOSTANDARD LVCMOS12 } [get_ports QSFP0_FS[0] ]; # Bank 64 VCCO - VCC1V2 Net "QSFP0_FS0" - IO_L10P_T1U_N6_QBC_AD4P_64
set_property -dict {PACKAGE_PIN AU22 IOSTANDARD LVCMOS12 } [get_ports QSFP0_FS[1] ]; # Bank 64 VCCO - VCC1V2 Net "QSFP0_FS1" - IO_L9N_T1L_N5_AD12N_64
set_property -dict {PACKAGE_PIN AT22 IOSTANDARD LVCMOS12 } [get_ports QSFP0_REFCLK_RESET]; # Bank 64 VCCO - VCC1V2 Net "QSFP0_REFCLK_RESET" - IO_L9P_T1L_N4_AD12P_64
#
# Input Clocks and Controls for QSFP28 Port 1
#
# MGT_SI570_CLOCK1 -> MGT Ref Clock 0 156.25MHz Default (User re-programmable)
# QSFP1_CLOCK -> MGT Ref Clock 1 User selectable by QSFP0_FS
#
set_property PACKAGE_PIN T10 [get_ports MGT_SI570_CLOCK1_N ]; # Bank 230 Net "MGT_SI570_CLOCK1_C_N" - MGTREFCLK0N_230
set_property PACKAGE_PIN T11 [get_ports MGT_SI570_CLOCK1_P ]; # Bank 230 Net "MGT_SI570_CLOCK1_C_P" - MGTREFCLK0P_230
set_property PACKAGE_PIN P10 [get_ports QSFP1_CLOCK_N ]; # Bank 230 Net "QSFP1_CLOCK_N" - MGTREFCLK1N_230
set_property PACKAGE_PIN P11 [get_ports QSFP1_CLOCK_P ]; # Bank 230 Net "QSFP1_CLOCK_P" - MGTREFCLK1P_230
#
# QSFP1 Control Signals
# RESETL - Active Low Reset output from FPGA to QSFP Module
# MODPRSL - Active Low Module Present input from QSFP to FPGA
# INTL - Active Low Interrupt input from QSFP to FPGA
# LPMODE - Active High Control output from FPGA to QSFP Module to put the device in low power mode (Optics Off)
# MODSEL - Active Low Enable output from FPGA to QSFP Module to select device for I2C Sideband Communication
#
set_property -dict {PACKAGE_PIN BC18 IOSTANDARD LVCMOS12 } [get_ports QSFP1_RESETL ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_RESETL_LS" - IO_L15N_T2L_N5_AD11N_64
set_property -dict {PACKAGE_PIN BC19 IOSTANDARD LVCMOS12 } [get_ports QSFP1_MODPRSL ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_MODPRSL_LS" - IO_L15P_T2L_N4_AD11P_64
set_property -dict {PACKAGE_PIN AV21 IOSTANDARD LVCMOS12 } [get_ports QSFP1_INTL ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_INTL_LS" - IO_L14N_T2L_N3_GC_64
set_property -dict {PACKAGE_PIN AV22 IOSTANDARD LVCMOS12 } [get_ports QSFP1_LPMODE ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_LPMODE_LS" - IO_L14P_T2L_N2_GC_64
set_property -dict {PACKAGE_PIN AY20 IOSTANDARD LVCMOS12 } [get_ports QSFP1_MODSELL ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_MODSELL_LS" - IO_L16P_T2U_N6_QBC_AD3P_64
#
#
# QSFP1 Clock Control Signals
# - FS[1:0] <-- Clock Select Pin FS[1:0] = 1X -> 161.132812 MHz 1.8V LVDS (default when FPGA pin Hi-Z due to 10K pullups)
# FS[1:0] = 01 -> 156.250000 MHz 1.8V LVDS
# - RESET <-- Device Reset - Asserting this pin (driving high) is required to change FS1,FS0 pin setting.
# PINS: "QSFP1_RECLK_RESET" - IO_L8N_T1L_N3_AD5N_64_AR21
#
set_property -dict {PACKAGE_PIN AR22 IOSTANDARD LVCMOS12 } [get_ports QSFP1_FS[0] ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_FS0" - IO_L8P_T1L_N2_AD5P_64
set_property -dict {PACKAGE_PIN AU20 IOSTANDARD LVCMOS12 } [get_ports QSFP1_FS[1] ]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_FS1" - IO_L7N_T1L_N1_QBC_AD13N_64
set_property -dict {PACKAGE_PIN AR21 IOSTANDARD LVCMOS12 } [get_ports QSFP1_REFCLK_RESET]; # Bank 64 VCCO - VCC1V2 Net "QSFP1_REFCLK_RESET" - IO_L8N_T1L_N3_AD5N_64Which says the REFCLK_RESET to the SI5335A devices are active high
The Si5335 datasheet says the RESET pin is active high.
Whereas Table 10: Bank 64 Miscellaneous I/O Descriptions in the Alveo™ U200 and U250 Accelerator Cards User Guide says the RESET pin is active low:
| Signal Name | Direction | Pin | I/O | Description |
|---|---|---|---|---|
| QSFP0_FS0 | Output | AT20 | IO_L10P_T1U_N6_QBC_AD4P_64 | SI5335A Frequency Select Signal. FS[1:0] = 01 -> CLK1A/1B: 156.25 MHz 1.8V LVDS FS[1:0] = 1X -> CLK1A/1B: 161.1328125 MHz 1.8V LVDS |
| QSFP0_FS1 | Output | AU22 | IO_L9N_T1L_N5_AD12N_64 | SI5335A Frequency Select Signal. FS[1:0] = 01 -> CLK1A/1B: 156.25MHz 1.8V LVDS FS[1:0] = 1X -> CLK1A/1B: 161.1328125 MHz 1.8V LVDS |
| QSFP0_REFCLK_RESET | Output | AT22 | IO_L9P_T1L_N4_AD12P_64 | SI5335A Active-Low, Reset Input to Clock Synthesizer. |
| QSFP1_REFCLK_RESET | Output | AR21 | IO_L8N_T1L_N3_AD5N_64 | SI5335A Active-Low, Reset Input to Clock Synthesizer. |
| QSFP1_FS0 | Output | AR22 | IO_L8P_T1L_N2_AD5P_64 | SI5335A Frequency Select Signal. FS[1:0] = 01 -> CLK1A/1B: 156.25MHz 1.8V LVDS FS[1:0] = 1X -> CLK1A/1B: 161.1328125 MHz 1.8V LVDS. |
| QSFP1_FS1 | Output | AU20 | IO_L7N_T1L_N1_QBC_AD13N_64 | SI5335A Frequency Select Signal. FS[1:0] = 01 -> CLK1A/1B: 156.25 MHz 1.8V LVDS FS[1:0] = 1X -> CLK1A/1B: 161.1328125 MHz 1.8V LVDS. |
Bank 64 has a 1.2V VCCO, and since the SI5335A VDD is 1.8V, 2.5V or 3.3V presumably there is a level shifter between the FPGA and SI5335A.
With PC powered off connect 100G DAC cable from U200 QSFP0 to ConnectX-4 MCX415A-CCAT
Power up PC and boot into openSUSE Leap 15.5 with Kernel 5.14.21-150500.55.62-default.
Load the U200_100G_ether_simplex_tx bitstream. This configures the CMAC as:
Run the cmac_link_control program to enable Tx RSFEC and enable transmission:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> ~/fpga_sio/software_tests/eclipse_project/bind_xilinx_devices_to_vfio.sh
IOMMU devices present: dmar0 dmar4 dmar5 dmar6 dmar7
Loading vfio-pci module
Bound vfio-pci driver to 0000:31:00.0 10ee:903f [0002:001d]
Waiting for /dev/vfio/22 to be created
Giving user permission to IOMMU group 22 for 0000:31:00.0 10ee:903f [0002:001d]
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> cmac_ethernet/cmac_link_control
Opening device 0000:31:00.0 (10ee:903f) with IOMMU group 22
Enabled bus master for 0000:31:00.0
core_version = 3.1
gt_reset_all = 0
usr_rx_serdes_reset = 0
usr_rx_reset = 0
usr_tx_reset = 0
switch_core_mode_reg = 1
configuration_tx_reg1_ctl_tx_enable = 0
configuration_tx_reg1_ctl_tx_send_lfi = 0
configuration_tx_reg1_ctl_tx_send_rfi = 0
configuration_tx_reg1_ctl_tx_send_idle = 0
configuration_tx_reg1_ctl_tx_test_pattern = 0
core_mode_reg = 1
rsfec_config_enable_ctl_rx_rsfec_enable = 0
rsfec_config_enable_ctl_tx_rsfec_enable = 0
Enabling TX RSFEC
core_version = 3.1
gt_reset_all = 0
usr_rx_serdes_reset = 0
usr_rx_reset = 0
usr_tx_reset = 0
switch_core_mode_reg = 1
configuration_tx_reg1_ctl_tx_enable = 0
configuration_tx_reg1_ctl_tx_send_lfi = 0
configuration_tx_reg1_ctl_tx_send_rfi = 0
configuration_tx_reg1_ctl_tx_send_idle = 0
configuration_tx_reg1_ctl_tx_test_pattern = 0
core_mode_reg = 1
rsfec_config_enable_ctl_rx_rsfec_enable = 0
rsfec_config_enable_ctl_tx_rsfec_enable = 1
Setting TX_ENABLE
core_version = 3.1
gt_reset_all = 0
usr_rx_serdes_reset = 0
usr_rx_reset = 0
usr_tx_reset = 0
switch_core_mode_reg = 1
configuration_tx_reg1_ctl_tx_enable = 1
configuration_tx_reg1_ctl_tx_send_lfi = 0
configuration_tx_reg1_ctl_tx_send_rfi = 0
configuration_tx_reg1_ctl_tx_send_idle = 0
configuration_tx_reg1_ctl_tx_test_pattern = 0
core_mode_reg = 1
rsfec_config_enable_ctl_rx_rsfec_enable = 0
rsfec_config_enable_ctl_tx_rsfec_enable = 1
Settings changed. Press return to close the VFIO devices.
Link is detected, with the ConnectX-4 still set to autonegotiate:
inux@DESKTOP-BVUMP11:~> sudo sudo /usr/sbin/ethtool enp74s0np0
Settings for eth12:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: None RS BASER
Speed: 100000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000004 (4)
link
Link detected: yes
RS FEC encoding has been detected:
linux@DESKTOP-BVUMP11:~> sudo sudo /usr/sbin/ethtool --show-fec enp74s0np0
FEC parameters for eth12:
Configured FEC encodings: Auto
Active FEC encoding: RS
ibv_devinfo reports a 100G link:
linux@DESKTOP-BVUMP11:~> ibv_devinfo -d mlx5_0 -v
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 12.28.2006
node_guid: 9803:9b03:006a:7ea1
sys_image_guid: 9803:9b03:006a:7ea1
vendor_id: 0x02c9
vendor_part_id: 4115
hw_ver: 0x0
board_id: MT_2180110032
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 262144
max_qp_wr: 32768
device_cap_flags: 0xed721c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
MEM_WINDOW
XRC
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
RAW_IP_CSUM
MANAGED_FLOW_STEERING
Unknown flags: 0xC8400000
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 16777216
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 4194304
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
general_odp_caps:
ODP_SUPPORT
ODP_SUPPORT_IMPLICIT
rc_odp_caps:
SUPPORT_SEND
SUPPORT_RECV
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
SUPPORT_SEND
xrc_odp_caps:
SUPPORT_SEND
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
completion timestamp_mask: 0x7fffffffffffffff
hca_core_clock: 156250kHZ
raw packet caps:
C-VLAN stripping offload
Scatter FCS offload
IP csum offload
Delay drop
device_cap_flags_ex: 0x15ED721C36
RAW_SCATTER_FCS
PCI_WRITE_END_PADDING
Unknown flags: 0x100000000
tso_caps:
max_tso: 262144
supported_qp:
SUPPORT_RAW_PACKET
rss_caps:
max_rwq_indirection_tables: 65536
max_rwq_indirection_table_size: 2048
rx_hash_function: 0x1
rx_hash_fields_mask: 0x800000FF
supported_qp:
SUPPORT_RAW_PACKET
max_wq_type_rq: 8388608
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
tag matching not supported
cq moderation caps:
max_cq_count: 65535
max_cq_period: 4095 us
num_comp_vectors: 63
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x40000000
port_cap_flags: 0x04010000
port_cap_flags2: 0x0000
max_vl_num: invalid value (0)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 1
gid_tbl_len: 256
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0000:9a03:9bff:fe6a:7ea1, RoCE v1
GID[ 1]: fe80::9a03:9bff:fe6a:7ea1, RoCE v2