Skip to content

Instantly share code, notes, and snippets.

@Chester-Gillon

Chester-Gillon/open-nic_on_Alveo_U200.md

Last active December 6, 2025 10:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Chester-Gillon/047dc6b5977f0aeefdd896b50eb662f4 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Chester-Gillon/047dc6b5977f0aeefdd896b50eb662f4 to your computer and use it in GitHub Desktop.
Download ZIP
open-nic on Alveo U200
Raw
open-nic_on_Alveo_U200.md

0. Introduction

Contains notes about using the open-nic project on an Alveo U200.

1. Clone projects

1.1 Cloned tagged projects via open-nic, but doesn't support the U200

Using SmartGit cloned https://github.com/Xilinx/open-nic into ~/U200_open-nic/open-nic.

Ran the script/checkout.sh from the above cloned repository, which then clones other repositories:

mr_halfword@Haswell-Ubuntu:~/U200_open-nic/open-nic/script$ ./checkout.sh ~/U200_open-nic/
Checking out OpenNIC version 1.0...
Cloning open-nic-shell into /home/mr_halfword/U200_open-nic/github.com/Xilinx/open-nic-shell...
Checking out open-nic-shell @ 1.0
Cloning open-nic-driver into /home/mr_halfword/U200_open-nic/github.com/Xilinx/open-nic-driver...
Checking out open-nic-driver @ 1.0
OpenNIC version 1.0 checked out

Looking at the /home/mr_halfword/U200_open-nic/github.com/Xilinx/open-nic-shell which got checked out shows that was a version which only supports the U250 or U280.

The open-nic/script/checkout.sh is checking out tag 1.0 of the open-nic-shell and open-nic-driver project.

Looks like the open-nic-shell project has been updated to support more Alveo cards than the U250 or U280 in the initial 1.0 tag, but new tags haven't been created.

Therefore, remove the above cloned projects.

Also, the open-nic project doesn't try and checkout the open-nic-dpdk project, which looks to have been created after the last update to open-nic.

1.2 Clone open-nic-shell, open-nic-driver and open-nic-dpdk projects directly

Using SmartGit:

  1. Clone https://github.com/Xilinx/open-nic-shell into ~/U200_open-nic/open-nic-shell. The README.md for this cloned project says supports the U200. There are open-nic-shell/constr/au200 and open-nic-shell/board_files/Xilinx/au200 directories.
  2. Clone https://github.com/Xilinx/open-nic-driver into ~/U200_open-nic/open-nic-driver
  3. Clone https://github.com/Xilinx/open-nic-dpdk into ~/U200_open-nic/open-nic-dpdk

2. Create abd build project

The open-nic-shell README.txt indicates the latest Vivado version used is 2022.1. Therefore, added Alveo support to an existing 2022.1 installation.

On the first attempt made an typo in the command line by omitting a space between two argument (-impl 0max_pkt_len 9600). That mistake didn't result in an obvious error. Rather created the project using the default of maximum packet length of 1514, one phyical function and one CMAC port:

$ /opt/Xilinx/Vivado/2022.1/bin/vivado -mode gui -source build.tcl -tclargs -board au200 -impl 0max_pkt_len 9600 -num_cmac_port 2 -num_phys_func 2

After realising the above mistake deleted the created build/au200 directory. Then re-created the project using:

mr_halfword@Haswell-Ubuntu:~/U200_open-nic/open-nic-shell/script$ /opt/Xilinx/Vivado/2022.1/bin/vivado -mode gui -source build.tcl -tclargs -board au200 -impl 0 -max_pkt_len 9600 -num_cmac_port 2 -num_phys_func 2

Where:

  • -board au200 is to target the Alveo U200
  • -impl 0 was to just create the project, and not start the implementation
  • -max_pkt_len 9600 is to configure with the maximum packet length
  • -num_cmac_port 2 -num_phys_func 2 as suggested by the open-nic-dpdk README

Creating the project took over one hour (time not measured exactly).

After generating the bitstream the Critical warnings are:

  • Synthesis:

    [Synth 8-4442] BlackBox module \system_config_inst/system_management_wiz_inst has unconnected pin vp

  • Implementation:

    [Netlist 29-160] Cannot set property 'IOSTANDARD', because the property does not exist for objects of type 'pin'. ["/home/mr_halfword/U200_open-nic/open-nic-shell/build/au200/vivado_ip/system_management_wiz/system_management_wiz.xdc":53]

  • Write Bitstream:

    [Board 49-67] The board_part definition was not found for xilinx.com:au200:part0:1.3. This can happen sometimes when you use custom board part. You can resolve this issue by setting 'board.repoPaths' parameter, pointing to the location of custom board files. Valid board_part values can be retrieved with the 'get_board_parts' Tcl command.

    [Vivado 12-1790] Evaluation License Warning: This design contains one or more IP cores that use separately licensed features. If the design has been configured to make use of evaluation features, please note that these features will cease to function after a certain period of time. Please consult the core datasheet to determine whether the core which you have configured will be affected. Evaluation features should NOT be used in production systems.

    Evaluation cores found in this design: IP core 'cmac_usplus_1' (cmac_usplus_core) was generated with multiple features: IP feature '[email protected]' was enabled using a design_linking license. IP feature '[email protected]' was enabled using a bought license. IP core 'cmac_usplus_0' (cmac_usplus_core) was generated with multiple features: IP feature '[email protected]' was enabled using a design_linking license. IP feature '[email protected]' was enabled using a bought license.

    Resolution: If a new IP Core license was added, in order for the new license to be picked up, the current netlist needs to be updated by resetting and re-generating the IP output products before bitstream generation.

The Synthesis and Implementation warnings are due to the SYSMON vp and vn ports not being connected. Doesn't seem to be an issue.

The "board_part definition was not found" doesn't seem to be an issue. The project summary shows is using /home/mr_halfword/U200_open-nic/open-nic-shell/board_files within the working copy.

The cmac_an_lt license warning doesn't seem to be an issue. Looking at the CMAC configuration the "Include AN/AT Logic" option isn't selected.

Parsing the built bitstream:

$ ~/fpga_sio/software_tests/eclipse_project/bin/release/xilinx_quad_spi/parse_bitstream_file /home/mr_halfword/U200_open-nic/open-nic-shell/build/au200/open_nic_shell/open_nic_shell.runs/impl_1/open_nic_shell.bit
Successfully parsed bitstream of length 36966352 bytes with 1308024 configuration packets with 3 SLRs
Read bitsteam from file /home/mr_halfword/U200_open-nic/open-nic-shell/build/au200/open_nic_shell/open_nic_shell.runs/impl_1/open_nic_shell.bit
.bit format header:
  design_name=open_nic_shell;COMPRESS=TRUE;UserID=DEADC0DE;Version=2022.1
  part_name=xcu200-fsgd2104-2-e
  date=2025/11/22
  time=17:02:57
SLR[0] Sync word at byte index 0x50
  Type 1 packet opcode NOP
  Type 1 packet opcode write register BSPI words 0000066C
  Type 1 packet opcode write register CMD command BSPI_READ
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register AXSS 66669999 - 12/12/2051 09:38:25
  Type 1 packet opcode write register FAR words 00000000
  Type 1 packet opcode write register RBCRC_SW words 00000000
  Type 1 packet opcode write register COR0 words 38423FE5
  Type 1 packet opcode write register COR1 words 00400000
  Type 1 packet opcode write register IDCODE XCU200
  Type 1 packet opcode write register CMD command FALL_EDGE
  Type 1 packet opcode write register CMD command SWITCH
  Type 1 packet opcode NOP
  Type 1 packet opcode write register MASK words 00000001
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00001000
  Type 1 packet opcode NOP (8 consecutive)
  Configuration data writes consisting of:
    284720 NOPs
    41620 FAR writes
    1741 WCFG commands
    1741 FDRI writes with a total of 496806 words
    293 MFW commands
    293 NULL commands
    39879 MFWR writes with a total of 558306 words
    176 Type 2 packets with a total of 2634504 words
  Type 1 packet opcode write register CRC words 6AAACDA2
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command GRESTORE
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DGHIGH_LFRM
  Type 1 packet opcode NOP (20 consecutive)
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00000000
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register FAR words 07FC0000
  Type 1 packet opcode write register MASK words 00000101
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register CRC words 5FFE959E
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (391 consecutive)
SLR[1] Sync word at byte index 0xFA85E4
  Type 1 packet opcode NOP
  Type 1 packet opcode write register BSPI words 0000066C
  Type 1 packet opcode write register CMD command BSPI_READ
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register AXSS 66669999 - 12/12/2051 09:38:25
  Type 1 packet opcode write register FAR words 00000000
  Type 1 packet opcode write register RBCRC_SW words 00000000
  Type 1 packet opcode write register COR0 words 38423FE5
  Type 1 packet opcode write register COR1 words 00400000
  Type 1 packet opcode write register IDCODE unknown (0x4b22093)
  Type 1 packet opcode write register CMD command SWITCH
  Type 1 packet opcode NOP
  Type 1 packet opcode write register MASK words 00000001
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00001000
  Type 1 packet opcode NOP (8 consecutive)
  Configuration data writes consisting of:
    372901 NOPs
    71621 FAR writes
    260 WCFG commands
    260 FDRI writes with a total of 58590 words
    67 MFW commands
    67 NULL commands
    71361 MFWR writes with a total of 999054 words
    2 Type 2 packets with a total of 6417 words
  Type 1 packet opcode write register CRC words 3A0177E4
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command GRESTORE
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DGHIGH_LFRM
  Type 1 packet opcode NOP (20 consecutive)
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00000000
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register FAR words 07FC0000
  Type 1 packet opcode write register MASK words 00000101
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register CRC words 5FFE959E
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (391 consecutive)
SLR[2] Sync word at byte index 0x15F6C24
  Type 1 packet opcode NOP
  Type 1 packet opcode write register BSPI words 0000066C
  Type 1 packet opcode write register CMD command BSPI_READ
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register AXSS 66669999 - 12/12/2051 09:38:25
  Type 1 packet opcode write register FAR words 00000000
  Type 1 packet opcode write register RBCRC_SW words 00000000
  Type 1 packet opcode write register COR0 words 38423FE5
  Type 1 packet opcode write register COR1 words 00400000
  Type 1 packet opcode write register IDCODE unknown (0x4b24093)
  Type 1 packet opcode write register CMD command SWITCH
  Type 1 packet opcode NOP
  Type 1 packet opcode write register MASK words 00000001
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00001000
  Type 1 packet opcode NOP (8 consecutive)
  Configuration data writes consisting of:
    318159 NOPs
    49483 FAR writes
    1417 WCFG commands
    1417 FDRI writes with a total of 403062 words
    256 MFW commands
    256 NULL commands
    48066 MFWR writes with a total of 672924 words
    158 Type 2 packets with a total of 1936353 words
  Type 1 packet opcode write register CRC words 408C9233
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command GRESTORE
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DGHIGH_LFRM
  Type 1 packet opcode NOP (20 consecutive)
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00000000
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register FAR words 07FC0000
  Type 1 packet opcode write register MASK words 00000101
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register CRC words 5FFE959E
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (397 consecutive)
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (10 consecutive)
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (404 consecutive)

The generated AXSS value of 66669999 looks wrong, since decodes as 12/12/2051 09:38:25. Haven't yet looked at the project to see where that value is coming from.

There is BUILD_TIMESTAMP parameter set at build time which can be read from a register in the system configuration block. This is a BCD enodeded MMDDhhmm (see Build Script Options) and therefore is a different lower precision format that the AXSS which includes the year.

3. How the design enumerates on the PCIe bus

After loading the bitstream the VFIO module binds to two physical functions:

linux@DESKTOP-BVUMP11:~> ~/fpga_sio/software_tests/eclipse_project/bind_xilinx_devices_to_vfio.sh 
IOMMU devices present: dmar0  dmar4  dmar5  dmar6  dmar7
Loading vfio-pci module
Bound vfio-pci driver to 0000:31:00.0 10ee:903f [10ee:0007]
Waiting for /dev/vfio/22 to be created
Giving user permission to IOMMU group 22 for 0000:31:00.0 10ee:903f [10ee:0007]
Bound vfio-pci driver to 0000:31:00.1 10ee:913f [10ee:0007]
Waiting for /dev/vfio/22 to be created
Giving user permission to IOMMU group 22 for 0000:31:00.1 10ee:913f [10ee:0007]

However, no BARs were assigned. dmesg reported errors about no free space:

[ 1867.715168] pcieport 0000:30:00.0: pciehp: Slot(2-2): Link Down
[ 1867.715179] pcieport 0000:30:00.0: pciehp: Slot(2-2): Card not present
[ 1867.715190] vfio-pci 0000:31:00.1: can't change power state from D3hot to D0 (config space inaccessible)
[ 1867.774905] vfio-pci 0000:31:00.1: can't change power state from D3cold to D0 (config space inaccessible)
[ 1867.774984] pci 0000:31:00.1: Removing from iommu group 22
[ 1867.775006] vfio-pci 0000:31:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[ 1867.834688] vfio-pci 0000:31:00.0: can't change power state from D3cold to D0 (config space inaccessible)
[ 1867.834851] pci 0000:31:00.0: Removing from iommu group 22
[ 1867.834865] pcieport 0000:30:00.0: pciehp: Slot(2-2): Card present
[ 1867.834866] pcieport 0000:30:00.0: pciehp: Slot(2-2): Link Up
[ 1867.968900] pci 0000:31:00.0: [10ee:903f] type 00 class 0x028000
[ 1867.968959] pci 0000:31:00.0: reg 0x10: [mem 0x00000000-0x0003ffff 64bit]
[ 1867.968994] pci 0000:31:00.0: reg 0x18: [mem 0x00000000-0x003fffff 64bit]
[ 1867.969055] pci 0000:31:00.0: Max Payload Size set to 256 (was 128, max 1024)
[ 1867.969628] pci 0000:31:00.0: Adding to iommu group 22
[ 1867.969707] pci 0000:31:00.1: [10ee:913f] type 00 class 0x028000
[ 1867.969758] pci 0000:31:00.1: reg 0x10: [mem 0x00000000-0x0003ffff 64bit]
[ 1867.969791] pci 0000:31:00.1: reg 0x18: [mem 0x00000000-0x003fffff 64bit]
[ 1867.969851] pci 0000:31:00.1: Max Payload Size set to 256 (was 128, max 1024)
[ 1867.970224] pci 0000:31:00.1: Adding to iommu group 22
[ 1867.970284] pcieport 0000:30:00.0: BAR 14: no space for [mem size 0x00a00000]
[ 1867.970291] pcieport 0000:30:00.0: BAR 14: failed to assign [mem size 0x00a00000]
[ 1867.970298] pci 0000:31:00.0: BAR 2: no space for [mem size 0x00400000 64bit]
[ 1867.970303] pci 0000:31:00.0: BAR 2: failed to assign [mem size 0x00400000 64bit]
[ 1867.970309] pci 0000:31:00.1: BAR 2: no space for [mem size 0x00400000 64bit]
[ 1867.970313] pci 0000:31:00.1: BAR 2: failed to assign [mem size 0x00400000 64bit]
[ 1867.970318] pci 0000:31:00.0: BAR 0: no space for [mem size 0x00040000 64bit]
[ 1867.970322] pci 0000:31:00.0: BAR 0: failed to assign [mem size 0x00040000 64bit]
[ 1867.970327] pci 0000:31:00.1: BAR 0: no space for [mem size 0x00040000 64bit]
[ 1867.970331] pci 0000:31:00.1: BAR 0: failed to assign [mem size 0x00040000 64bit]
[ 1867.970336] pcieport 0000:30:00.0: PCI bridge to [bus 31-39]
[ 1867.970342] pcieport 0000:30:00.0:   bridge window [io  0x6000-0x6fff]
[ 1867.970375] pcieport 0000:30:00.0:   bridge window [mem 0x38bf80000000-0x38bfffffffff 64bit pref]
[ 1867.970402] PCI: No. 2 try to assign unassigned res
[ 1867.970411] pcieport 0000:30:00.0: BAR 14: no space for [mem size 0x00a00000]
[ 1867.970416] pcieport 0000:30:00.0: BAR 14: failed to assign [mem size 0x00a00000]
[ 1867.970422] pci 0000:31:00.0: BAR 2: no space for [mem size 0x00400000 64bit]
[ 1867.970426] pci 0000:31:00.0: BAR 2: failed to assign [mem size 0x00400000 64bit]
[ 1867.970431] pci 0000:31:00.1: BAR 2: no space for [mem size 0x00400000 64bit]
[ 1867.970435] pci 0000:31:00.1: BAR 2: failed to assign [mem size 0x00400000 64bit]
[ 1867.970440] pci 0000:31:00.0: BAR 0: no space for [mem size 0x00040000 64bit]
[ 1867.970444] pci 0000:31:00.0: BAR 0: failed to assign [mem size 0x00040000 64bit]
[ 1867.970449] pci 0000:31:00.1: BAR 0: no space for [mem size 0x00040000 64bit]
[ 1867.970453] pci 0000:31:00.1: BAR 0: failed to assign [mem size 0x00040000 64bit]
[ 1867.970458] pcieport 0000:30:00.0: PCI bridge to [bus 31-39]
[ 1867.970463] pcieport 0000:30:00.0:   bridge window [io  0x6000-0x6fff]
[ 1867.970495] pcieport 0000:30:00.0:   bridge window [mem 0x38bf80000000-0x38bfffffffff 64bit pref]

I'm not sure why the pcieport has BAR 14 since doesn't seem a valid BAR.

Rebooted with the bitstream loaded to let the BIOS re-enumerate. The BARs are now assigned:

linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> dump_info/dump_pci_info_pciutils 
domain=0000 bus=31 dev=00 func=01 rev=00
  vendor_id=10ee (Xilinx Corporation) device_id=913f (Device 913f) subvendor_id=10ee subdevice_id=0007
  iommu_group=81
  driver=vfio-pci
  physical_slot=2-2
  control: I/O- Mem- BusMaster- ParErr+ SERR+ DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=97c40000 size=40000 is_IO=0 is_prefetchable=0 is_64=1
  bar[2] base_addr=97800000 size=400000 is_IO=0 is_prefetchable=0 is_64=1
  Capabilities: [40] Power Management
  Capabilities: [60] MSI-X
  Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 8 GT/s Max width x16
    Negotiated link status: Current speed 8 GT/s Width x16
    Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
    DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
    DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
    LnkCap: Port # 0 ASPM not supported
            L0s Exit Latency More than 4 μs
            L1 Exit Latency More than 64 μs
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
    LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  domain=0000 bus=30 dev=00 func=00 rev=04
    vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
    iommu_group=51
    driver=pcieport
    physical_slot=2
    control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
    status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
    Capabilities: [40] Bridge subsystem vendor/device ID
    Capabilities: [60] Message Signaled Interrupts
    Capabilities: [90] PCI Express v2 Root Port, MSI 0
      Link capabilities: Max speed 8 GT/s Max width x16
      Negotiated link status: Current speed 8 GT/s Width x16
      Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
      DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
              ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
      DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
              RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
      DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
      LnkCap: Port # 5 ASPM not supported
              L0s Exit Latency 256 ns to less than 512 ns
              L1 Exit Latency 8 μs to less than 16 μs
              ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
      LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
              ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
      LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
              Slot #2 PowerLimit 0.000W Interlock- NoCompl-
    Capabilities: [e0] Power Management

domain=0000 bus=31 dev=00 func=00 rev=00
  vendor_id=10ee (Xilinx Corporation) device_id=903f (Device 903f) subvendor_id=10ee subdevice_id=0007
  iommu_group=81
  driver=vfio-pci
  physical_slot=2-2
  control: I/O- Mem- BusMaster- ParErr+ SERR+ DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=97c00000 size=40000 is_IO=0 is_prefetchable=0 is_64=1
  bar[2] base_addr=97400000 size=400000 is_IO=0 is_prefetchable=0 is_64=1
  Capabilities: [40] Power Management
  Capabilities: [60] MSI-X
  Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 8 GT/s Max width x16
    Negotiated link status: Current speed 8 GT/s Width x16
    Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
    DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
    DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
    LnkCap: Port # 0 ASPM not supported
            L0s Exit Latency More than 4 μs
            L1 Exit Latency More than 64 μs
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
    LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  domain=0000 bus=30 dev=00 func=00 rev=04
    vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
    iommu_group=51
    driver=pcieport
    physical_slot=2
    control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
    status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
    Capabilities: [40] Bridge subsystem vendor/device ID
    Capabilities: [60] Message Signaled Interrupts
    Capabilities: [90] PCI Express v2 Root Port, MSI 0
      Link capabilities: Max speed 8 GT/s Max width x16
      Negotiated link status: Current speed 8 GT/s Width x16
      Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
      DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
              ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
      DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
              RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
      DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
      LnkCap: Port # 5 ASPM not supported
              L0s Exit Latency 256 ns to less than 512 ns
              L1 Exit Latency 8 μs to less than 16 μs
              ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
      LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
              ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
      LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
              Slot #2 PowerLimit 0.000W Interlock- NoCompl-
    Capabilities: [e0] Power Management

4. CMAC configuration

Looking at the CMAC configuration in the open-nic-shell project, for what to copy in other projects:

Setting QSFP0 (QUAD231) QSFP1 (QUAD230)
CMAC Core Selection CMACE4 Y0Y6 CMACE4 Y0Y7
GT Group Selection X1Y48~X1Y51 X1Y44~X1Y47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment