Notes about 100G Ethernet switches for a home lab, looking for either:
- Cheap low port count new switches.
- Available second hand switches with possibly more ports than required.
- Options for mixing some low speed ports, as have some other 10G devices.
Found on Reddit "Cheap" 100G switches, pros and cons of each option? for initial suggestions.
Manufacturers product page says:
There are four 100 Gigabit QSFP28 ports. Use them to connect other 100 Gigabit devices or grab a breakout cable to connect multiple 25 Gigabit devices: each port is connected to four 25 Gigabit lines, allowing you to connect 16 x 25 Gigabit devices. In fact, you can mix and match all kinds of connections this way: 100 Gigabits on one port, 40 Gigabits on the other, 4x25 Gigabits on the third, and maybe even some 10G or 1G cables on the fourth. The mighty Marvell DX4310 switch-chip will handle everything without breaking a sweat!
New price on Amazon - Mikrotik CRS504-4XQ-IN QSFP28 M RM £627.35
MikroTik CRS504-4XQ-IN Review Momentus 4x 100GbE and 25GbE Desktop Switch contains:
The other big one is a huge missing feature. With 100GbE, 200GbE, 400GbE, and so forth networks, RDMA has become a mainstream feature and RoCE (V1/V2) have become extremely popular. This is largely due to the amount of data that has to move through the network stack. Bypassing that network stack via direct memory to NIC transactions saves an enormous amount of CPU cycles. We could not figure out how to get features like PFC working on this switch. We went to the MikroTik help docs and searched for RDMA, and we got absolutely nothing.
Second hand on ebay “Very Clean, Fully Tested Arista DCS-7160-48YC6-R Switch Rear-to-Front Airflow. Arista 7160, High Capacity 48 x 25GbE SFP and 6 x 100GbE QSFP switch, rear to front air, 2 x AC” £849.00
Manufacturers 7160 Series Data Center Switch datasheet has the overview:
7160-48YC6: 48 port 25G SFP and 6 port 100G QSFP
- Offer 48 wirespeed 10/25G ports with 6 100G ports for up to 72 total 25G or 10G ports
- Easy migration from 1/10G to 1/10/25G using a familiar SFP connection
- Six 100G QSFP ports allow choice of 6x 100GbE, 24x 25GbE, 6x 40GbE, or 24x 10GbE
- 5 speeds for flexible 10GbE, 25GbE, 40GbE, 50GbE and 100GbE with optics or cables
- 3.6Tbps of wire speed performance with 24MB of buffer
Second hand on ebay Dell Z9100-ON , 32 Port 100Gb QSFP28 2x 10Gb SFP+ Switch, Tested, Warranty £495.00
Output shows running Dell EMC Application Software Version: 9.14(1.14). Possible license issues with trying to update to OS10:
- Looking for a cheap 100G network switch
- Dell Z9100-ON
To use OS10 you will need a license, and in many cases the license will be lost when buying a second-hand device, so you can only get an "indefinite" trial system (cannot be used after a certain period of time, needs to be reinstalled to "trial" again)
Dell datasheet says:
The compact Z9100-ON design provides industry-leading density of either 32 ports of 100GbE, 64 ports of 50GbE, 32 ports of 40GbE, 128 ports of 25GbE or 128 ports 10GbE and two SFP+ ports of 10GbE/1GbE/100MbE to conserve rack space while enabling denser footprints and simplifying migration to 100Gbps in the data center core. Priority-based flow control (PFC), data center bridge exchange (DCBX) and enhanced transmission selection (ETS) make the Z9100-ON ideally suited for DCB environments.
1RU high-density 10/25/40/50/100GbE fixed switch with choice of up to 32 ports of 100GbE (QSFP28), 64 ports of 50GbE (QSFP+), 32 ports of 40GbE (QSFP+), 128 ports of 25GbE (SFP28) or 128+2 ports of 10GbE (using breakout cable)
Some configurartion documentation - Dell EMC Networking OS Configuration Guide for the Z9100–ON System 9.14.2.4:
Splitting 100G Ports The platform supports splitting a single 100G QSFP 28 port into any of the following ports:
- Two 50G ports
- Four 25G ports
- One 40G port
- Four 10G ports
FEC Configuration FEC configurations are available on 100–Gigbit, 50–Gigabit and 25–Gigabit Ethernet interfaces. To configure FEC, use the following commands.
Enable FEC.
INTERFACE mode fec enableThe CL91 keyword enables RS-FEC on 100G interfaces as per IEEE. The system displays an error message when this is applied on 25G or 50G interfaces.
The CL74 keyword enables BaseR-FEC on 25G or 50G interfaces as per 25G/50G Ethernet Consortium. The system displays an error message when this is applied on 100G interfaces.
To disable FEC, use the
no fec enablecommand.
From Need advice on complete homelab refresh:
- I run a Celestica Seastone DX010 since 2021 and I've made a few posts and Github issues about it. It is fiddly and doesn't exactly "just work", especially if you're making frequent changes. I used to own a Dell Z9100 (sold it here) which "just worked" but it was picky about transceivers and wasn't nearly as flexible. If you do pick up a DX010 I will help you. Using RoCE/SMB Direct over 100G has been a game changer for the way I access storage.
Product page Half-width Rackmount 100GbE L3 Lite Managed Switch, cost-optimized for enterprise IT:
The QSW-M7308R-4X is a 100GbE L3 Lite Managed Switch with four 100GbE QSFP28 and eight 25GbE SFP28 ports.
The 100GbE ports support breakout into 4x 25GbE links.
The datasheet says the supported standards includes IEEE 802.3ae 10G Fiber, but isn't clear if the supports running at 10 GbE.
QNAP QSW-M7308R-4X 8x 25GbE and 4x 100GbE Managed Switch Review says uses a Marvell Prestera 98DX7324 switch chip. The 98DX73xx product brief contains:
Multi-Rate 1/2.5/5/10/25/40/50/100/200/400G Ethernet ports
Amazon QSW-M7308R-4X Ultra-high-speed managed switch with four 100GbE QSFP28 and eight 25GbE SFP28 ports £1,341.47