PCI Express Bifurcation in the 100G Era

By: Sven Olav Lund, Sr. Product Manager, Napatech

Over the last few years, the increase in internet bandwidth has caused carriers to invest in equipment that can meet the requirements of the increasing network traffic. The rapid growth in 100G network speeds has led to the deployment of new high-volume network equipment. The first phase involves upgrading or replacing the switch and router equipment. This naturally leads to a need for network monitoring and subsequent investments in 100G network monitoring equipment.

As a result of all the new or upgraded equipment being introduced, there arises a new challenge when it comes to rack space, power consumption and cooling. The combination of a Commercial Off the Shelf server (COTS) and a 100G network adapter is a commonly used approach for building a 100G network monitoring solution. A small 1U COTS server is the best fit to meet the challenges of limited rack space in data centers or telco central offices. The 100G network adapter must comply with half-length PCI form factor in order to fit into a compact 1U COTS server.

However, there are some challenges in using compact PCIe Gen3-based 100G solutions.

Challenges of Compact PCIe Gen3-based 100G Monitoring Solutions

In order to capture the full 100G traffic and transfer it to the server host memory, a 16-lane PCIe Gen3 throughput is required. The FPGA technologies available today do not support native 16-lane PCIe Gen 3, but support a 2 x 8-lane PCIe Gen3 interface. Supporting a native 16-lane PCIe Gen 3 requires a PCIe switch component mounted on the network adapter. Furthermore, gigabytes of on-board memory are also needed to ensure packet delivery in case the server system gets overloaded at peak conditions. After installing the major components (FPGA, SDRAM, network ports and power supplies) on a half-length PCIe form factor, there is usually no space for a PCIe switch component, which can have certain consequences.

To enable the throughput of 16-lane PCIe Gen3, the COTS server must support PCIe bifurcation.

What is PCIe Bifurcation?

PCI Express bifurcation means splitting the PCI Express bus into smaller buses, and in this case splitting a 16-lane PCIe Gen 3 into a 2 x 8-lane PCIe Gen3 interface. Servers supporting PCIe bifurcation have a BIOS setting for enabling the feature. When a PCIe bifurcation is enabled, the data that would usually be transferred over a 16-lane PCIe Gen 3 is split into 2 parallel 8-lane PCIe Gen3 interfaces. The network adapter supporting the PCIe bifurcation enables the same data transfer scheme, using two x8 PCIe Gen3 interface.

Server Support for Bifurcation

One of the major benefits of building network monitoring solutions based on COTS servers is the freedom to freely select the server vendor. Not all server vendors support bifurcation today, but there is an increasing pressure from the industry due to the need for implementing compact solutions based on the most commonly available FPGA technology. HP and Supermicro have servers that support PCIe bifurcation today, and other major vendors will follow suit soon.

What’s next?

As FPGA technologies evolve, more and more features are getting integrated into the hardened part of the FPGA. Hardened 16-lane PCIe Gen3 will most likely be on the roadmaps from the leading FPGA vendors and available in the market in 2017-2018. Consequently, it will be possible to build compact NICs with full support for 16-lane PCIe Gen3 support, and the bifurcation issue will not be relevant any more.

When stepping up to PCIe Gen 4, we will probably see the same scenario again, followed by yet another FPGA roadmap to solve the problem.

About the author:

Sven Olav Lund is a Senior Product Manager at Napatech and has over 30 years of experience in the IT and Telecom industry. Prior to joining Napatech in 2006, Sven Olav was a Software Architect for home media gateway products at Triple Play Technologies. From 2002 to 2004 he worked a Software Architect for mobile phone platforms at Microcell / Flextronics ODM and later at Danish Wireless Design / Infineon AG. As a Software Engineer, Sven Olav started his career architecting and developing software for various gateway and router products at Intel and Case Technologies. He has an MSc degree in Electrical Engineering from the Danish Technical University.

 

Tags: 100G Ethernet, Napatech, PCI Gen-3

Category: News