QoS for Virtual Machines in a 25GbE NIC and Open Switch Environment

High VM-density driving an aggregation of I/O

Cloud service providers and large enterprises operate server farms with ultra-high VM-density.  According to a CSP survey by Infonetics, the average number of VMs per server was 42 in 2015, growing to 98 in 2017. New, powerful server processors from Intel support even more VMs and IO.

QoS a must with 25GbE connectivity to a virtualized server

CSPs and enterprise server architects discovered high VM-density leads to an aggregation of I/O, I/O bottlenecks, and that non-critical apps can be noisy neighbors which disrupt traffic to critical apps. Deploying 25, 50 or 100GbE NICs provides the bandwidth needed to eliminate an overall I/O bottleneck, but a QoS policy is necessary to partition the bandwidth to different VMs to ensure performance is guaranteed to specific business critical VMS and applications.

Deploying QoS in a 25GbE Environment

The new generation of high-bandwidth 25GbE NICs with extensive HW offloads has led to a new best practice of deploying NIC-based partitions to enable QoS, then shaping virtual networks by making use of granular controls for ports and resource pools can further increase application performance and availability. The result is maximum use of HW offloads for more efficient use of server CPU, and guaranteed bandwidth to critical applications.

Architecture for Layering SW-based Services on HW-based Virtual Networking Services

QoS in a 25GbE Environment Reference Architecture

This high-density virtualized server environment was constructed in a 2550100 Solutions Lab and included 72 virtual machines per server. The 25GbE NICS in each physical server were partitioned using QLogic NPAR hardware partitioning. Each physical NIC was partitioned into 8 vNICs. One VM which represented a critical business app was allocated a 10GbE port, while 7 groups of VMs shared the remaining vNICs.  Load balancing and teaming was layered on the hardware-based portioned using VMware ESX Network I/O Control (NIOC). Listed below are the components used in the test configuration.

Products Used

The products listed below were used in the reference architecture configured in a 2550100 Solutions Lab.

Qty	Product	Model	Description
4	Operating System	Windows Server 2012	Windows Server 2012 R2 is an operating platform which can run the largest workloads with support for up to 64 processors VHDX virtual hard disks up to 64 terabytes .
4	Hypervisor	VMware vSphere 6.0	VMware vSphere v6 features support for per-VM Distributed vSwitch bandwidth reservations to guarantee isolation and enforce limits on bandwidth.  V6 also allows vMotion traffic a dedicated networking stack.
4	Servers	Dell PowerEdge 630	PowerEdge R630 support the latest Intel Xeon E5-2600 v4 processors; up to 22 cores; up to 24 DIMMs high-capacity DDR4 memory; Up to 24 1.8” SSDs (23TB); up to 3 PCIe 3.0 expansion slots; and up to 4 Express Flash NVMe PCIe SSDs.
1	Storage	Kaminario All Flash Array	The K2 array is comprised of K-blocks — building blocks that include Active-Active controllers, one or more drive shelves, and connectivity for scaling out. The K2 platform scales-out to two, three, and four K-block configurations.  Even when scaling out the cluster remains fully N-ways Active-Active.
2	Switches	Supermicro SSE-C3632	Offering thirty-two QSFP28 Ethernet ports, SSE-C3632S switch enables robust layer-3 IP fabric for flexible layer-2 overlay in Ethernet fabric architecture.
2	Switch OS	Cumulus Linux	Cumulus Linux is a networking-focused Linux distribution. Switches running Cumulus Linux provide standard networking functions such as bridging, routing, VLANs, MLAGs, IPv4/IPv6, OSPF/BGP, access control, VRF, and VxLAN …
4	Adapters	QLogic QL45000 Series	The QL4521x Series supports speeds of 25Gbps and 10Gbps. FastLinQ 45000 Series Controllers enable SR-IOV, RoCE, iSCSI, FCoE, and DCB. They also support PCIe Gen 3.0, along with embedded virtual bridging and other switching technologies for high-performance DMA and VM-to-VM switching.
20	Optical Transceivers	Finisar 100GBASE-SR4 QSFP28	Finisar’s FTLC9551REPM 100G QSFP28 transceiver modules are designed for use in 100 Gigabit Ethernet links over multimode fiber.
1	Traffic Generator	Xcellon-Multis QSFP28 Enhanced Load Module	Xcellon-Multis provides the world’s first 100/50/25GbE multi-rate test system to satisfy equipment maker test needs ranging from basic interoperability and functional test, to high-port count performance tests. As organizations implement this same high-density, high bandwidth networking and network computing equipment in their own networks, they need this same test solution to verify performance and functionality prior to deployment.