Yesterday I wrote that you don’t need DCB technologies to implement FCoE in your network. The FC-BB-5 standard is quite explicit (it also says that 802.1Qbb is the other option):
Lossless Ethernet may be implemented through the use of some Ethernet extensions. A possible Ethernet extension to implement Lossless Ethernet is the PAUSE mechanism defined in IEEE 802.3-2008.
The PAUSE mechanism (802.3x) gives you lossless behavior, but results in undesired side effects when you run LAN and SAN traffic across a converged Ethernet infrastructure.
Traffic blocking with the PAUSE mechanism
The PAUSE mechanism is part of the Ethernet (802.3) standard and allows a receiver on a point-to-point Ethernet link to stop the adjacent sender thereby preventing a buffer overflow and packet loss.
Imagine a simple FCoE network with a server, a storage array and a switch, with server sending large amounts of data to the storage array.
When the server overloads the storage array with the data it’s sending, the storage array sends a PAUSE frame back to the switch.
The switch stops sending data to the storage array after receiving the PAUSE frame and data sent by the server start to accumulate in switch’s internal buffers until the switch has to tell the server to pause.
At that moment, the server’s Ethernet interface is effectively blocked, which is not a problem if you have a dedicated FCoE infrastructure. The same result is unacceptable in a converged infrastructure, where FCoE and LAN traffic share the same links.
Traffic blocking with Priority Flow Control (802.1Qbb)
802.1Qbb is a simple extension of the 802.3x mechanism: the PAUSE frame contains a 8-bit bit mask of 802.1p priorities (specifying which traffic classes should be paused) and a timer for each priority specifying how long the traffic in that priority class should be paused. The per-priority PAUSE mechanism allows the storage array to tell the switch it should stop sending just the FCoE traffic (assuming FCoE traffic is marked with priority value=3).
Likewise, the switch can tell the server to stop sending FCoE traffic and the LAN traffic is not impacted.
It’s also possible (at least in theory) to combine 802.3x and 802.1Qbb. For example, the storage array could use the 802.3x PAUSE mechanism to slow down the switch, whereas the switch (after noticing its priority-3 queues are filling up) could use 802.1Qbb PAUSE frame to tell the server to stop sending FCoE traffic.
More details
The PFC mechanism can quickly result in head-of-line blocking and extended congestion and is thus applicable only to small bridged domain. It should be combined with congestion notification/avoidance mechanisms (for example, 802.1Qau) in larger domains.
PFC was designed for use on point-to-point links (it does not work in PON environments) and cannot be used together with 802.3x on the same link (two competing PAUSE mechanisms on the same link make no sense). It needs DCBX standard to negotiate the parameters between adjacent nodes, including the number of traffic classes that can support PFC and the priorities for which PFC should be enabled. A standard-compliant implementation of 802.1Qbb thus requires support for DCBX as well.
The timings are quite strict (sender should stop sending in ~ 600 nanoseconds), making a hardware implementation the only viable option.
Pre-standard implementations (speculative)
As the 802.1Qbb addendum hasn’t been ratified yet, all current PFC implementations are by definition pre-standard. However, the format of the PAUSE message hasn’t changed from the very early drafts, indicating that the existing hardware implementations will probably need just a software upgrade to support potential late changes to the DCBX protocol.
Need more?
You’ll get an overview of DCB, FCoE and numerous other Data Center technologies in my Data Center 3.0 for Networking Engineers webinar (register here).
The 802.1bb page has links to numerous presentations.
Priority Flow Control: Build Reliable Layer 2 Infrastructure white paper from Cisco has great in-depth description of 802.1Qbb and planning recommendations for Nexus 5000 and Cisco’s CNA.
