Peering into RapidIO's Move to the Data Plane

Peering into RapidIO's Move to the Data Plane
By Chuck Hill, Motorola Computer Group, CommsDesign.com
November 11, 2003 (12:41 p.m. EST)
URL: http://www.eetimes.com/story/OEG20031111S0015

For the past year, there has been an expressed interest in the communication design community to redefine the interconnect architectures used in today's networking and telecom equipment, sparking designers to eye newer interconnects like RapidIO, PCI-Express, and HyperTransport. But to be successful, these interconnects must effectively work in both control plane and data plane scenarios.

Recognizing this concern, the RapidIO Trade Association has added flow control, data streaming, and traffic management capabilities to its base specification in order to make RapidIO a better option for handling data plane traffic. Below, we'll look at these enhancements and show how they will promote the usage of RIO as a system-wide I/O for next-gen networking and telecom architectures.

Brief RapidIO Overview
The RapidIO architecture has its roots as a technology for connecting processing devices together with a distributed memory mapped architectu re. A message passing programming model and a globally shared distributed memory programming model enable both general purpose multiprocessing and distributed I/O processing to co-exist in the same interconnect.

RapidIO supports a variety of data sizes for efficient operation of message passing and memory-mapped devices. A directory based coherency mechanism is defined to provide processing elements with all of the necessary support to provide for hardware maintained coherent memory (Table 1)

The RapidIO interconnect is specified in a three-layer architectural hierarchy which includes a logical layer, a physical layer, and a transport layer (Figure 1).

The logical layer specification defines the overall protocol and packet formats. This is the information necessary for end points to initiate and complete a transaction. The logical layer defines the presentation to the application.

The common transport specification provides the necessary routing information for a packet to move from end point to end point. The common transport unifies all the traffic so any RapidIO switch can carry traffic from all logical and physical layers.

The physical layer specifications contain the device level interface, such as the packet transport mechanisms, flow control, electrical characteristics, and low-level error management. Physical specifications include 8- and 16-bit LVDS parallel interfaces for low latency as well as 1X/4X multi-gigabit serial interfaces for low pin counts and longer transmission distances.

The RapidIO interconnect architecture is also extensible at both the logical and physical layers. For example, the trade assoc iation has recently completed a specification that defines end-to-end flow control enhancements to improve congestion management. The association is also currently defining data streaming logical layer will define a logical transaction layer optimized for the variable sized payloads typical in networking workloads. Let's look at both of these extensions in more detail starting with the flow control mechanism.

Controlling Flows
All RapidIO physical layers have a link-based flow-control mechanism for flow management between two devices. This ensures that individual link receivers will not be overrun with data, but doesn't address the situation where multiple traffic sources might individually or collectively create congestion scenarios two or more switches removed from the traffic sources. The recently released flow-control specification defines a mechanism to deal with these scenarios and provide end-to-end flow control within a system.

End-to-end flow control allows systems to manage co ngestion. Congestion can occur naturally in larger diameter fabrics. The statistical nature of traffic can cause short term over utilization of resources. Without the ability to regulate traffic at the source, this can lead to a cascading congestion effect, lowering overall fabric performance.

As fabric utilization exceeds 60%, congestion can significantly affect traffic latency. For applications where overprovisioning of bandwidth is the method used to insure performance, the recently completed flow-control extensions can expand the effectiveness of the available bandwidth and reduce latency. Another feature provided by the flow control extensions is the ability to managing traffic flow back to the originating endpoints. This is essential for fault management. Without a means to shut down traffic sources in a system, a single failed endpoint can bring an entire fabric and system down by spewing traffic into the system.

To review, the two flow control mechanisms in RapidIO are each optimized for sol ving different but related traffic congestion problems in a system. Link-based flow control is designed to prevent traffic overruns between two directly connected devices in a system. An example of this would be a situation where a receiving device has run out of buffer space to place incoming data. Upon detecting this situation the receiver can send a message to the sender to either stop or slow its rate of transmission.

End-to-end flow control is used when multiple devices have created a traffic congestion scenario downstream in a system. This congestion may be severe enough to cause unrelated traffic flows in the system to be adversely affected as well. Unlike the link-based flow control mechanism, for which signaling is confined to between two adjacent devices, the end-to-end flow control can span any number of devices in the system. It also offers flow-control management based on flow priorities, allowing higher priority traffic to continue to flow through the system while lower priority tra ffic is throttled.

Data Streaming at the Logical Layer
Network traffic is different in character than that of most computing applications. Network traffic usually consists of a stream of data, consisting of multiple or even continuous transactions unique to specific destinations. These "flows" can range from very small to very large data grams. Network traffic today uses may different protocols. It is important for today's telecom equipment to be protocol agnostic.

Telecom traffic adds an additional requirement to manage quality of service (QoS). Traffic behavior can range from requiring highly deterministic delivery of all traffic classes to solely requiring best effort. Traffic regulation may be required on a per flow basis. This gives network service providers the ability to regulate and charge for services. This finely grained traffic classification model is an important characteristic that distinguishes the data streaming logical layer from the other logical layers.

Through the n ew data streaming specifications, network traffic, upon entering a managed system, is classified based on specific fields in the protocol. This application specific process creates a Stream ID that when combined with a class of service field and the source and destination IDs, forms a unique key. This virtual Stream ID can be subdivided to support virtual queuing at the "on ramp" to the fabric. Further use of the class field by the transport layer assists the fabric in prioritizing traffic (Figure 2).

At the destination the full key is used to restore the data to an individual user queue. The RapidIO streaming logical layer supports millions of individual flows per source/destination pair. The use of virtual Stream IDs eliminates the need for protocol specific reclassification upon arrival at the destination.

To support the quality of service needs for traffic requiring deterministic latency, the data streaming spec allows the RapidIO interconnect to transport traffic in fixed units. This gives the system the ability to interleave important traffic at fixed intervals, even if larger, less important streams have already been launched. With variable incoming transaction sizes, the data streaming logical layer segments and reassembles (SARs) the traffic. By including the SAR protocol in the logical layer, fabric endpoint interoperability is assured.

Advanced Traffic Management
The data streaming logical layer includes additional features for advanced traffic management (Figure 3). These additions provide a higher level of congestion avoidance. Support for finely grained queuing and scheduling allows tighter coupling between distributed traffic management elements and a higher level of QoS. Fabric resources can be regulated and even coupled to service level contracts.

Within systems, traffic management can be implemented with an appropriate amount of functionality for the task at hand. For local connectivity and relatively low bandwidth utilization applications basic flow control mechanism can provide an adequate level of support to protect the fabric from data loss and link congestion. For larger systems or systems where link utilizations are high, the new flow control extensions provide mechanisms to regulate traffic and effectively increase system bandwidth and reduce overall transaction latency. For systems where, performance guarantees are required for the various classes of traffic moving through the fabric, class-based queuing mechanisms can be implemented.

These advanced traffic management mechanisms provide for the fine grained management of individual traffic classes within the system. Class-based queuing improves the fabric's ability to provide more deterministic latency for special traffic classes. For example voice traffic may be prioritized over email traffic as it traverses the fabric backplane.

Advanced traffic management provides a method for separate traffic managers within the system to act as a single unit. When ingress-egress pairs collaborate on what traffic transitions the fabric, latency is minimized, and bandwidth is maximized.

This hierarchy of flow-control mechanisms defined within the RapidIO interconnect architecture (link, end-to-end, advanced traffic management) enables a range of fabric implementations to be developed in a compatible RapidIO architecture framework. Simple fabric implementations with minimal QoS features can be built and be compatible with more sophisticated fabric components which offer higher QoS capabilities for applications that require them. The use of RapidIO throughout the system provides a common electrical, link and base protocol layer shared by all devices, which allows base level communication throughout the system and also allows individual devices and links to optimize their communication bandwidth and latency through more sophisticated traffic management mechanisms,

System-Wide I/O
RapidIO can be interfaced to a variety of different traffic management devices, or bridged to other computing environments. The semantics of the data streaming logical layer are consistent with the Network Processor Forum Common Switch Interchange (CSIX) and Network Processor Streaming Interface (NPSI) specifications. This enables endpoints with minimal silicon footprints for specialized traffic management devices.

The rich set of features and the high level of interoperability standardized by the RapidIO specifications will encourage endpoints with native RapidIO interfaces. Thus, RapidIO can become the common denominator for a piece of communication equipment and thus simplify board desig n.

The common memory and messaging transactions defined within the RapidIO architecture are easily bridged between communications protocols, network processing constructs, and general purpose processing. This versatility provides for low overhead transaction interoperability between control and data plane applications. For example, the interworking of Ethernet and ATM can be managed very efficiently among a group of processors sharing memory or passing packets as messages across a RapidIO fabric.

Wrap Up
The RapidIO interconnect technology continues to expand its usefulness by adding features targeted at the telecom data plane. These features include support for streaming data types, flow control, advanced traffic management, and higher performance transports. By also supporting control-plane connectivity, RapidIO has become an excellent system-wide I/O choice for designers looking to evolve their interconnect architecture.

About the Author
Chuck Hill is a sys tem architect for Motorola Computer Group. He has a master's degree in engineering from Arizona State University and can be reached at charles.c.hill@motorola.com.