With StarFabric as an on-ramp, the PCI Express Advanced Switching is ready

With StarFabric as an on-ramp, the PCI Express Advanced Switching is ready
By Tim Miller, Vice President, Sales and Marketing, StarGen, Inc., Marlborough, Mass., President, StarFabric Trade Association, EE Times
January 27, 2003 (11:38 a.m. EST)
URL: http://www.eetimes.com/story/OEG20030124S0029

Last August, the StarFabric Trade Association (SFTA) and the PCI Express Advanced Switching (AS) Working Group announced plans to ensure interoperability between the StarFabric architecture and emerging PCI Express AS interconnect standards. Since that time, the architects of the StarFabric switched serial interconnect have been deeply involved in helping develop the PCI Express AS specification. Because of the high degree of alignment in capabilities and features, the SFTA has officially endorsed AS as the complementary high-speed roadmap for today's 2.5Gbit/second StarFabric.

Essentially, PCI Express AS will take advantage of the physical and link layer of PCI Express Base. The communication and embedded computing enhancements are added at the transaction layer.

The basis for the high level of commonality between StarFabric and AS is the shared vision of the original architects of what the next-generation interconnect required. Both groups knew that shared bus-based architectures were reaching their upper limits of usefulness. In order to reach new levels of performance and scalability, a new serial switched-based architecture was required.

Both groups appreciated the importance of the huge installed base of PCI hardware and software. It was strongly belie ved that a new architecture could not abandon this base and still become a ubiquitous standard. An evolutionary approach, which protected the investment by hundreds of companies in PCI hardware and software, was required.

Both groups also understood the unique requirements of distributed processing in communications, storage, and embedded applications. Thus, they created advanced features addressing high-availability, multiple types and classes of traffic, and flexible scalability. The belief was that the next-generation ubiquitous interconnect standard technology must be an open standard, which would create a vibrant ecosystem of the widest range of participants. This would stimulate innovation and provide system designers with the widest range of low-cost components.

For efficient support of distributed multiprocessing applications, both StarFabric and AS support path routing. Path routing eliminates the need for every switch to have look-up tables and specific software to route pack ets. The source of the packet specifies the path that the packet will travel through the fabric to the destination. Path routing is also reversible. When a destination node receives a packet it can easily compute the return path to the source. This enables switches to notify the source of any failures without software intervention. The hardware-based rapid response greatly increases system robustness.

Path routing also supports protection. Since a node can uniquely determine the origin of every packet, it can accept or refuse access based on the originating node.Multicast capability is also supported, which enables a single source packet to be sent to many destination nodes.

Protocol- agnostic

Both architectures are protocol-agnostic. Essentially, AS is an encapsulation architecture where each encapsulated protocol is given a unique Protocol Encapsulation Interface (PEI) number. There are total of 255 PEIs. PEI 0 through 7 are used for fabric managem ent. The PCI-SIG will define PEIs 8 through 223 based on standard protocols. PEIs 224 through 254 can be defined by end-users.

Additionally, both StarFabric and AS support multiple classes of service. StarFabric defines eight types of traffic: provisioning, high-priority isochronous, isochronous, high-priority asynchronous, asynchronous, multicast, address-routed, and special. AS defines eight classes of services as well. AS provides an additional degree of freedom where each TC can be mapped to a specific virtual channel (VC) at run time. For example, if a device only supported 4 VCs, the system designer can map the 8 TCs into the 4 VCs in a manner that fine-tunes the system traffic.

Each of the classes of service noted above has its own credit pool in each device. Congestion in one class will not affect other traffic. For example, a burst in asynchronous data will not affect a stream of isochronous data. Additionally, unused bandwidth of one class can be utilized by other classes .

Credit-based flow control is used in both architectures. This technique guarantees progress through the fabric. A frame or packet is only sent if there is sufficient buffer space in the next node. Frames are never dropped due to congestion, and re-transmission is only required when a failure has occurred.

Both StarFabric and AS are robust fabrics and are considered loss-less in the absence of failures. Each packet is CRC protected and the CRC is checked at every link. Failures are detected in hardware and are reported and managed. Errors are explicitly routed to the fabric management node. Recovery from errors can be accomplished in both hardware and software.

The architectures were designed from the onset with high-availability (HA) in mind. While HA in general is a multi-layer issue, these interconnect technologies have a number of hardware-bas ed HA features and additional primitives for the upper layers to use.

The use of redundant links with hardware recovery is one of the essential HA silicon-based features. Designers can architect a system with parallel redundant fabrics and each end node would have an interface to both the primary and the secondary fabric. In the event that the primary link fails, the silicon would automatically switch to the backup path.

StarFabric and AS both offer advanced feature support that is accessible with the addition of some software to provide basic fabric set-up, operation, maintenance, and event handling. For these features, StarFabric and AS can utilize the same low-level load-store programming model. The StarFabric software model is a layered software approach that can hide the hardware implementation details from higher-level software. It contains bus drivers, fabric-primitives libraries, fabric-distributed libraries, and fabric management layers.

The fabric driver is respo nsible for the run-time fabric control and management policies. It interacts with architecture-defined capabilities of each node. It performs fabric discovery at initialization where each node is identified, and the fabric topology is determined. It is responsible for connection creation and tear down, provisioning bandwidth, resource allocation, and synchronization. It also handles fabric events, such as hot plug, power management, and fault management.

The Fabric Primitives Library (FPL) is a highly-portable operating system-independent layer. It provides a consistent API for low-level fabric routines and hides the register level complexities from the system designer. The FPL is typically linked with the fabric driver.

The device driver controls the implementation-specific capabilities of each node and prepares the node for run-time operations. It translates local interrupts into fabric events.

Common traits

Because the software model and the functionality supported by StarFabric and AS are similar, there is a high-degree of commonality built in. Application level software and some of the high-level fabric modules can be transportable between StarFabric and AS. Investment in this level of software on StarFabric can easily migrate to AS when available.

While the two architectures are highly aligned there is one major difference and that is the speed and cost of the physical layer. StarFabric links are based on 622 Mbit/second Low Voltage Differential Signaling (LVDS) base technology. Each link uses 4 pairs to make a single 2.5Gbit/sec link. This technology is very low-cost and easy to implement at the system level. Standard connectors can be used within a chassis, and inexpensive CAT5 cable can be used in chassis-to-chassis applications. PCI Express uses 2.5Gbit/sec signaling per differential pair with a 4X link being 10Gbit/sec or four times faster than StarFabric. PCI Express has also specified even higher speed links of up to 32 differential pairs. At these speeds, the connectors and cables required are naturally more expensive.

Communication equipment vendors are seeking a long-term solution to their system interconnect needs. They are demanding broad-based industry standards with the scalability, HA, and QoS attributes they require.

StarFabric is suited for 2.5Gbit/sec and below applications. AS will provide similar capabilities as StarFabric but at higher performance. It will be available starting in 2004 and is suited for 10Gbit/sec and above applications