Peripheral Component Interconnect Express (PCIe®) 6.0 is a crucial technology for AI data centers, which adds many advances in modulation and signaling but also complicates device testing.
Thanks to its high speeds, PCIe is a foundational technology underneath many key industries for nearly three decades — including the artificial intelligence (AI) ecosystem, high-performance computing (HPC), data centers, enterprise networking, storage devices, and more. According to the PCI-SIG standards committee, PCIe® 7.0 specification will double PCIe® 6.0 bandwidth up to 512GB/s bi-directionally on an x16 connection.
The ability to look under the hood of this ubiquitous technology is essential. Protocol analyzers are the instruments that provide such X-ray vision into PCIe. Protocol analyzers and exercisers work in tandem to enable deterministic testing of all PCIe® scenarios. Protocol analyzers must neither degrade nor improve signal integrity but dutifully maintain it as is to ensure accurate test results. This article will introduce the basics of PCIe protocol testing, what protocol analyzers can do, how they work, and how they overcome tough PCIe validation challenges.
What is a protocol analyzer?
Figure 1. Keysight P5570A PCIe 6.0 protocol analyzer in the background with an exerciser in the foreground, both fixed to a testing backplane at the bottom
In electronics testing and instrumentation, a protocol analyzer is a device that can capture, decode, analyze, and retransmit the data flowing between two or more systems at various protocol layers over some communication channel.
An ideal protocol analyzer exhibits full transparency. It does not introduce additional signal integrity errors due to its own circuits or cables, does not affect the functionality of any communicating system, has minimal latency, and has near-zero performance costs.
What protocols can be analyzed with a protocol analyzer?
The protocols for which analyzers are extensively used include:
- PCIe
- Compute Express Link (CXL)
- Universal Serial Bus (USB)
- Common interfacing protocols, like the Inter-Integrated Circuit (I2C and I3C) and Serial Peripheral Interface (SPI)
- Networking protocols, like Ethernet
- Specialized protocols, like the Controller Area Network (CAN) used in automobiles
How does a protocol analyzer work?
For protocol analysis, engineers can choose between dedicated hardware instruments, on-device software features of general-purpose instruments, or offline analysis software. Each has various pros and cons as outlined below.
Dedicated protocol analyzer and exerciser hardware
The most rigorous, in-depth testing and analysis are provided by dedicated protocol analyzer and exerciser hardware.
For example, in the above illustration, dedicated PCIe protocol analyzer hardware, like the Keysight P5570APCIe 6.0protocol analyzer, captures PCIe communication between a root complex and a peripheral device without introducing any interference or signal integrity errors. It sends the captured data in real time to a separate computer for analysis and visualization.
Either the root complex or the peripheral, or both, can be rigorously and accurately emulated using dedicated protocol exerciser hardware like the Keysight P5573A PCIe 6.0 protocol exerciser.
This combination of dedicated hardware instruments enables comprehensive functional and performance testing of all scenarios and edge cases.
Protocol decoding capabilities of general-purpose instruments
Protocol decoding capabilities are also available on general-purpose instruments like oscilloscopes. However, while very convenient for quick functional checks, they may not have the in-depth and comprehensive analysis capabilities of dedicated hardware.
For example, Keysight InfiniiVision oscilloscopes can capture in-vehicle network protocols, and an on-device software component decodes those captured bits into protocol-specific commands and data shown directly on the instrument displays. Keysight Infiniium UXR-series oscilloscopes have similar protocol decode software for PCIe and other high-speed digital standards.
Why are protocol analyzers critical for PCIe device manufacturers?
The PCI Special Interest Group (PCI-SIG), responsible for the PCIe specifications, also runs a compliance program to certify PCIe devices and systems. This official label assures other device manufacturers and system integrators that a product has undergone rigorous testing to verify compliance with the PCIe specifications and interoperability with other devices.
Two key areas of compliance testing are:
- link protocol testing to verify link-level behavior
- transaction protocol testing to qualify transaction-level behavior
PCIe protocol analyzers and exercisers are critical for all these compliance tests.
How are PCIe protocol analyzers crucial for AI data centers?
PCIe protocol analyzers are invaluable for the interoperability and compliance testing of processors, server motherboards, and high-performance peripherals like server-grade GPUs, gigabit ethernet cards, InfiniBand network cards, solid state drives, and redundant arrays of independent disks (RAID) storage.
Ethernet links in data centers are reaching speeds of 800 gigabits per second and are on the verge of doubling soon. PCIe 6.0 and the upcoming 7.0 are critical for facilitating such high speeds between CPUs and peripherals on data center servers. By providing detailed diagnostics and verification, PCIe protocol analyzers help maintain the high-speed, reliable connections that modern AI workloads demand.
What are some challenges in PCIe protocol testing?
Testing PCIe protocols in next-generation AI data centers presents three critical challenges.
Capture high data rates without data loss
The high data rates of PCIe 6.0 (up to 256 gigabytes per second each way) require high-speed instruments that can capture the data without loss.
Maintain signal integrity without interference
High data rates result in reflections and crosstalk. Preventing such signal degradation and channel impairments is crucial. However, the analyzer must work as if it’s not present in the test setup at all. It must not inadvertently improve signal degradation or correct timing errors.
Visualize complex protocol transactions
Each generation of the PCIe specification introduces more complex transactions. Sophisticated decoding and analysis capabilities are essential to keep up.
What key PCIe concepts should you know to use protocol analyzers effectively?
Knowing how PCIe works helps your engineers and testers use protocol analyzers more effectively. We outline some key concepts and advances below.
1. Protocol layers and data packets
The PCIe initialization and operation steps are organized under three communication protocol layers:
- Physical layer: This is the lowest layer. It manages the interface initialization process, serializes the data from the upper layers, and transmits it to the other device at compatible signaling speeds.
- Data link layer: This middle layer does link management and data integrity through error detection and correction. When transmitting, this layer receives transaction layer packets (TLPs), adds data integrity codes, and sends them to the physical layer for transmission. On the receiving side, it checks TLP integrity, forwards valid packets to the transaction layer, and requests retransmission of erroneous packets. It also creates its own data link layer packets (DLLPs) for link management.
- Transaction layer: This is the highest layer where read and write operations are communicated through TLPs. The TLP format supports various forms of addressing relevant to the transaction. This layer also facilitates sideband signals like power management requests.
2. Flow control units (FLITs)
At the lowest (physical) layer, payloads from the transaction layer or data link layer are chopped up into FLITs. The encoding scheme for the data inside a FLIT depends on the PCIe operating mode.
A head FLIT signals the start of a transmission. Then, one or more data FLITs containing the packet payloads are sent. Finally, a tail FLIT indicates that the entire data sequence has been sent.
To reach high bandwidths, PCIe 6.0 uses 4-level pulse amplitude modulation (PAM4) signaling, which carries more chances of data errors. Forward error correction is essential. But that requires fixed-size FLITs, which is why they are fixed at 256 bytes.
3. Link training and status state machine (LTSSM)
PCIe link initialization consists of multiple steps in the physical layer during which the devices negotiate parameters like lane counts and data rates. The link transitions through a set of logical states represented by the LTSSM. Some of the key states include:
- Detect: The initial state where the devices are looking for a connection.
- Polling: The devices are trying to communicate and negotiate settings.
- Configuration: The devices are setting up their configuration parameters.
- Training: The devices exchange information to optimize the link’s performance.
- L0: This is the normal operating state where data and control packets can be sent and received.
- Hot reset: This state allows a configured link and device to be reset using in-band signaling.
4. Forward error correction (FEC)
FEC helps mitigate potentially higher bit error rates due to PAM4 without compromising the transfer speed. Instead of requesting retransmissions, PCIe 6.0 inserts FEC and cyclic redundancy check (CRC) codes in the FLITs. These codes enable the receiving device to detect if there are payload errors and algorithmically recover the original data without retransmissions.
What is the difference between a PCIe protocol analyzer and a PCIe exerciser?
An analyzer just captures the protocol-level data exchange between two devices. It is strictly read-only. No data is created on its own nor is the captured data modified in any way.
In contrast, an exerciser emulates the PCIe interface of an endpoint or a root complex. It can synthesize sequences of PCIe requests or responses, which makes it invaluable for thoroughly testing the gamut of PCIe interactions, corner cases, and error scenarios.
Analyzers and exercisers are often used together during compliance and interoperability testing.
How is a protocol analyzer used for debugging PCIe devices?
PCIe protocol analyzers and exercisers enable testing of many possible device topologies.
The following test use cases are common:
- Analyze PCIe traffic between a root complex and an endpoint: The analyzer is placed between a physical root complex and a PCIe endpoint device to test their interoperability.
- Root complex emulation for testing endpoints: The exerciser emulates a root complex. It’s connected to the protocol analyzer, which is then connected to the PCIe device under test. This configuration enables the device manufacturer to conduct PCIe conformance tests.
- Endpoint emulation for a testing root complex (RC): The exerciser emulates various endpoint devices. It’s connected to the analyzer. The RC under test is connected to the other end of the analyzer. This allows you to subject the RC to a variety of PCIe communications from devices.
How do PCIe protocol analyzers decode and present captured traffic?
For effective debugging and troubleshooting, the Keysight P5570A analyzer and its P5500 software present detailed information from all the protocol layers in friendly graphical user interfaces.
The low-level link initialization is a critical process and the source of many errors. An LTSSM view to see the state changes is invaluable for understanding what’s happening under the hood, as shown below.
Once the link is up, a lane view gives insights into low-level data transmission by showing the exact symbols and their two-bit sequences being sent in each PCIe lane as shown here.
Are the FLITs being sent correctly? Are FEC and CRC fine? These questions can be answered by checking the FLIT view as shown below.
The high-level transactions can be analyzed from the transaction decode screen.
The TLPs and DLLPs of transactions can be viewed in the packet view and traffic overview screens shown below.
Measuring PCIe performance is as critical as getting the functionality right. The performance view shows key timing and speed metrics to identify bottlenecks as shown here.
How do PCIe protocol analyzers account for forward error correction?
Protocol analyzers and exercisers like the P5570A and P5573A are designed to not affect the functionality or performance of FEC mechanisms through the following strategies:
- Non-intrusive designs: The device does not modify or interfere with the FEC process at all. They are designed to ensure full transparency to the link.
- Low latency interception: The FEC latency must be below two nanoseconds to meet PCIe performance standards. Protocol analyzers are designed to work within these stringent latency requirements, ensuring that engineers get an accurate view of the true performance of the link.
- Awareness of flow control and signaling advances: The analyzers and exercisers are designed to work with PCIe 6.0 improvements like fixed-size FLITs and PAM4 signaling that are critical to the FEC process.
- Error correction monitoring: Advanced protocol analyzers like the P5570A can monitor the effectiveness of FEC. They can analyze the status of FEC steps and check whether the FECs are successful and CRCs pass or fail.
- Comprehensive test automation: Effective validation of FEC performance requires extensive test automation to run stress tests and simulations that exercise it under a wide swathe of conditions. This includes deliberately fluctuating signal conditions and injecting errors using application programming interfaces (APIs) to assess FEC robustness.
What are the benefits of modular protocol analyzers and exercisers over traditional methods?
Self-contained modular analyzers and exercisers with interposer form factors provide several benefits over traditional analyzers:
- Cables used in traditional designs degrade signal integrity, complicating the link initialization steps.
- Modular designs minimize their impact on the link using fine-tuned equalization and amplification to maintain the original signal quality.
- Interposer instruments allow plug-and-play functionality with direct connections between the device and the analyzer or exerciser.
Qualify your PCIe devices with Keysight protocol analyzers
Keysight’sPCIe 6.0 and 5.0 protocol solutions offer the most accurate insights into your PCIe devices with high signal integrity that largely removes its effects on the link. They enable your engineers to focus on actual device debugging instead of wondering whether their test equipment is introducing or masking any issues.
We continually pioneer groundbreaking measurement tools that push the boundaries of high-speed digital innovation. PCIe 7.0®, aiming at a 2025 release, is the next-gen interconnect technology that will increase data transfer speeds to 128 GT/s per pin — critical for efficiently scaling AI training and computation in data centers
Contact us for insights on using these instruments to help you boost the confidence of your quality assurance workflows.