Mini-PCIe

Not to be confused with PCI-X.
PCI Express
200px
Year created 2004
Created by Intel · Dell · HP · IBM
Supersedes AGP · PCI · PCI-X
Width in bits 1–32
Number of devices One device each on each endpoint of each connection. PCI Express switches can create multiple endpoints out of one endpoint to allow sharing one endpoint with multiple devices.
Capacity

Per lane (each direction):

  • v1.x: 250 MB/s (2.5 GT/s)
  • v2.x: 500 MB/s (5 GT/s)
  • v3.0: 985 MB/s (8 GT/s)
  • v4.0: 1969 MB/s (16 GT/s)

So, a 16-lane slot (each direction):

  • v1.x: 4 GB/s (40 GT/s)
  • v2.x: 8 GB/s (80 GT/s)
  • v3.0: 15.75 GB/s (128 GT/s)
  • v4.0: 31.51 GB/s (256 GT/s)
Style Serial
Hotplugging interface Yes, if ExpressCard, Mobile PCI Express Module or XQD card
External interface Yes, with PCI Express External Cabling, such as Thunderbolt

PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe, is a high-speed serial computer expansion bus standard designed to replace the older PCI, PCI-X, and AGP bus standards. PCIe has numerous improvements over the aforementioned bus standards, including higher maximum system bus throughput, lower I/O pin count and smaller physical footprint, better performance-scaling for bus devices, a more detailed error detection and reporting mechanism (Advanced Error Reporting (AER) [1]), and native hot-plug functionality. More recent revisions of the PCIe standard support hardware I/O virtualization.

The PCIe electrical interface is also used in a variety of other standards, most notably ExpressCard, a laptop expansion card interface.

Format specifications are maintained and developed by the PCI-SIG (PCI Special Interest Group), a group of more than 900 companies that also maintain the conventional PCI specifications. PCIe 3.0 is the latest standard for expansion cards that is in production and available on mainstream personal computers.[2][3]

Applications

PCI Express operates in consumer, server, and industrial applications, as a motherboard-level interconnect (to link motherboard-mounted peripherals), a passive backplane interconnect and as an expansion card interface for add-in boards.

In virtually all modern (as of 2012) PCs, from consumer laptops and desktops to enterprise data servers, the PCIe bus serves as the primary motherboard-level interconnect, connecting the host system-processor with both integrated-peripherals (surface-mounted ICs) and add-on peripherals (expansion cards.) In most of these systems, the PCIe bus co-exists with one or more legacy PCI buses, for backward compatibility with the large body of legacy PCI peripherals.

Architecture

Conceptually, the PCIe bus is like a high-speed serial replacement of the older PCI/PCI-X bus,[4] an interconnect bus using shared address/data lines.

A key difference between PCIe bus and the older PCI is the bus topology. PCI uses a shared parallel bus architecture, where the PCI host and all devices share a common set of address/data/control lines. In contrast, PCIe is based on point-to-point topology, with separate serial links connecting every device to the root complex (host). Due to its shared bus topology, access to the older PCI bus is arbitrated (in the case of multiple masters), and limited to one master at a time, in a single direction. Furthermore, the older PCI clocking scheme limits the bus clock to the slowest peripheral on the bus (regardless of the devices involved in the bus transaction). In contrast, a PCIe bus link supports full-duplex communication between any two endpoints, with no inherent limitation on concurrent access across multiple endpoints.

In terms of bus protocol, PCIe communication is encapsulated in packets. The work of packetizing and de-packetizing data and status-message traffic is handled by the transaction layer of the PCIe port (described later). Radical differences in electrical signaling and bus protocol require the use of a different mechanical form factor and expansion connectors (and thus, new motherboards and new adapter boards); PCI slots and PCIe slots are not interchangeable. At the software level, PCIe preserves backward compatibility with PCI; legacy PCI system software can detect and configure newer PCIe devices without explicit support for the PCIe standard, though PCIe's new features are inaccessible.

The PCIe link between two devices can consist of anywhere from 1 to 32 lanes. In a multi-lane link, the packet data is striped across lanes, and peak data-throughput scales with the overall link width. The lane count is automatically negotiated during device initialization, and can be restricted by either endpoint. For example, a single-lane PCIe (×1) card can be inserted into a multi-lane slot (×4, ×8, etc.), and the initialization cycle auto-negotiates the highest mutually supported lane count. The link can dynamically down-configure the link to use fewer lanes, thus providing some measure of failure tolerance in the presence of bad or unreliable lanes. The PCIe standard defines slots and connectors for multiple widths: ×1, ×4, ×8, ×16, ×32. This allows PCIe bus to serve both cost-sensitive applications where high throughput is not needed, as well as performance-critical applications such as 3D graphics, network (10 Gigabit Ethernet, multiport Gigabit Ethernet), and enterprise storage (SAS, Fibre Channel.)

As a point of reference, a PCI-X (133 MHz 64-bit) device and PCIe device using four lanes (×4), Gen1 speed have roughly the same peak transfer rate in a single-direction: 1064 MB/sec. The PCIe bus has the potential to perform better than the PCI-X bus in cases where multiple devices are transferring data communicating simultaneously, or if communication with the PCIe peripheral is bidirectional.

Interconnect

PCIe devices communicate via a logical connection called an interconnect[5] or link. A link is a point-to-point communication channel between two PCIe ports, allowing both to send/receive ordinary PCI-requests (configuration read/write, I/O read/write, memory read/write) and interrupts (INTx, MSI, MSI-X). At the physical level, a link is composed of one or more lanes.[5] Low-speed peripherals (such as an 802.11 Wi-Fi card) use a single-lane (×1) link, while a graphics adapter typically uses a much wider (and thus, faster) 16-lane link.

Lane

A lane is composed of two differential signaling pairs: one pair for receiving data, the other for transmitting. Thus, each lane is composed of four wires or signal traces. Conceptually, each lane is used as a full-duplex byte stream, transporting data packets in eight-bit 'byte' format, between endpoints of a link, in both directions simultaneously.[6] Physical PCIe slots may contain from one to thirty-two lanes, in powers of two (1, 2, 4, 8, 16 and 32).[5] Lane counts are written with an × prefix (e.g., ×16 represents a sixteen-lane card or slot), with ×16 being the largest size in common use.[7]

Serial bus

The bonded serial format was chosen over a traditional parallel bus format due to the latter's inherent limitations, including single-duplex operation, excess signal count and an inherently lower bandwidth due to timing skew. Timing skew results from separate electrical signals within a parallel interface traveling down different-length conductors, on potentially different printed circuit board layers, at possibly different signal velocities. Despite being transmitted simultaneously as a single word, signals on a parallel interface experience different travel times and arrive at their destinations at different moments. When the interface clock rate is increased to a point where its inverse (i.e., its clock period) is shorter than the largest possible time between signal arrivals, the signals no longer arrive with sufficient coincidence to make recovery of the transmitted word possible. Since timing skew over a parallel bus can amount to a few nanoseconds, the resulting bandwidth limitation is in the range of hundreds of megahertz.

A serial interface does not exhibit timing skew because there is only one differential signal in each direction within each lane, and there is no external clock signal since clocking information is embedded within the serial signal. As such, typical bandwidth limitations on serial signals are in the multi-gigahertz range. PCIe is just one example of a general trend away from parallel buses to serial interconnects. Other examples include Serial ATA, USB, SAS, FireWire (1394) and RapidIO.

Multichannel serial design increases flexibility by allocating slow devices to fewer lanes than fast devices.

Form factors

PCI Express (standard)

A PCIe card fits into a slot of its physical size or larger (maximum ×16), but may not fit into a smaller PCIe slot (e.g.,a ×16 card in a ×8 slot). Some slots use open-ended sockets to permit physically longer cards and negotiate the best available electrical connection. The number of lanes actually connected to a slot may also be less than the number supported by the physical slot size.

An example is a ×8 slot that actually only runs at ×1. These slots allow any ×1, ×2, ×4 or ×8 card, though only running at ×1 speed. This type of socket is called a ×8 (×1 mode) slot, meaning it physically accepts up to ×8 cards but only runs at ×1 speed. This is also sometimes specified as "×size (@×capacity)" (for example, "×16(@×8)"). The advantage is that it can accommodate a larger range of PCIe cards without requiring motherboard hardware to support the full transfer rate. This keeps design and implementation costs down.

Pinout

The following table identifies the conductors on each side of the edge connector on a PCI Express card. The solder side of the printed circuit board (PCB) is the A side, and the component side is the B side.[8] PRSNT1# and PRSNT2# pins must be slightly shorter than the rest, to ensure that a hot-plugged card is fully inserted. The WAKE# pin uses full voltage to wake the computer, but must be pulled high from the standby power to indicate that the card is wake capable.[9]

PCI express ×16 connector pinout
Pin Side B Side A Comments
1 +12 V PRSNT1# Must connect to furthest-apart PRSNT2#
2 +12 V +12 V
3 +12 V +12 V
4 Ground Ground
5 SMCLK TCK SMBus and JTAG port pins
6 SMDAT TDI
7 Ground TDO
8 +3.3 V TMS
9 TRST# +3.3 V
10 +3.3 V aux +3.3 V Standby power
11 WAKE# PERST# Link reactivation; power and REFCLK stabilized
Key notch
12 Reserved Ground
13 Ground REFCLK+ Reference clock differential pair
14 HSOp(0) REFCLK- Lane 0 transmit data, + and −
15 HSOn(0) Ground
16 Ground HSIp(0) Lane 0 receive data, + and −
17 PRSNT2# HSIn(0)
18 Ground Ground
PCI ×1 board ends at pin 18
19 HSOp(1) Reserved Lane 1 transmit data, + and −
20 HSOn(1) Ground
21 Ground HSIp(1) Lane 1 receive data, + and −
22 Ground HSIn(1)
23 HSOp(2) Ground Lane 2 transmit data, + and −
24 HSOn(2) Ground
25 Ground HSIp(2) Lane 2 receive data, + and −
26 Ground HSIn(2)
27 HSOp(3) Ground Lane 3 transmit data, + and −
28 HSOn(3) Ground
29 Ground HSIp(3) Lane 3 receive data, + and −
30 Reserved HSIn(3)
31 PRSNT2# Ground
32 Ground Reserved
PCI ×4 board ends at pin 32
33 HSOp(4) Reserved Lane 4 transmit data, + and −
34 HSOn(4) Ground
35 Ground HSIp(4) Lane 4 receive data, + and −
36 Ground HSIn(4)
37 HSOp(5) Ground Lane 5 transmit data, + and −
38 HSOn(5) Ground
39 Ground HSIp(5) Lane 5 receive data, + and −
40 Ground HSIn(5)
41 HSOp(6) Ground Lane 6 transmit data, + and −
42 HSOn(6) Ground
43 Ground HSIp(6) Lane 6 receive data, + and −
44 Ground HSIn(6)
45 HSOp(7) Ground Lane 7 transmit data, + and −
46 HSOn(7) Ground
47 Ground HSIp(7) Lane 7 receive data, + and −
48 PRSNT2# HSIn(7)
49 Ground Ground
PCI ×8 board ends at pin 49
50 HSOp(8) Reserved Lane 8 transmit data, + and −
51 HSOn(8) Ground
52 Ground HSIp(8) Lane 8 receive data, + and −
53 Ground HSIn(8)
54 HSOp(9) Ground Lane 9 transmit data, + and −
55 HSOn(9) Ground
56 Ground HSIp(9) Lane 9 receive data, + and −
57 Ground HSIn(9)
58 HSOp(10) Ground Lane 10 transmit data, + and −
59 HSOn(10) Ground
60 Ground HSIp(10) Lane 10 receive data, + and −
61 Ground HSIn(10)
62 HSOp(11) Ground Lane 11 transmit data, + and −
63 HSOn(11) Ground
64 Ground HSIp(11) Lane 11 receive data, + and −
65 Ground HSIn(11)
66 HSOp(12) Ground Lane 12 transmit data, + and −
67 HSOn(12) Ground
68 Ground HSIp(12) Lane 12 receive data, + and −
69 Ground HSIn(12)
70 HSOp(13) Ground Lane 13 transmit data, + and −
71 HSOn(13) Ground
72 Ground HSIp(13) Lane 13 receive data, + and −
73 Ground HSIn(13)
74 HSOp(14) Ground Lane 14 transmit data, + and −
75 HSOn(14) Ground
76 Ground HSIp(14) Lane 14 receive data, + and −
77 Ground HSIn(14)
78 HSOp(15) Ground Lane 15 transmit data, + and −
79 HSOn(15) Ground
80 Ground HSIp(15) Lane 15 receive data, + and −
81 PRSNT2# HSIn(15)
82 Reserved Ground
Legend
Ground pin Zero volt reference
Power pin Supplies power to the PCIe card
Output pin Signal from the card to the motherboard
Input pin Signal from the motherboard to the card
Open drain May be pulled low and/or sensed by multiple cards
Sense pin Tied together on card
Reserved Not presently used, do not connect

Power

All sizes of ×4 and ×8 PCI Express cards are allowed a maximum power consumption of 25 W. All ×1 cards are initially 10 W; full-height cards may configure themselves as 'high-power' to reach 25 W, while half-height ×1 cards are fixed at 10 W. All sizes of ×16 cards are initially 25 W; like ×1 cards, half-height cards are limited to this number while full-height cards may increase their power after configuration. They can use up to 75 W (3.3 V/3 A + 12 V/5.5 A), though the specification demands that the higher-power configuration be used for graphics cards only, while cards of other purposes are to remain at 25 W.[10][11] Optional connectors add 75 W (6-pin) and/or 150 W (8-pin) power for up to 300 W total (2×75 W + 1×150 W). Some cards are using two 8-pin connectors, but this has not been standardized yet, therefore such cards must not carry the official PCI-Express logo. This configuration would allow 375 W total (1×75 W + 2×150 W) and will likely be standardized by PCI-SIG with the PCI-Express 4.0 standard. The 8-pin PCI-Express connector could be mistaken with the EPS12V connector, which is mainly used for powering SMP and multi-core systems.

PCI Express Mini Card

PCI Express Mini Card (also known as Mini PCI Express, Mini PCIe, Mini PCI-E, and PEM) is a replacement for the Mini PCI form factor, based on PCI Express. It is developed by the PCI-SIG. The host device supports both PCI Express and USB 2.0 connectivity, and each card may use either standard. Most laptop computers built after 2005 are based on PCI Express and can have several Mini Card slots.

Physical dimensions

PCI Express Mini Cards are 30×50.95 mm. There is a 52-pin edge connector, consisting of two staggered rows on a 0.8 mm pitch. Each row has eight contacts, a gap equivalent to four contacts, then a further 18 contacts. A half-length card is also specified 30×26.8 mm. Cards have a thickness of 1.0 mm (excluding components).

Electrical interface

PCI Express Mini Card edge connectors provide multiple connections and buses:

  • PCIe ×1
  • USB 2.0
  • SMBus
  • Wires to diagnostics LEDs for wireless network (i.e., Wi-Fi) status on computer's chassis
  • SIM card for GSM and WCDMA applications. (UIM signals on spec)
  • Future extension for another PCIe lane
  • 1.5 and 3.3 volt power

Mini PCI Express & mSATA

Despite sharing the mini-PCI Express form factor, an mSATA slot is not necessarily electrically compatible with Mini PCI Express. For this reason, only certain notebooks are compatible with mSATA drives. Most compatible systems are based on Intel's Sandy Bridge processor architecture, using the Huron River platform. But for a mSATA/mini-PCI-E connector, the only prerequisite is that there is a switch which makes it either a mSATA or a mini-PCI-E slot and can be implemented on any platform.

Notebooks like Lenovo's T-Series, W-Series, and X-Series ThinkPads released in March–April 2011 have support for an mSATA SSD card in their WWAN card slot. The ThinkPad Edge E220s/E420s, and the Lenovo IdeaPad Y460/Y560 also support mSATA.[12]

Some notebooks (notably the Asus Eee PC, the Apple MacBook Air, and the Dell mini9 and mini10) use a variant of the PCI Express Mini Card as an SSD. This variant uses the reserved and several non-reserved pins to implement SATA and IDE interface passthrough, keeping only USB, ground lines, and sometimes the core PCIe 1x bus intact.[13] This makes the 'miniPCIe' flash and solid state drives sold for netbooks largely incompatible with true PCI Express Mini implementations.

Also, the typical Asus miniPCIe SSD is 71 mm long, causing the Dell 51 mm model to often be (incorrectly) referred to as half length. A true 51 mm Mini PCIe SSD was announced in 2009, with two stacked PCB layers, which allows for higher storage capacity. The announced design preserves the PCIe interface, making it compatible with the standard mini PCIe slot. No working product has yet been developed.

Intel has numerous desktop boards with the PCIe ×1 Mini-Card slot which typically do not support mSATA SSD. A list of desktop boards that natively support mSATA in the PCIe ×1 Mini-Card slot (typically multiplexed with a SATA port) is provided on the Intel Support site.[14]

PCI Express External Cabling

PCI Express External Cabling (also known as External PCI Express, Cabled PCI Express, or ePCIe) specifications were released by the PCI-SIG in February 2007.[15][16]

Standard cables and connectors have been defined for ×1, ×4, ×8, and ×16 link widths, with a transfer rate of 250 MB/s per lane. The PCI-SIG also expects the norm will evolve to reach 500 MB/s, as in PCI Express 2.0. The maximum cable length remains undetermined. An example of the uses of Cabled PCI Express is a metal enclosure, containing a number of PCI slots and PCI-to-ePCIe adapter circuitry. This device would not be possible had it not been for the ePCIe spec.

Derivative forms

There are several other expansion card types derived from PCIe. These include:

  • Low-height card
  • ExpressCard: successor to the PC Card form factor (with ×1 PCIe and USB 2.0; hot-pluggable)
  • PCI Express ExpressModule: a hot-pluggable modular form factor defined for servers and workstations
  • XQD card: a PCI Express-based flash card standard by the CompactFlash Association
  • XMC: similar to the CMC/PMC form factor (VITA 42.3)
  • AdvancedTCA: a complement to CompactPCI for larger applications; supports serial based backplane topologies
  • AMC: a complement to the AdvancedTCA specification; supports processor and I/O modules on ATCA boards (×1, ×2, ×4 or ×8 PCIe).
  • FeaturePak: a tiny expansion card format (43 × 65 mm) for embedded and small form factor applications; it implements two ×1 PCIe links on a high-density connector along with USB, I2C, and up to 100 points of I/O.
  • Universal IO: A variant from Super Micro Computer Inc designed for use in low profile rack mounted chassis. It has the connector bracket reversed so it cannot fit in a normal PCI Express socket, but is pin compatible and may be inserted if the bracket is removed.
  • Thunderbolt: A variant from Intel that combines DisplayPort and PCIe protocols in a form factor compatible with Mini DisplayPort.
  • Serial Digital Video Out: some 9xx series Intel chipsets allow for adding an additional output for the integrated video into a PCIe slot (mostly dedicated and 16 lanes)
  • M.2 (formerly known as NGFF)

History and revisions

While in early development, PCIe was initially referred to as HSI (for High Speed Interconnect), and underwent a name change to 3GIO (for 3rd Generation I/O) before finally settling on its PCI-SIG name PCI Express. A technical working group named the Arapaho Work Group (AWG) drew up the standard. For initial drafts, the AWG consisted only of Intel engineers; subsequently the AWG expanded to include industry partners.

PCIe is a technology under constant development and improvement. As of 2013 the PCI Express implementation has reached version 4.

PCI Express 1.0a

In 2003, PCI-SIG introduced PCIe 1.0a, with a per-lane data rate of 250 MB/s and a transfer rate of 2.5 gigatransfers per second (GT/s). Transfer rate is expressed in transfers per second instead of bits per second because the number of transfers includes the overhead bits, which do not provide additional throughput.[17]

PCIe 1.x uses an 8b/10b encoding scheme that results in a 20 percent ((10-8)/10) overhead on the raw bit rate. It uses a 2.5 GHz clock rate, therefore delivering an effective 250 000 000 bytes per second (250 MB/s) maximum data rate.[18]

PCI Express 1.1

In 2005, PCI-SIG[19] introduced PCIe 1.1. This updated specification includes clarifications and several improvements, but is fully compatible with PCI Express 1.0a. No changes were made to the data rate.

PCI Express 2.0

PCI-SIG announced the availability of the PCI Express Base 2.0 specification on 15 January 2007.[20] The PCIe 2.0 standard doubles the transfer rate compared with PCIe 1.0 to 5 GT/s and the per-lane throughput rises from 250 MB/s to 500 MB/s. This means a 32-lane PCIe connector (×32) can support throughput up to 16 GB/s aggregate.

PCIe 2.0 motherboard slots are fully backward compatible with PCIe v1.x cards. PCIe 2.0 cards are also generally backward compatible with PCIe 1.x motherboards, using the available bandwidth of PCI Express 1.1. Overall, graphic cards or motherboards designed for v2.0 will work with the other being v1.1 or v1.0a.

The PCI-SIG also said that PCIe 2.0 features improvements to the point-to-point data transfer protocol and its software architecture.[21]

Intel's first PCIe 2.0 capable chipset was the X38 and boards began to ship from various vendors (Abit, Asus, Gigabyte) as of October 21, 2007.[22] AMD started supporting PCIe 2.0 with its AMD 700 chipset series and nVidia started with the MCP72.[23] All of Intel's prior chipsets, including the Intel P35 chipset, supported PCIe 1.1 or 1.0a.[24]

Like 1.x, PCIe 2.0 uses an 8b/10b encoding scheme, therefore delivering, per-lane, an effective 4 Gbit/s max transfer rate from its 5 GT/s raw data rate.

PCI Express 2.1

PCI Express 2.1 supports a large proportion of the management, support, and troubleshooting systems planned for full implementation in PCI Express 3.0. However, the speed is the same as PCI Express 2.0.

PCI Express 3.0

PCI Express 3.0 Base specification revision 3.0 was made available in November 2010, after multiple delays. In August 2007, PCI-SIG announced that PCI Express 3.0 would carry a bit rate of 8 gigatransfers per second (GT/s), and that it would be backward compatible with existing PCIe implementations. At that time, it was also announced that the final specification for PCI Express 3.0 would be delayed until 2011.[25] New features for the PCIe 3.0 specification include a number of optimizations for enhanced signaling and data integrity, including transmitter and receiver equalization, PLL improvements, clock data recovery, and channel enhancements for currently supported topologies.[26]

Following a six-month technical analysis of the feasibility of scaling the PCIe interconnect bandwidth, PCI-SIG's analysis found that 8 gigatransfers per second can be manufactured in mainstream silicon process technology, and can be deployed with existing low-cost materials and infrastructure, while maintaining full compatibility (with negligible impact) to the PCIe protocol stack.

PCIe 3.0 upgrades the encoding scheme to 128b/130b from the previous 8b/10b, reducing the overhead to approximately 1.54% ((130-128)/130), as opposed to the 20% of PCIe 2.0. This is achieved by a technique called "scrambling" that applies a known binary polynomial to a data stream in a feedback topology. Because the scrambling polynomial is known, the data can be recovered by running it through a feedback topology using the inverse polynomial.[27] PCIe 3.0's 8 GT/s bit rate effectively delivers 985 MB/s per lane, double PCIe 2.0 bandwidth. PCI-SIG expects the PCIe 3.0 specifications to undergo rigorous technical vetting and validation before being released to the industry. This process, which was followed in the development of prior generations of the PCIe Base and various form factor specifications, includes the corroboration of the final electrical parameters with data derived from test silicon and other simulations conducted by multiple members of the PCI-SIG.

On November 18, 2010, the PCI Special Interest Group officially published the finalized PCI Express 3.0 specification to its members to build devices based on this new version of PCI Express.[28]

AMD's latest flagship graphic card, the Radeon HD 7970, launched on January 9, 2012, was the world's first PCIe 3.0 graphic card.[29] Initial reviews suggest that the new interface would not improve graphic performance compared to earlier PCIe 2.0, which, at the time of writing, is still under-utilized. However, the new interface would prove advantageous when used for general purpose computing with technologies like OpenCL, CUDA and C++ AMP.[30]

PCI Express 4.0

On November 29, 2011, PCI-SIG announced PCI Express 4.0 featuring 16 GT/s, still based on copper technology. Additionally, active and idle power optimizations are to be investigated. Final specifications are expected to be released in 2014–15.[31]

Current status

As of 2013 PCI Express has replaced AGP as the default interface for graphics cards on new systems. Almost all models of graphics cards released since 2010 by AMD (ATI) and NVIDIA use PCI Express. NVIDIA uses the high-bandwidth data transfer of PCIe for its Scalable Link Interface (SLI) technology, which allows multiple graphics cards of the same chipset and model number to run in tandem, allowing increased performance. AMD has also developed a multi-GPU system based on PCIe called CrossFire. AMD and NVIDIA have released motherboard chipsets that support as many as four PCIe ×16 slots, allowing tri-GPU and quad-GPU card configurations.

Extensions and future directions

Some vendors offer PCIe over fiber products,[32][33] but these generally find use only in specific cases where transparent PCIe bridging is preferable to using a more mainstream standard (such as InfiniBand or Ethernet) that may require additional software to support it; current implementations focus on distance rather than raw bandwidth and typically do not implement a full ×16 link.

Certain data-center applications (such as large computer clusters) require the use of fiber-optic interconnects due to the distance limitations inherent in copper cabling. Typically, a network-oriented standard such as Ethernet or Fibre Channel suffices for these applications, but in some cases the overhead introduced by routable protocols is undesirable and a lower-level interconnect, such as InfiniBand, RapidIO, or NUMAlink is needed. Local-bus standards such as PCIe and HyperTransport can in principle be used for this purpose,[34] but as of 2012 no major vendors offer systems in this vein.

Thunderbolt was developed by Intel as a general-purpose high speed interface combining a ×4 PCIe link with DisplayPort and was originally intended to be an all-fiber interface, but due to early difficulties in creating a consumer-friendly fiber interconnect, most early implementations are hybrid copper-fiber systems. A notable exception, the Sony VAIO Z VPC-Z2, uses a nonstandard USB port with an optical component to connect to an outboard PCIe display adapter. Apple has been the primary driver of Thunderbolt adoption through 2011, though several other vendors [35] have announced new products and systems featuring Thunderbolt.

Hardware protocol summary

The PCIe link is built around dedicated unidirectional couples of serial (1-bit), point-to-point connections known as lanes. This is in sharp contrast to the earlier PCI connection, which is a bus-based system where all the devices share the same bidirectional, 32-bit or 64-bit parallel bus.

PCI Express is a layered protocol, consisting of a transaction layer, a data link layer, and a physical layer. The Data Link Layer is subdivided to include a media access control (MAC) sublayer. The Physical Layer is subdivided into logical and electrical sublayers. The Physical logical-sublayer contains a physical coding sublayer (PCS). The terms are borrowed from the IEEE 802 networking protocol model.

Physical layer

The PCIe Physical Layer (PHY, PCIEPHY, PCI Express PHY, or PCIe PHY) specification is divided into two sub-layers, corresponding to electrical and logical specifications. The logical sublayer is sometimes further divided into a MAC sublayer and a PCS, although this division is not formally part of the PCIe specification. A specification published by Intel, the PHY Interface for PCI Express (PIPE),[36] defines the MAC/PCS functional partitioning and the interface between these two sub-layers. The PIPE specification also identifies the physical media attachment (PMA) layer, which includes the serializer/deserializer (SerDes) and other analog circuitry; however, since SerDes implementations vary greatly among ASIC vendors, PIPE does not specify an interface between the PCS and PMA.

At the electrical level, each lane consists of two unidirectional LVDS or PCML pairs at 2.525 Gbit/s. Transmit and receive are separate differential pairs, for a total of four data wires per lane.

A connection between any two PCIe devices is known as a link, and is built up from a collection of one or more lanes. All devices must minimally support single-lane (×1) link. Devices may optionally support wider links composed of 2, 4, 8, 12, 16, or 32 lanes. This allows for very good compatibility in two ways:

  • A PCIe card physically fits (and works correctly) in any slot that is at least as large as it is (e.g., an ×1 sized card will work in any sized slot);
  • A slot of a large physical size (e.g., ×16) can be wired electrically with fewer lanes (e.g., ×1, ×4, ×8, or ×12) as long as it provides the ground connections required by the larger physical slot size.

In both cases, PCIe negotiates the highest mutually supported number of lanes. Many graphics cards, motherboards and BIOS versions are verified to support ×1, ×4, ×8 and ×16 connectivity on the same connection.

Even though the two would be signal-compatible, it is not usually possible to place a physically larger PCIe card (e.g., a ×16 sized card) into a smaller slot —though if the PCIe slots are altered or a riser is used most motherboards will allow this. Typically the technique is used for displaying to multiple monitors in a simulator configuration.

The width of a PCIe connector is 8.8 mm, while the height is 11.25 mm, and the length is variable. The fixed section of the connector is 11.65 mm in length and contains two rows of 11 (22 pins total), while the length of the other section is variable depending on the number of lanes. The pins are spaced at 1 mm intervals, and the thickness of the card going into the connector is 1.8 mm.[37][38]

Lanes Pins Length
Total Variable Total Variable
×1 2×18 = 36[39] 2×7 = 14 25 mm 7.65 mm
×4 2×32 = 64 2×21 = 42 39 mm 21.65 mm
×8 2×49 = 98 2×38 = 76 56 mm 38.65 mm
×16 2×82 = 164 2×71 = 142 89 mm 71.65 mm

Data transmission

PCIe sends all control messages, including interrupts, over the same links used for data. The serial protocol can never be blocked, so latency is still comparable to conventional PCI, which has dedicated interrupt lines.

Data transmitted on multiple-lane links is interleaved, meaning that each successive byte is sent down successive lanes. The PCIe specification refers to this interleaving as data striping. While requiring significant hardware complexity to synchronize (or deskew) the incoming striped data, striping can significantly reduce the latency of the nth byte on a link. Due to padding requirements, striping may not necessarily reduce the latency of small data packets on a link.

As with other high data rate serial transmission protocols, the clock is embedded in the signal. At the physical level, PCI Express 2.0 utilizes the 8b/10b encoding scheme[27] to ensure that strings of consecutive ones or consecutive zeros are limited in length. This coding was used to prevent the receiver from losing track of where the bit edges are. In this coding scheme every eight (uncoded) payload bits of data are replaced with 10 (encoded) bits of transmit data, causing a 20% overhead in the electrical bandwidth. To improve the available bandwidth, PCI Express version 3.0 employs 128b/130b encoding instead: similar but with much lower overhead.

Many other protocols (such as SONET) use a different form of encoding known as scrambling to embed clock information into data streams. The PCIe specification also defines a scrambling algorithm, but it is used to reduce electromagnetic interference (EMI) by preventing repeating data patterns in the transmitted data stream.

Data link layer

The Data Link Layer performs three vital services for the PCIe express link:

  1. sequence the transaction layer packets (TLPs) that are generated by the transaction layer,
  2. ensure reliable delivery of TLPs between two endpoints via an acknowledgement protocol (ACK and NAK signaling) that explicitly requires replay of unacknowledged/bad TLPs,
  3. initialize and manage flow control credits

On the transmit side, the data link layer generates an incrementing sequence number for each outgoing TLP. It serves as a unique identification tag for each transmitted TLP, and is inserted into the header of the outgoing TLP. A 32-bit cyclic redundancy check code (known in this context as Link CRC or LCRC) is also appended to the end of each outgoing TLP.

On the receive side, the received TLP's LCRC and sequence number are both validated in the link layer. If either the LCRC check fails (indicating a data error), or the sequence-number is out of range (non-consecutive from the last valid received TLP), then the bad TLP, as well as any TLPs received after the bad TLP, are considered invalid and discarded. The receiver sends a negative acknowledgement message (NAK) with the sequence-number of the invalid TLP, requesting re-transmission of all TLPs forward of that sequence-number. If the received TLP passes the LCRC check and has the correct sequence number, it is treated as valid. The link receiver increments the sequence-number (which tracks the last received good TLP), and forwards the valid TLP to the receiver's transaction layer. An ACK message is sent to remote transmitter, indicating the TLP was successfully received (and by extension, all TLPs with past sequence-numbers.)

If the transmitter receives a NAK message, or no acknowledgement (NAK or ACK) is received until a timeout period expires, the transmitter must retransmit all TLPs that lack a positive acknowledgement (ACK). Barring a persistent malfunction of the device or transmission medium, the link-layer presents a reliable connection to the transaction layer, since the transmission protocol ensures delivery of TLPs over an unreliable medium.

In addition to sending and receiving TLPs generated by the transaction layer, the data-link layer also generates and consumes DLLPs, data link layer packets. ACK and NAK signals are communicated via DLLPs, as are flow control credit information, some power management messages and flow control credit information (on behalf of the transaction layer.)

In practice, the number of in-flight, unacknowledged TLPs on the link is limited by two factors: the size of the transmitter's replay buffer (which must store a copy of all transmitted TLPs until the remote receiver ACKs them), and the flow control credits issued by the receiver to a transmitter. PCI Express requires all receivers to issue a minimum number of credits, to guarantee a link allows sending PCIConfig TLPs and message TLPs.

Transaction layer

PCI Express implements split transactions (transactions with request and response separated by time), allowing the link to carry other traffic while the target device gathers data for the response.

PCI Express uses credit-based flow control. In this scheme, a device advertises an initial amount of credit for each received buffer in its transaction layer. The device at the opposite end of the link, when sending transactions to this device, counts the number of credits each TLP consumes from its account. The sending device may only transmit a TLP when doing so does not make its consumed credit count exceed its credit limit. When the receiving device finishes processing the TLP from its buffer, it signals a return of credits to the sending device, which increases the credit limit by the restored amount. The credit counters are modular counters, and the comparison of consumed credits to credit limit requires modular arithmetic. The advantage of this scheme (compared to other methods such as wait states or handshake-based transfer protocols) is that the latency of credit return does not affect performance, provided that the credit limit is not encountered. This assumption is generally met if each device is designed with adequate buffer sizes.

PCIe 1.x is often quoted to support a data rate of 250 MB/s in each direction, per lane. This figure is a calculation from the physical signaling rate (2.5 Gbaud) divided by the encoding overhead (10 bits per byte.) This means a sixteen lane (×16) PCIe card would then be theoretically capable of 16×250 MB/s = 4 GB/s in each direction. While this is correct in terms of data bytes, more meaningful calculations are based on the usable data payload rate, which depends on the profile of the traffic, which is a function of the high-level (software) application and intermediate protocol levels.

Like other high data rate serial interconnect systems, PCIe has a protocol and processing overhead due to the additional transfer robustness (CRC and acknowledgements). Long continuous unidirectional transfers (such as those typical in high-performance storage controllers) can approach >95% of PCIe's raw (lane) data rate. These transfers also benefit the most from increased number of lanes (×2, ×4, etc.) But in more typical applications (such as a USB or Ethernet controller), the traffic profile is characterized as short data packets with frequent enforced acknowledgements.[40] This type of traffic reduces the efficiency of the link, due to overhead from packet parsing and forced interrupts (either in the device's host interface or the PC's CPU.) Being a protocol for devices connected to the same printed circuit board, it does not require the same tolerance for transmission errors as a protocol for communication over longer distances, and thus, this loss of efficiency is not particular to PCIe.

Uses

External PCIe cards

Theoretically, external PCIe could give a notebook the graphics power of a desktop, by connecting a notebook with any PCIe desktop video card (enclosed in its own external housing, with strong power supply and cooling); possible with an ExpressCard interface or a Thunderbolt interface. The ExpressCard interface provides bit rates of 5 Gbit/s (0.5 GB/s throughput), whereas the Thunderbolt interface provides bit rates of up to 10 Gbit/s (1 GB/s throughput). However, high-end video cards can go up to PCIe 3.0 ×16, which can transfer at 128 Gbit/s (15.75 GB/s), which means data transfer to an external video card may perform 10 times slower than to a video card connected directly to the motherboard.[41][42][43][44][45]

IBM/Lenovo has also included a PCI-Express slot in their Advanced Docking Station 250310U. It provides a half-sized slot with an ×16 length socket, but only ×1 connectivity.[46] However, docking stations with expansion slots are becoming less common as the laptops are getting more advanced video cards and either DVI-D interfaces, or DVI-D pass through for port replicators and docking stations.

Additionally, Nvidia has developed Quadro Plex external PCIe video cards that can be used for advanced graphic applications. These video cards require a PCI Express ×8 or ×16 slot for the interconnection cable.[47] In 2008, AMD announced the ATI XGP technology, based on a proprietary cabling system that is compatible with PCIe ×8 signal transmissions.[48] This connector is available on the Fujitsu Amilo and the Acer Ferrari One notebooks. Only Fujitsu has an actual external box available, which also works on the Ferrari One. Recently Acer launched the Dynavivid graphics dock for XGP.

There are now card hubs in development that one can connect to a laptop through an ExpressCard slot, though they are currently rare, obscure, or unavailable on the open market. These hubs can accept full-sized cards.

Magma and ViDock also make use of ExpressCard and implement the usage of external graphic systems. ViDock ia an expansion chassis tailored specifically for adapting PCI Express graphics cards for use with ExpressCard equipped portable computers. This enables connecting PCIe cards externally. Developments in these technologies are still developing; other examples are MSI GUS, Asus XG Station.

Recently, Intel and Apple introduced Thunderbolt, allowing for external PCI(e) devices to transfer at double the speeds of the ExpressCard interface. However a mid-range external video card would still be severely throttled by the slow connection.

Thunderbolt has given opportunity to new and faster products to connect with a PCIe card externally. Magma has released the ExpressBox 3T, which can hold up to three PCIe cards (two at 8× and one at 4×); this allows for a better workstation when a notebook lacks many ports. MSI also released the Thunderbolt GUS II, a PCIe chassis dedicated for video cards.[49] Other products such as the Sonnet’s Echo Express and mLogic’s mLink are Thunderbolt PCIe chassis in a smaller form factor; these allow connectivity to low-profile video cards, sound cards, network cards, memory, storage, etc.[50] However, all these products require the use of a Thunderbolt port (i.e. Thunderbolt devices), making them incompatible with the vast majority of computers.

External memory

PCI Express protocol can be used as data interface to flash memory devices, such as memory cards and solid state drives. One such format is XQD card developed by the CompactFlash Association, SATA Express[51] and SCSI Express.[52]

Many high-performance, enterprise-class solid state drives are designed as PCI Express RAID controller cards with flash memory chips placed directly on the circuit board; this allows much higher transfer rates (over 1 Gbyte/s) and IOPS (I/O operations per second) (over 1 million) comparing to Serial ATA or SAS drives.

OCZ and Marvell co-developed the native PCIe solid state drive controller Kilimanjaro that is utilized in OCZ's Z-Drive 5. The Z-Drive 5 is designed for a PCIe 3.0 ×16 slot and when the highest capacity (12 TB) version is installed in such a slot it can run up to 7.2 gigabytes per second sequential transfers and up to 2.52 million IOPS in random transfers.[53]

Competing protocols

Several communications standards have emerged based on high bandwidth serial architectures. These include InfiniBand, RapidIO, HyperTransport, QPI and StarFabric. The differences are based on the tradeoffs between flexibility and extensibility vs latency and overhead. An example of such a tradeoff is adding complex header information to a transmitted packet to allow for complex routing (PCI Express is not capable of this). The additional overhead reduces the effective bandwidth of the interface and complicates bus discovery and initialization software. Also making the system hot-pluggable requires that software track network topology changes. Examples of buses suited for this purpose are InfiniBand and StarFabric.

Another example is making the packets shorter to decrease latency (as is required if a bus must operate as a memory interface). Smaller packets mean packet headers consume a higher percentage of the packet, thus decreasing the effective bandwidth. Examples of bus protocols designed for this purpose are RapidIO and HyperTransport.

PCI Express falls somewhere in the middle, targeted by design as a system interconnect (local bus) rather than a device interconnect or routed network protocol. Additionally, its design goal of software transparency constrains the protocol and raises its latency somewhat.

Development tools

When developing and/or troubleshooting the PCI Express bus, examination of hardware signals can be very important to find the problems. Oscilloscopes, logic analyzers and bus analyzers are tools that collect, analyze, decode, store signals so people can view the high-speed waveforms at their leisure.

See also

References

Further reading

  • 1120 pp.
  • , 1056 pp.
  • , 325 pp.

External links

  • .
  • .
  • (Requires PCI-SIG membership)
  • , the industry organization that maintains and develops the various PCI standards.
  • .
  • .
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.