Bunch of Wires (BoW) PHY Specification
The Open Domain-Specific Architecture BoW Workstream
DRAFT Version 1.9d
January 2nd, 2023
This section provides glossary used in this specification.
Term | Abbreviation | Definition |
---|---|---|
Bunch of Wires | BoW | The name for the PHY specification defined in this document |
Die-to-die | D2D | Generic term used to refer to on-package interconnect |
BoW Mode | N/A | A specific defined mode of operation for a BoW interface |
picojoules per bit | pJ/bit | Energy required to transport a bit of data over a D2D interface |
PHY | The set of circuitry physically communicating bits from one die to another | |
Core Logic | Digital logic transmitting data to/from the PHY | |
Control Logic | Logic used to manage the operation of the PHY | |
Tera/Giga bits per second | Tbps/Gbps | Measures of the speed of data transmission on the PHY |
Beachfront | The length of die edge required by a PHY implementation | |
Clock | A signal that regulates the speed of data transmission | |
Bump | Solder balls grown on a die to allow connection to off-die wires | |
Channel | A term for the physical connection between a transmitter and a receiver | |
Test | The process of verifying functional correctness of a circuit | |
Initialization | The process of preparing an interface for data transmission | |
This document uses the following terms as defined below.
Contributions to this Specification are made under the terms and conditions set forth in the modified Open Web Foundation Contributor License Agreement (“OWF CLA 1.0”) (“Contribution License”) by:
ANALOG PORT, BLUE CHEETAH ANALOG DESIGN, D-MATRIX, IBM, KEYSIGHT, TESSOLVE, VENTANA MICRO
You can review the signed copies of the applicable Contributor License(s) for this Specification on the OCP website at http://www.opencompute.org/products/specsanddesign
Usage of this Specification is governed by the terms and conditions set forth in the modified Open Web Foundation Final Specification Agreement (“OWFa 1.0”) (“Specification License”).
Notes:
NOTWITHSTANDING THE FOREGOING LICENSES, THIS SPECIFICATION IS PROVIDED BY OCP “AS IS” AND OCP EXPRESSLY DISCLAIMS ANY WARRANTIES (EXPRESS, IMPLIED, OR OTHERWISE), INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, FITNESS FOR A PARTICULAR PURPOSE, OR TITLE, RELATED TO THE SPECIFICATION. NOTICE IS HEREBY GIVEN, THAT OTHER RIGHTS NOT GRANTED AS SET FORTH ABOVE, INCLUDING WITHOUT LIMITATION, RIGHTS OF THIRD PARTIES WHO DID NOT EXECUTE THE ABOVE LICENSES, MAY BE IMPLICATED BY THE IMPLEMENTATION OF OR COMPLIANCE WITH THIS SPECIFICATION. OCP IS NOT RESPONSIBLE FOR IDENTIFYING RIGHTS FOR WHICH A LICENSE MAY BE REQUIRED IN ORDER TO IMPLEMENT THIS SPECIFICATION. THE ENTIRE RISK AS TO IMPLEMENTING OR OTHERWISE USING THE SPECIFICATION IS ASSUMED BY YOU. IN NO EVENT WILL OCP BE LIABLE TO YOU FOR ANY MONETARY DAMAGES WITH RESPECT TO ANY CLAIMS RELATED TO, OR ARISING OUT OF YOUR USE OF THIS SPECIFICATION, INCLUDING BUT NOT LIMITED TO ANY LIABILITY FOR LOST PROFITS OR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT, SPECIAL OR PUNITIVE DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF ACTION OF ANY KIND WITH RESPECT TO THIS SPECIFICATION, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND EVEN IF OCP HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The Bunch of Wires (BoW) specification defines a versatile, open and interoperable physical interface between two chiplets or chip-scale-packages (CSP) in a common package, and is fully backwards compatible with the Bunch of Wires specification. This document specifies the BoW interface PHY layer, and defines a set of die-to-die parallel interfaces that implementers / adopters the the flexibility to trade off throughput/chipedge for design complexity, cost, and packaging technology. The use of BoW is expected to be confined to connect die placed close to one another within the same package. In this environment, signal attenuation is small and the interface can be simple. The definition of the BoW interface aims to meet the following OCP tenets as follows:
The BoW specification provides several key advantages for chiplet-based systems:
Compared to SerDes, BoW uses a lower data rate/wire so it requires more wires. However the lower data rates allow use of single-ended signaling and denser wire packing. In addition, in laminates, BoW can take advantage of multiple wiring layers and in advanced packaging it can take advantage of the much-increased wire density.
Date | Revision | Author | Description |
---|---|---|---|
07/26/22 | Draft 1.1a | Elad Alon | Initial updates from 1.0 to 1.1 |
09/30/22 | Draft 1.1b | Marek Hempel, | Half-slice and power management added |
Elad Alon | |||
10/26/22 | Draft 1.1c | Ken Poulton, | Configurable directionality, redundancy |
Shahab Ardalan | added | ||
Elad Alon | |||
1/2/23 | Draft 1.1d | Elad Alon | Updated timing requirements, sideband |
slice definition added | |||
1/18/23 | Draft 1.9 | Bapi Vinnakota | Updated ODSA overview |
The scope of this document has several levels.
The specification of the BoW interface includes these requirements:
The specification includes recommendations for these elements:
The following activities are outside the scope of this document:
The following aspects may be addressed in subsequent versions of this specification:
This section provides an overview of the BoW physical interface (PHY) and its use in a multi-chiplet design.
The specifications must be met over process variation, supply voltage range and temperature range (PVT). Each implementation must document its supported I/O voltage range, supply voltage range and temperature range.
Table 2 summarize the conformance points that shall be met in order to comply with the BoW specification. Each of the conformance points is discussed in the specification.
Description | Section | Detail |
---|---|---|
BoW Modes | 5.4 | |
Die-to-die Signals (Wires) | 6.2 | |
Slice Logic Interface | 6.4.1 | |
BoWx Modes and Reach | 7 | Table 8 |
Wire and Slice Ordering | 8 | |
Voltages and Termination Resistance | 9.1 | |
PHY Protection | 9.2 | |
ESD | 9.3 | |
Return Loss and Parasitic Capacitance | 9.4 | |
Clocking | 10.2 | |
Clock and Data Specs | 10 | |
Channel Skew | 12.1 | |
External Facilities | 14.1 | |
Initialization | 14.2 | |
Control Register Mapping | 16 | |
BoW is an energy-efficient, easy-to-use PHY interface between a pair of die inside a single package as shown in Figure 1. The BoW PHY is defined as a single unidirectional slice. Multiple slices are combined to create links of the desired throughput. A link may be symmetric, asymmetric or unidirectional. The BoW PHYs between two die are physically connected through wires on a substrate or interposer. A BoW PHY does not have enough drive strength for off-package interfaces, nor is it designed for buses that are entirely on die.
This document specifies the protocol for a BoW PHY slice. The aggregation of multiple PHYs into a link is beyond the scope of this document.
A BoW PHY slice either transmits or receives 16 bits of data between die. The BoW is a source-synchronous PHY and each transmitting PHY slice transmits a complementary clock signal CLK+ and CLK- with the data. A BoW PHY optionally has two additional wires designated FEC (for Forward Error Correction) and AUX, for other optional functions such as Data Bus Inversion (DBI).
Within the package, the BoW datapath is transported on physical passive wires between the pair of connected die. The specifics of the wires, such as their density, maximum length, impedance characteristics and how they are realized vary with the packaging technology. In order to minimize power, unterminated and source-terminated links will have short reaches requiring chips to be adjacent.
A BoW PHY must be operable in one of the BoW Modes listed in ascending order in Table 3. A BoW Mode defines the speed of clock and data of the PHY on the die-to-die wires. In all modes, the data must be clocked DDR: the chip-to-chip data wire bit rate is double the clock wire frequency. All BoW interfaces faster than BoW-64 should also be able to support BoW-64. Supporting rates other than the defined modes is an implementation choice. There is more detail on BoW Modes in section 7.
BoW Mode | Slice Data Rate | Wire Bit Rate | TxClk |
---|---|---|---|
Gbps | Gbps/wire | GHz | |
BoW-32 | 32 | 2 | 1 |
BoW-64 | 64 | 4 | 2 |
BoW-128 | 128 | 8 | 4 |
BoW-256 | 256 | 16 | 8 |
BoW-384 | 384 | 24 | 12 |
BoW-512 | 512 | 32 | 16 |
Figure 2 shows the tradeoff between package, data rate, termination, and reach. Source-terminated BoW on laminate allows a longer reach than advanced packaging, but the wider design rules in laminate means that both of these cases are barely able to reach 8 Gbps/wire. A doubly-terminated link offers longer distances and higher rates, but requires a more complicated receiver design.
Figure 3 shows the logic interface between a BoW slice and the digital Link Layer logic in a chip. The speed at the logic interface (Figure 1) is implementation-dependent. Typically, PCLK will be the TxClk frequency divided by a power of 2, so 250, 500 and 1000 MHz are common rates. The data at the logic interface is SDR (bit rate equal to PCLK frequency).
This section specifies the control data signals into and out of device logic and package for BoW RX and TX slices.
As shown in Figure 1, each BoW slice consists of a differential clock pair, 16 single-ended data wires, and an optional pair of wires FEC and AUX.
FEC (Forward Error Correction) is an optional signal that allows using FEC to improve the bit error rate (BER), or may be used for redundancy for defect repair. By using an additional wire when FEC is enabled, the payload data rate is not affected and the wire data rate is unaffected. This allows F(PCLK) = F(TxClk) / 2n with FEC off or on, which simplifies the clock generation and serialization functions. If used, FEC is implemented in the Link Layer, and the PHY treats the FEC bit the same as the other data bits.
AUX is an optional signal that may be used for purposes such as Data Bus Inversion (DBI), flow control, redundancy for defect repair, etc. The Link Layers of Chiplets A and B will need to agree on the details on FEC and AUX usage. An implementation may choose to support the FEC and AUX wires, or to omit both of them. If FEC and AUX are included in a PHY implementation, the PHY carries them in the same way as the data bits without acting on the content.
Table 4 summarizes these signals.
Function | # Wires | Signal Name | Notes |
---|---|---|---|
Clock | 2 | CLK+, CLK- | Differential |
Data | 16 | D0-15 | |
Forward Error | 0/1 | FEC | Optional |
Correction | |||
Auxiliary | 0/1 | AUX | Optional |
Data Bus Inversion (DBI) may be used to mitigate simultaneous switching output (SSO) noise or to optimize energy of a BoW PHY by reducing the number of BoW data wires that switch between adjacent data transfer cycles. DBI functionality is optional; it one of several possible uses of the AUX wire. If implemented, DBI is in the Link Layer and must be implemented on both RX and TX.
Figure 3 shows the data and control signals in the interfaces to the logic in the die in each BoW transmit and receive slice. The data at the slice logic interface must be SDR (Single Data Rate - bit rate equal to the PCLK frequency).
The signals in Table 6 shall constitute the data and clocks in the logic interface of the PHY. N is the ratio of the chip-to-chip per-wire data rate to the logic interface per-wire data rate.
Signal | # Bits | TX Slice | RX Slice | Description |
---|---|---|---|---|
PD | 16*N | In | Out | Data |
PFEC | N or 0 | In | Out | Forward Error Correction (optional) |
PAUX | N or 0 | In | Out | Auxiliary uses (optional) |
PCLK | 1 | Out | Out | |
TxClk | 1 | In | NA | Comes from a PLL or other clock source, |
not the Link Layer. | ||||
The TxClk source is usually shared | ||||
among many TX slices. | ||||
May be differential | ||||
RxClk | 1 or 0 | NA | Out | May be differential |
Figure 4 shows the block diagram of a bidirectional slice. A bidirectional slice shall have one set of 18 or 20 signal bumps and wires and two sets of signals connecting to the chiplet's core logic (or Link Layer).
In mission mode, each slice must have only the PHY TX or the PHY RX enabled. For loopback test, both may be enabled. For loopback, it is recommended to enable testing one slice at a time to avoid drawing both RX and TX power of all the slices in the whole link at the same time.
The data at the slice logic interface must be SDR (Single Data Rate - bit rate equal to the TXPCLK/RXPCLK frequency).
The signals in Table 6 shall constitute the data and clocks in the logic interface of a bidirectional PHY. N is the ratio of the chip-to-chip per-wire data rate to the logic interface per-wire data rate.
Signal | # Bits | Direction | Description |
---|---|---|---|
TXD | 16*N | In | TX Data |
TXFEC | N or 0 | In | Forward Error Correction (optional) |
TXAUX | N or 0 | In | Auxiliary uses (optional) |
TXPCLK | 1 | Out | SDR clock for the TXD, TXFEC, TXAUX bits |
TXCLK | 1 | In | Comes from a PLL or other clock source, |
not the Link Layer. | |||
The TXCLK source is usually shared | |||
among many TX slices. | |||
May be differential | |||
RXD | 16*N | Out | RX Data |
RXFEC | N or 0 | Out | Forward Error Correction (optional) |
RXAUX | N or 0 | Out | Auxiliary uses (optional) |
RXPCLK | 1 | Out | SDR clock for the RXD, RXFEC, RXAUX bits |
RXCLK | 1 or 0 | Out | May be differential |
A BoW interface slice must provide the control and status signals shown in Table 7.
Signal | # Bits | TX Slice | RX Slice | Description |
---|---|---|---|---|
PHYResetB | 1 | In | In | Resets the BoW slice. |
0 causes a reset | ||||
PHYReady | 1 | Out | Out | Indicates that the PHY is ready to |
transmit/receive mission mode data. | ||||
1 indicates ready | ||||
PHYIdle | 1 or 0 | In | N/A | Optional signal |
Active high indicates to the TX slice that | ||||
it should enter the clock gated | ||||
state on the next parallel word | ||||
aligned clock edge | ||||
The PHYResetB pin shall be asserted by the link controller to initialize the PHY. While the PHYResetB signal is asserted, the PHY shall stay in its reset state. When the PHYResetB signal is de-asserted, the PHY shall perform any necessary self-alignment. The reset states are otherwise implementation-dependent and shall be documented in the datasheet of a particular implementation.
On a TX slice, the PHY shall assert PHYReady to indicate it is transmitting appropriate CLK and PCLK signals, and that it is ready to transmit data.
On an RX slice, when PHYResetB is deasserted, the PHY assumes that the corresponding TX slice is sending CLK and that the TX Link Layer is sending training data on the data wires.
After the RX slice clock self-alignments are complete, each RX PHY slice shall assert its PHYReady pin. How an RX PHY slice determines completion of the self-alignment is implementation-dependent. For instance, it may be determined by observing the settling of the DLL or by a simple timer. PHYReady asserted indicates that any data received will be captured correctly.
Further description of this optional signal and its functionality are provided in Section 11.
There shall be an AMBA APB programming interface to control internal registers for control and status readout of the PHY.
The internal registers are implementation-dependent. The internal registers shall be fully documented in the PHY datasheet.
There shall be a Link Controller (LC) outside the PHY. This will manage initialization of the Link. It may reside on one of the chiplets of the link, in a third chiplet in the package or outside the package.
Communication from the Link Controller across chiplets shall be by a transport mechanism outside the BoW link. This could be a serial link like SPI or I2C, but this is not specified at this time.
Link initialization is described in Section 14. Clocks are described in 10.2.
A BoW PHY slice must conform to at least one of the BoW Modes seen in Table 3. The recommended maximum wire reach for different packaging types and terminations is seen in Table 8. Exceeding these reach values may degrade the voltage margins at the receiver. See section 12 for how TX, RX and channels are qualified.
“Laminate” is intended to include organic laminate packages (a.k.a. “buildup”") and similar technologies with approximately 25 μm line and space rules. The minimum wire length for closely spaced chips in these technologies is around 3 mm for the slice closest to the chip edge.
“Advanced” is intended to include silicon interposer and similar technologies. These have much finer line and space dimensions, but traces are usually much more resistive than in organic laminate packages and will be limited to much shorter trace lengths. Due to these short traces, termination is not expected to be useful for implementations targeting Advanced packaging. The minimum wire length in these technologies may be less than 1 mm.
Package | Laminate | Laminate | Laminate | Advanced | ||
---|---|---|---|---|---|---|
Termination | None | Source | Double | None | ||
BoW Mode | Wire Bit Rate | TxClk | Reach | Reach | Reach | Reach |
(Gbps/wire) | (GHz) | (mm) | (mm) | (mm) | (mm) | |
BoW-32 | 2 | 1 | 10 | 20 | 25+ | 4 |
BoW-64 | 4 | 2 | NA | 10 | 25+ | 2 |
BoW-128 | 8 | 4 | NA | 5 | 25+ | 2 |
BoW-256 | 16 | 8 | NA | NA | 25+ | 2 |
BoW-384 | 24 | 12 | NA | NA | 25+ | 2 |
BoW-512 | 32 | 16 | NA | NA | 25+ | 2 |
Adding termination increases the speed and/or reach, at the expense of greater design complexity and power.
The physical diagrams and descriptions in this document must be interpreted as looking down at the top layer of the unpackaged chiplets. Since these are flip-chip packages, these views are equivalent to looking through the bottom of the package with the balls up (dead bug view). For the view as seen looking down on a package as mounted on a PCB (live bug view), these views must be mirrored.
A BoW link between two chiplets is made up of wires, slices, and stacks as seen in Figure 5.
The minimal bidirectional reference link is shown in Figure 6.