bowspec_logo




Bunch of Wires (BoW) PHY Specification

The Open Domain-Specific Architecture BoW Workstream

DRAFT Version 1.9d
January 2nd, 2023



























Glossary of Terms

This section provides glossary used in this specification.

Term Abbreviation Definition
Bunch of Wires BoW The name for the PHY specification defined in this document
Die-to-die D2D Generic term used to refer to on-package interconnect
BoW Mode N/A A specific defined mode of operation for a BoW interface
picojoules per bit pJ/bit Energy required to transport a bit of data over a D2D interface
PHY The set of circuitry physically communicating bits from one die to another
Core Logic Digital logic transmitting data to/from the PHY
Control Logic Logic used to manage the operation of the PHY
Tera/Giga bits per second Tbps/Gbps Measures of the speed of data transmission on the PHY
Beachfront The length of die edge required by a PHY implementation
Clock A signal that regulates the speed of data transmission
Bump Solder balls grown on a die to allow connection to off-die wires
Channel A term for the physical connection between a transmitter and a receiver
Test The process of verifying functional correctness of a circuit
Initialization The process of preparing an interface for data transmission
Table 1. Glossary of Terms

Language

This document uses the following terms as defined below.

1. License Agreement

1.1. Open Web Foundation (OWF) CLA

Contributions to this Specification are made under the terms and conditions set forth in the modified Open Web Foundation Contributor License Agreement (“OWF CLA 1.0”) (“Contribution License”) by:

ANALOG PORT, BLUE CHEETAH ANALOG DESIGN, D-MATRIX, IBM, KEYSIGHT, TESSOLVE, VENTANA MICRO

You can review the signed copies of the applicable Contributor License(s) for this Specification on the OCP website at http://www.opencompute.org/products/specsanddesign

Usage of this Specification is governed by the terms and conditions set forth in the modified Open Web Foundation Final Specification Agreement (“OWFa 1.0”) (“Specification License”).

Notes:

  1. The above license does not apply to the Appendix or Appendices. The information in the Appendix or Appendices is for reference only and non-normative in nature.

NOTWITHSTANDING THE FOREGOING LICENSES, THIS SPECIFICATION IS PROVIDED BY OCP “AS IS” AND OCP EXPRESSLY DISCLAIMS ANY WARRANTIES (EXPRESS, IMPLIED, OR OTHERWISE), INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, FITNESS FOR A PARTICULAR PURPOSE, OR TITLE, RELATED TO THE SPECIFICATION. NOTICE IS HEREBY GIVEN, THAT OTHER RIGHTS NOT GRANTED AS SET FORTH ABOVE, INCLUDING WITHOUT LIMITATION, RIGHTS OF THIRD PARTIES WHO DID NOT EXECUTE THE ABOVE LICENSES, MAY BE IMPLICATED BY THE IMPLEMENTATION OF OR COMPLIANCE WITH THIS SPECIFICATION. OCP IS NOT RESPONSIBLE FOR IDENTIFYING RIGHTS FOR WHICH A LICENSE MAY BE REQUIRED IN ORDER TO IMPLEMENT THIS SPECIFICATION. THE ENTIRE RISK AS TO IMPLEMENTING OR OTHERWISE USING THE SPECIFICATION IS ASSUMED BY YOU. IN NO EVENT WILL OCP BE LIABLE TO YOU FOR ANY MONETARY DAMAGES WITH RESPECT TO ANY CLAIMS RELATED TO, OR ARISING OUT OF YOUR USE OF THIS SPECIFICATION, INCLUDING BUT NOT LIMITED TO ANY LIABILITY FOR LOST PROFITS OR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT, SPECIAL OR PUNITIVE DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF ACTION OF ANY KIND WITH RESPECT TO THIS SPECIFICATION, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND EVEN IF OCP HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

2. OCP Tenets Compliance

The Bunch of Wires (BoW) specification defines a versatile, open and interoperable physical interface between two chiplets or chip-scale-packages (CSP) in a common package, and is fully backwards compatible with the Bunch of Wires specification. This document specifies the BoW interface PHY layer, and defines a set of die-to-die parallel interfaces that implementers / adopters the the flexibility to trade off throughput/chipedge for design complexity, cost, and packaging technology. The use of BoW is expected to be confined to connect die placed close to one another within the same package. In this environment, signal attenuation is small and the interface can be simple. The definition of the BoW interface aims to meet the following OCP tenets as follows:

2.1. Openness

2.2. Efficiency

2.3. Scale

2.4. Impact

The BoW specification provides several key advantages for chiplet-based systems:

Compared to SerDes, BoW uses a lower data rate/wire so it requires more wires. However the lower data rates allow use of single-ended signaling and denser wire packing. In addition, in laminates, BoW can take advantage of multiple wiring layers and in advanced packaging it can take advantage of the much-increased wire density.

3. Revision Table

Date Revision Author Description
07/26/22 Draft 1.1a Elad Alon Initial updates from 1.0 to 1.1
09/30/22 Draft 1.1b Marek Hempel, Half-slice and power management added
Elad Alon
10/26/22 Draft 1.1c Ken Poulton, Configurable directionality, redundancy
Shahab Ardalan added
Elad Alon
1/2/23 Draft 1.1d Elad Alon Updated timing requirements, sideband
slice definition added
1/18/23 Draft 1.9 Bapi Vinnakota Updated ODSA overview

4. Scope

The scope of this document has several levels.

  1. The specification of the BoW interface includes these requirements:

    1. Operating modes
    2. Chip-to-chip wire signals
    3. Wire ordering
    4. Timing and electrical specifications on the chip-to-chip interface
    5. Signals at the logic (Link Layer) interface
    6. Configuration, initialization, calibration
    7. Functions that must be supported at the Link Layer or above

  2. The specification includes recommendations for these elements:

    1. Bump patterns
    2. Arrangement of multiple slices in a link
    3. Arrangement of wires in laminate and advanced packaging
    4. signal integrity of the wire channel
    5. Configuration and management programming
    6. Design for test and test methods
    7. Performance estimates
    8. Conformance verification

  3. The following activities are outside the scope of this document:

    1. Specific implementations of the interface
    2. Integration of the interface with system-level data flow
    3. The use of this interface outside of a package or entirely inside a chip
    4. Definition of protocols for logical data transfer

  4. The following aspects may be addressed in subsequent versions of this specification:

    1. Simultaneous bidirectional data (full duplex on each wire)
    2. Security

5. BoW Overview

This section provides an overview of the BoW physical interface (PHY) and its use in a multi-chiplet design.

5.1. Key Features and Conformance

The specifications must be met over process variation, supply voltage range and temperature range (PVT). Each implementation must document its supported I/O voltage range, supply voltage range and temperature range.

Table 2 summarize the conformance points that shall be met in order to comply with the BoW specification. Each of the conformance points is discussed in the specification.

Description Section Detail
BoW Modes  5.4
Die-to-die Signals (Wires)  6.2
Slice Logic Interface  6.4.1
BoWx Modes and Reach  7 Table 8
Wire and Slice Ordering  8
Voltages and Termination Resistance  9.1
PHY Protection  9.2
ESD  9.3
Return Loss and Parasitic Capacitance  9.4
Clocking  10.2
Clock and Data Specs  10
Channel Skew  12.1
External Facilities  14.1
Initialization  14.2
Control Register Mapping  16
Table 2. BoW Conformance Summary TO BE UPDATED

5.2. BoW Slice

BoW is an energy-efficient, easy-to-use PHY interface between a pair of die inside a single package as shown in Figure 1. The BoW PHY is defined as a single unidirectional slice. Multiple slices are combined to create links of the desired throughput. A link may be symmetric, asymmetric or unidirectional. The BoW PHYs between two die are physically connected through wires on a substrate or interposer. A BoW PHY does not have enough drive strength for off-package interfaces, nor is it designed for buses that are entirely on die.

This document specifies the protocol for a BoW PHY slice. The aggregation of multiple PHYs into a link is beyond the scope of this document.

bow_overview2

Figure 1. BoW Overview

A BoW PHY slice either transmits or receives 16 bits of data between die. The BoW is a source-synchronous PHY and each transmitting PHY slice transmits a complementary clock signal CLK+ and CLK- with the data. A BoW PHY optionally has two additional wires designated FEC (for Forward Error Correction) and AUX, for other optional functions such as Data Bus Inversion (DBI).

5.3. BoW Wires

Within the package, the BoW datapath is transported on physical passive wires between the pair of connected die. The specifics of the wires, such as their density, maximum length, impedance characteristics and how they are realized vary with the packaging technology. In order to minimize power, unterminated and source-terminated links will have short reaches requiring chips to be adjacent.

5.4. BoW Modes

A BoW PHY must be operable in one of the BoW Modes listed in ascending order in Table 3. A BoW Mode defines the speed of clock and data of the PHY on the die-to-die wires. In all modes, the data must be clocked DDR: the chip-to-chip data wire bit rate is double the clock wire frequency. All BoW interfaces faster than BoW-64 should also be able to support BoW-64. Supporting rates other than the defined modes is an implementation choice. There is more detail on BoW Modes in section 7.

BoW Mode Slice Data Rate Wire Bit Rate TxClk
Gbps Gbps/wire GHz
BoW-32 32 2 1
BoW-64 64 4 2
BoW-128 128 8 4
BoW-256 256 16 8
BoW-384 384 24 12
BoW-512 512 32 16
Table 3. BoW Modes

bowspec_figtradeoff2

Figure 2. BoW Data Rate vs. Reach tradeoff TO BE UPDATED

Figure 2 shows the tradeoff between package, data rate, termination, and reach. Source-terminated BoW on laminate allows a longer reach than advanced packaging, but the wider design rules in laminate means that both of these cases are barely able to reach 8 Gbps/wire. A doubly-terminated link offers longer distances and higher rates, but requires a more complicated receiver design.

5.5. Logic Interface

Figure 3 shows the logic interface between a BoW slice and the digital Link Layer logic in a chip. The speed at the logic interface (Figure 1) is implementation-dependent. Typically, PCLK will be the TxClk frequency divided by a power of 2, so 250, 500 and 1000 MHz are common rates. The data at the logic interface is SDR (bit rate equal to PCLK frequency).

bowspec_figslicelogicint

Figure 3. BoW slice logic interface

6. Signal Definitions

This section specifies the control data signals into and out of device logic and package for BoW RX and TX slices.

6.1. Directionality

  • A BoW link may be bidirectional (each side contains both RX and TX slices) or unidirectional (each chiplet's side contains only TX or only RX slices)
  • Each BoW PHY slice is unidirectional when in operation
  • A BoW PHY slice is said to be bidirectional if it can be configured to operate in TX or in RX
  • This spec does not include full-duplex operation (simultaneous TX and RX over the same wires)

6.2. Die-to-die Signals (Wires)

As shown in Figure 1, each BoW slice consists of a differential clock pair, 16 single-ended data wires, and an optional pair of wires FEC and AUX.

FEC (Forward Error Correction) is an optional signal that allows using FEC to improve the bit error rate (BER), or may be used for redundancy for defect repair. By using an additional wire when FEC is enabled, the payload data rate is not affected and the wire data rate is unaffected. This allows F(PCLK) = F(TxClk) / 2n with FEC off or on, which simplifies the clock generation and serialization functions. If used, FEC is implemented in the Link Layer, and the PHY treats the FEC bit the same as the other data bits.

AUX is an optional signal that may be used for purposes such as Data Bus Inversion (DBI), flow control, redundancy for defect repair, etc. The Link Layers of Chiplets A and B will need to agree on the details on FEC and AUX usage. An implementation may choose to support the FEC and AUX wires, or to omit both of them. If FEC and AUX are included in a PHY implementation, the PHY carries them in the same way as the data bits without acting on the content.

Table 4 summarizes these signals.

Function # Wires Signal Name Notes
Clock 2 CLK+, CLK- Differential
Data 16 D0-15
Forward Error 0/1 FEC Optional
Correction
Auxiliary 0/1 AUX Optional
Table 4. BoW Signals at the Die To Die Interface

6.2.1. DBI on the AUX wire

Data Bus Inversion (DBI) may be used to mitigate simultaneous switching output (SSO) noise or to optimize energy of a BoW PHY by reducing the number of BoW data wires that switch between adjacent data transfer cycles. DBI functionality is optional; it one of several possible uses of the AUX wire. If implemented, DBI is in the Link Layer and must be implemented on both RX and TX.

Figure 3 shows the data and control signals in the interfaces to the logic in the die in each BoW transmit and receive slice. The data at the slice logic interface must be SDR (Single Data Rate - bit rate equal to the PCLK frequency).

6.3.1. Slice Logic Interface: Clock and Data Signals

The signals in Table 6 shall constitute the data and clocks in the logic interface of the PHY. N is the ratio of the chip-to-chip per-wire data rate to the logic interface per-wire data rate.

Signal # Bits TX Slice RX Slice Description
PD 16*N In Out Data
PFEC N or 0 In Out Forward Error Correction (optional)
PAUX N or 0 In Out Auxiliary uses (optional)
PCLK 1 Out Out
TxClk 1 In NA Comes from a PLL or other clock source,
not the Link Layer.
The TxClk source is usually shared
among many TX slices.
May be differential
RxClk 1 or 0 NA Out May be differential
Table 5. Logic Interface Signals

bidi-block

Figure 4. BoW Bidirectional slice block diagram

Figure 4 shows the block diagram of a bidirectional slice. A bidirectional slice shall have one set of 18 or 20 signal bumps and wires and two sets of signals connecting to the chiplet's core logic (or Link Layer).

In mission mode, each slice must have only the PHY TX or the PHY RX enabled. For loopback test, both may be enabled. For loopback, it is recommended to enable testing one slice at a time to avoid drawing both RX and TX power of all the slices in the whole link at the same time.

The data at the slice logic interface must be SDR (Single Data Rate - bit rate equal to the TXPCLK/RXPCLK frequency).

6.4.1. Slice Logic Interface: Clock and Data Signals

The signals in Table 6 shall constitute the data and clocks in the logic interface of a bidirectional PHY. N is the ratio of the chip-to-chip per-wire data rate to the logic interface per-wire data rate.

Signal # Bits Direction Description
TXD 16*N In TX Data
TXFEC N or 0 In Forward Error Correction (optional)
TXAUX N or 0 In Auxiliary uses (optional)
TXPCLK 1 Out SDR clock for the TXD, TXFEC, TXAUX bits
TXCLK 1 In Comes from a PLL or other clock source,
not the Link Layer.
The TXCLK source is usually shared
among many TX slices.
May be differential
RXD 16*N Out RX Data
RXFEC N or 0 Out Forward Error Correction (optional)
RXAUX N or 0 Out Auxiliary uses (optional)
RXPCLK 1 Out SDR clock for the RXD, RXFEC, RXAUX bits
RXCLK 1 or 0 Out May be differential
Table 6. Bidirectional Logic Interface Signals

6.5.1. Slice Logic Interface: Control Signals

A BoW interface slice must provide the control and status signals shown in Table 7.

Signal # Bits TX Slice RX Slice Description
PHYResetB 1 In In Resets the BoW slice.
0 causes a reset
PHYReady 1 Out Out Indicates that the PHY is ready to
transmit/receive mission mode data.
1 indicates ready
PHYIdle 1 or 0 In N/A Optional signal
Active high indicates to the TX slice that
it should enter the clock gated
state on the next parallel word
aligned clock edge
Table 7. Logic Interface Control Signals
6.5.1.1. PHYResetB TX and RX

The PHYResetB pin shall be asserted by the link controller to initialize the PHY. While the PHYResetB signal is asserted, the PHY shall stay in its reset state. When the PHYResetB signal is de-asserted, the PHY shall perform any necessary self-alignment. The reset states are otherwise implementation-dependent and shall be documented in the datasheet of a particular implementation.

6.5.1.2. PHYReady TX

On a TX slice, the PHY shall assert PHYReady to indicate it is transmitting appropriate CLK and PCLK signals, and that it is ready to transmit data.

6.5.1.3. PHYReady RX

On an RX slice, when PHYResetB is deasserted, the PHY assumes that the corresponding TX slice is sending CLK and that the TX Link Layer is sending training data on the data wires.

After the RX slice clock self-alignments are complete, each RX PHY slice shall assert its PHYReady pin. How an RX PHY slice determines completion of the self-alignment is implementation-dependent. For instance, it may be determined by observing the settling of the DLL or by a simple timer. PHYReady asserted indicates that any data received will be captured correctly.

6.5.1.4. PHYIdle TX

Further description of this optional signal and its functionality are provided in Section 11.

6.5.2. Programming

There shall be an AMBA APB programming interface to control internal registers for control and status readout of the PHY.

The internal registers are implementation-dependent. The internal registers shall be fully documented in the PHY datasheet.

There shall be a Link Controller (LC) outside the PHY. This will manage initialization of the Link. It may reside on one of the chiplets of the link, in a third chiplet in the package or outside the package.

Communication from the Link Controller across chiplets shall be by a transport mechanism outside the BoW link. This could be a serial link like SPI or I2C, but this is not specified at this time.

Link initialization is described in Section 14. Clocks are described in 10.2.

7. BoW Modes and Reach

A BoW PHY slice must conform to at least one of the BoW Modes seen in Table 3. The recommended maximum wire reach for different packaging types and terminations is seen in Table 8. Exceeding these reach values may degrade the voltage margins at the receiver. See section 12 for how TX, RX and channels are qualified.

“Laminate” is intended to include organic laminate packages (a.k.a. “buildup”") and similar technologies with approximately 25 μm line and space rules. The minimum wire length for closely spaced chips in these technologies is around 3 mm for the slice closest to the chip edge.

“Advanced” is intended to include silicon interposer and similar technologies. These have much finer line and space dimensions, but traces are usually much more resistive than in organic laminate packages and will be limited to much shorter trace lengths. Due to these short traces, termination is not expected to be useful for implementations targeting Advanced packaging. The minimum wire length in these technologies may be less than 1 mm.

Package Laminate Laminate Laminate Advanced
Termination None Source Double None
BoW Mode Wire Bit Rate TxClk Reach Reach Reach Reach
(Gbps/wire) (GHz) (mm) (mm) (mm) (mm)
BoW-32 2 1 10 20 25+ 4
BoW-64 4 2 NA 10 25+ 2
BoW-128 8 4 NA 5 25+ 2
BoW-256 16 8 NA NA 25+ 2
BoW-384 24 12 NA NA 25+ 2
BoW-512 32 16 NA NA 25+ 2
Table 8. Recommended BoW Wire Reaches

Adding termination increases the speed and/or reach, at the expense of greater design complexity and power.

8. BoW Physical Configuration

8.1. Dead-Bug Views

The physical diagrams and descriptions in this document must be interpreted as looking down at the top layer of the unpackaged chiplets. Since these are flip-chip packages, these views are equivalent to looking through the bottom of the package with the balls up (dead bug view). For the view as seen looking down on a package as mounted on a PCB (live bug view), these views must be mirrored.

8.2. BoW Components

components

Figure 5. BoW Link Components

A BoW link between two chiplets is made up of wires, slices, and stacks as seen in Figure 5.

  • The signal traces in the package between chiplets are called wires.
  • A slice is the the basic unit of a BoW PHY. It must have 18 or 20 signal bumps. It must have 2 bumps for the differential clock and 16 single-ended data bumps. It may also have the optional single-ended signals AUX and FEC. The long edge of a slice must be parallel to the chip edge.
  • A stack is composed of one or more slices stacked from the chip edge towards the center. The slice positions are designated A, B, C, etc, starting with the slice closest to the edge of the chip.
  • A link from one chiplet to another is composed of one or more stacks placed along the chip edge. A link may be configured with equal numbers of RX and TX slices, or it may be asymmetric or one-way.

The minimal bidirectional reference link is shown in Figure 6.