VIA meets SCI

The following figure illustrates the general architecture of the board.

Now let's step through the subsequent components:

• DEC21554 PCI-PCI Bridge

  Although this bridge is located along with the critical data path and thus increases latency somewhat it makes a lot of things more easy to realize.

  The first thing here is the handling of the Flash EEPROM. The DEC Bridge has an interface to such component where it's possible to integrate a boot ROM with minor additional glue logic. Because there's anyways such ROM needed for FPGA initialization, the boot ROM interface can be used to update the FPGA initialization stream. The lower 256kB of the Flash are intended for boot code while the upper 256kB are needed for FPGA initialization.

  Another purpose of the DEC Bridge is simplification of the PCI design. Because this bridge is a so-called non-transparent PCI-PCI bridge it is actually not a PCI-PCI bridge following the PCI Specification for such bridges. Therefore the DEC21554 is visible as a normal PCI device offering only several address windows. This fact eliminates the need for implementation of a PCI configuration space inside the PCI FPGA.
  The 21554 is also able to make a direct offset translation of primary PCI addresses into secondary PCI addresses. This makes it possible to use hard-coded address decoders inside the PCI FPGA rather than a comparator as it is normally required for PCI.

• PCI FPGA

  The PCI FPGA is maybe the most important unit of the whole design because it realizes substantial functions. It is based on an ORCA 3T80 FPGA manufactured by Lucent Technologies. Basic functions of the PCI FPGA are:

  • Raw access to Dual Ported Memory (DPM), static RAM, and several Control and Status Registers used for miscellaneous control.
  • Translation of PCI accesses into SCI transactions. This is also known as Transparent Mode that is required to access memory of remote nodes using simple load/store instructions.
  • Translation of incoming SCI transactions into PCI transactions using a PCI Master. Of course, this is also an essential function to realize a distributed shared memory as well as other data transfers such as remote DMA (RDMA).
  • Handling of Doorbell accesses. Since the hardware provides multiple virtual hardware instances used directly by user processes, there are several control and status registers required that may be read or written by the user process without the need to make operating system kernel calls.
  • A DMA engine that executes descriptors posted by user processes. This DMA engine automatically approves the right of the user process whether it is allowed to access a certain local or remote memory page or not.
    This DMA Engine will provide the most powerful data transfer with a minimal CPU utilization.

  Details of the PCI FPGA are quite complicated. There's even a small highly specialized processor integrated to not loose to overview about the design. PCI FPGA internals will be discussed in a separate page later.

• SCI FPGA

  The SCI FPGA is based on the same chip as the PCI FPGA (ORCA 3T80). The functions of the SCI FPGA are concentrated on SCI packet handling. That is, it is responsible for receiving and sending of SCI packets from/to the SCI Link Controller. Also, the SCI FPGA is responsible to detect problems during packet transmission (such as timeouts) and to take some appropriate actions (such as packet retransmission) in these cases.
  The SCI FPGA is able to support for up to 64 outstanding SCI transactions and up to 63 (maybe also 127) incoming packets.

  Below you can see the block diagram of the SCI FPGA. To view the figure at a higher resolution (recommended) please click on it!

  

• Dual Ported Memory

  The primary function of the Dual Ported Memory (DPM) is storage of SCI packets or rather the data body of SCI packets. Because of the two completely asynchronous and independent ports of this buffer memory it is possible to receive a new SCI packet while data of a formerly received SCI packet is written into PCI memory, for example.

  As mentioned previously, actually only the data body of SCI packets have to be stored in the DPM. Other informations (such as address and status) are exchanged between both FPGAs using a different path (see figure). However, because of some constructive reasons whole SCI packets (at least incoming ones) are stored in the DPM for now.

• Static Memory

  This additional memory contains three large tables:

  • Downstream Address Translation and Protection Table. This table is used to generate global SCI addresses for transparent transactions and RDMA operations. In other words, the downstream address translation table realizes a virtual view onto imported SCI memory. Every entry in this table contains some other control informations and a protection tag as specified by VIA. Although an original Virtual Interface Architecture device has no downstream address translation table it is needed in our design to realize a distributed shared memory. In order to guarantee a protected RDMA controlled from user-level the hardware must check if the user process is allowed to access an imported page or not. For transparent transactions the protection tag can be ignored.
  • Upstream Address Translation and Protection Table. Such translation table is required by VIA, but is completely new for a PCI-SCI design. It allows a virtual view onto the local hosts' memory. Each entry in this table can point to any local physical memory page. This allows that any arbitrary physical memory page can be made accessible for the PCI-SCI bridge. As conclusion, every arbitrary page can also be made available in the global SCI address space and therefore exported to other nodes. Similar as in case of the Downstream Address Translation table each entry is protected by a protection tag.
  • VI Context Memory. For each supported virtual interface (VI) there are several informations to keep in mind. For instance what is the protection tag of this VI, where is the next DMA descriptor to be executed, etc. All these informations are stored inside the VI context memory. Some of these informations will be indirectly accessible for user processes via the Doorbell mechanism, others will be accessible by the VIA kernel agent only.

  The basic SRAM size of 1MByte supports 80kEntries for the downstream address translation table (64bits/entry), 32kEntries for the upstream address translation table (32bits/entry), and 1kEntries for the VI context memory (256Bytes/entry).
  Assuming a page size of 4kBytes for exported and imported memory pages this results in 320MBytes of importable memory and 128MBytes of exportable memory.

• Dolphin LC2 - SCI Link Controller

  The LC-2 made by Dolphin Interconnect Solutions realizes the timing critical functions of the high-speed SCI connections. Actually the LC-2 is able to achive a link bandwidth of 500MByte/s (cumulative bandwidth of 1GByte/s since data is transferred on link-in and link-out in parallel). However, for now we tested it only with a link frequency of 100MHz. That is, the bandwidth of one link is 'only' 400MByte/s.