Sparsr Memory Map

This page lists every memory on a Sparsr card, with its address and size. It also says which instructions can address each one. It describes the planned architecture.

What runs today. The Sparsr VM in the SDK has this address map, 64 KiB of IMEM and DMEM, the WMEM byte window, the DMA registers and XMEM. Through the host library it has the card's 4,096 WMEM rows, and a smaller XMEM, as section 6 shows. The FPGA hardware has 4 KiB of IMEM and DMEM and a small row memory today, and XMEM, the DMA and the byte window are still being built.

1. The memories

Memory	Physical memory	Size	Unit	Reached by	Contents
Scalar registers	on-chip	32 x 32 bits	register	RV32I instructions	scalar values
Wide registers	on-chip	32 x 8,192 bits	register	wide instructions	raw rows
IMEM, instruction memory	on-chip BRAM	64 KiB	byte	the core's fetch	the kernel's code
DMEM, data memory	on-chip BRAM	64 KiB	byte	RV32I loads and stores	stack, globals, tables
WMEM, wide memory	on-chip URAM	4,096 rows, 4 MiB	row	wide load and store by row number; RV32I loads and stores by byte; the DMA	raw rows, always
XMEM, extended memory	high-bandwidth memory next to the FPGA	16 GiB	row	the DMA and the host only	rows as the host sent them, raw or LIL
DMA registers	on-chip	24 bytes	word	RV32I loads and stores, and the host	copy requests and status
Host RAM	the host computer	the host's	byte	the host CPU only	anything

The table gives these rules:

A wide instruction addresses WMEM and nothing else. No instruction addresses XMEM. A kernel gets a row from XMEM by asking the DMA to copy it into WMEM.
The scalar core has one address space. IMEM, DMEM, the WMEM byte window and the DMA registers are all in it. WMEM is the one memory that has both row numbers and byte addresses.
Rows in WMEM are always raw. A row changes form only on the path into WMEM, where the card's one decoder expands a compressed row.

The host's own RAM is never visible from the card. The host library moves data across PCIe in frames. See Host Wire Format.

2. The address map

A kernel sees one flat, byte-addressed, little-endian 32-bit address space. Each memory has a window of its own. Only the start of each window is implemented, and the rest is reserved so that the base addresses never move.

Address range	Region	What a kernel may do there
`0x0000_0000` to `0x0FFF_FFFF`	IMEM	Fetch instructions. A load or store here is a fatal fault.
`0x1000_0000` to `0x7FFF_FFFF`	Unmapped	Nothing. Every access is a fatal fault.
`0x8000_0000` to `0x8FFF_FFFF`	DMEM	Load and store. A fetch from here is a fatal fault.
`0x9000_0000` to `0x9FFF_FFFF`	WMEM byte window	Load and store on WMEM rows. A fetch from here is a fatal fault.
`0xA000_0000` to `0xFFDF_FFFF`	Unmapped	Nothing. Every access is a fatal fault.
`0xFFE0_0000` to `0xFFFF_FFFF`	Memory-mapped I/O	The DMA registers, at `0xFFE0_0000`. Any other address is a fatal fault.

XMEM has no address in this map.

The map gives these rules:

A data access to IMEM faults. So *(int *)0 stops the batch with a fault. It does not return your first instruction.
A fetch from DMEM, WMEM or the I/O window faults. Data never runs as code.
An access past the implemented part of a window faults. A window is 256 MiB, and only the start of it exists.

A fatal fault ends the batch and tells the host why. The core does not continue past it.

3. IMEM and DMEM

IMEM holds the kernel, and DMEM holds its stack, globals and scalar data. Each is 64 KiB of on-chip memory.

The host loads the kernel image into IMEM, and the core starts at address 0.
A C kernel never writes DMEM addresses by hand. The SDK's link script places the stack and the globals.
A kernel reads and writes DMEM with the ordinary RV32I loads and stores, at 0x8000_0000 plus a byte offset.
DMEM is a good place for run-time arguments, such as the row numbers a kernel should work on. Then one kernel, loaded once, serves every call.

DMEM is one memory with one speed. The host does not manage it in tiers.

4. The WMEM byte window

A kernel can read and write the bytes of a WMEM row with ordinary RV32I loads and stores. Byte k of row n is at this address:

0x9000_0000 + n * 1,024 + k

Rows are little-endian, so lane 0 of a row is the row's first four bytes.
The window covers WMEM's 4,096 rows, which is 4 MiB of addresses. An address past the last row faults.
The byte window and the wide load and store reach the same rows. A byte stored through the window is in the row the next wide load reads.

The main use is keeping a result beside the data. A reduction such as a population count leaves a 32-bit number in a scalar register. The kernel stores that number in a WMEM row, and the result stays with the rows it describes.

The SDK header gives a pointer to a row's lanes:

uint32_t *_sparsr_wmem_row(uint32_t row);   /* SPARSR_WMEM_BASE + row * SPARSR_WIDE_BYTES */

SPARSR_WMEM_BASE is 0x90000000u. The link script also exports the window as __wmem_base and __wmem_length.

5. The DMA registers

The DMA is the engine that copies rows between XMEM and WMEM. A kernel and the host drive it through the same six registers, at the start of the I/O window.

Address	Register	What it holds
`0xFFE0_0000`	Source row	The first row to copy from
`0xFFE0_0004`	Destination row	The first row to copy to
`0xFFE0_0008`	Count	How many rows to copy
`0xFFE0_000C`	Control	Bit 0 starts the request. Bit 1 is the direction: clear copies XMEM into WMEM, set copies WMEM into XMEM.
`0xFFE0_0010`	Status	Bit 0 is done, bit 1 is busy, and bits 8 to 15 hold the fault of the last request. Read-only.
`0xFFE0_0014`	Rows done	How many rows the last request copied. Read-only.

Every access is a 32-bit, word-aligned load or store. A byte or half-word access is a fatal fault, and so is a store to a read-only register.
The SDK header defines these addresses as SPARSR_DMA_SRC to SPARSR_DMA_ROWS_DONE, and the link script exports the block as __dma_base and __dma_length.

See WMEM, XMEM and the DMA for how a copy behaves.

6. Sizes on each device

	Planned architecture	Sparsr VM, through the host library	Today's FPGA hardware
Wide register width	8,192 bits	4,096 bits, or 8,192 with `SPARSR_VM_WIDE_BITS=8192`	8,192 bits
IMEM	64 KiB	64 KiB	4 KiB
DMEM	64 KiB	64 KiB	4 KiB
WMEM	4,096 rows	4,096 rows	a small row memory
WMEM byte window	yes	yes	not yet
XMEM	16 GiB	1,024 rows	not yet
DMA	yes	yes	not yet

XMEM through the host library stays at 1,024 rows, on purpose. The host library keeps a record of every XMEM row it can hand out, 40 bytes each. The full 16 GiB would be 16.8 million rows, and the records alone would take 671 MB of the host's memory. A program that needs a larger off-chip tier in the VM builds its own device with more XMEM rows.

A row's bytes move when the width changes. Row 0 starts at 0x9000_0000 at every width, and every other row starts at n times the width in bytes. Build a kernel for the width of the device it runs on. SPARSR_WIDE_BITS, SPARSR_WIDE_BYTES and SPARSR_WIDE_LANES in the SDK header say which width a kernel is built for.