Architecture on When Moore's Law Ends

IO Coherence

Mon, 18 May 2020 00:00:00 +0000

Cache coherence between CPUs are most explained in textbooks, but IO coherence is not well understood. Recently I’m involved in architecture discussion about IO coherence, and found this paper, “Maintaining I/O Data Coherence in Embedded Multicore Systems” by Thomas B. Berg 2009, very useful coming to explain what is IO coherence and how to implement it in embedded system.

I/O Coherence#

Producer-consumer model#

Most mechanisms for passing data between IO device and CPU, either CPU -> IO or IO -> CPU, use the classic producer-cosumer model.

ARM AMBA Protocl

Mon, 25 Mar 2019 00:00:00 +0000

AMBA (ARM Advanced Microcontroller Bus Architecture)

1. AXI#

AXI protocol is a point-to-point protocol
- So no matter what the network channels really use, as long as its ports comply AXI protocol, IP can be connected to them
Main features
- Separate read/write channels
  - Improve bandwidth
- Multiple outstanding requests
- No strict timing relationship between address and data phases
- Unaligned data transfer
- Out-of-order transaction completion
- Implicitly incremental address for burst transfer
Transfer vs transaction
- Transfer is a handshake: valid/ready
- Transaction is composed of multiple transfers
  - Active transaction: already started but not finished.
Detail signals
- AxBURST [1:0]
  - 00 = FIXED (same address) = for FIFOs
  - 01 = INCR = for block transfer
  - 10 = WRAP = suitable for cache line, critical word first
    - Must be aligned
  - 11 = reserved
- AxPROT [2:0] defines 3 levels of access protection privileges
  - Bit 0: Privileged or not
  - Bit 1: Secure or not
  - Bit 2 : instruction or not (hint only)
- AxCACHE [3:0]
  - Bit 0: bufferable or not
  - Bit 1: modifiable or not
    - Merge-able or split-able
  - Bit 2: read allocate (hint only)
  - Bit 3: write allocate (hint only)
    - If both RA and WA are disabled, then can skip cache and directly pass to the memory controller for access
- xRESP [1:0]
  - For read, each transfer of the burst has a RRESP
  - For write, at completion of the burst BRESP is issued
  - 00 = normal access success = excluesive access failed
- WSTRB [n-1:0]: byte valid
- AxLOCK for atomic access
  - Lock access is removed from AXI3 to AXI4
  - In AXI4 only exclusive is supported for better network fabric performance, to enable the implementation of semaphore type operations without requiring the bus to remain locked
- AxQOS [3:0] defines the priority of a transaction, larger number means higher priority
  - Priority is guaranteed by arbiters
- AxREGION [3:0] for a single physical slave that provides multiple logical interfaces in different address region of the whole memory space
- AxUSER implemenation defined width, optional. So can be imcompatible
- Dependency
  - WVALID can assert before AWVALID
  - WLAST must complete before BVALID
  - RVALID cannot be asserted until ARADDR has been transferred
Atomic access
- AXI3 has “locked access” which blocks all other masters to locked slave, while AXI4 impoved it to “exclusive access” which only blocks access to particular region.
- Locked transaction
  - Ensure no outstanding transaction
  - Initial lock transfer, complete with an unlocked transfer
  - Performance impact is huge
  - Lock access is enforced by network fabric
- Exclusive access
  - Semaphore style READ & WRITE
  - Requires slave hardware support
    - “Exclusive access monitor” to save exclusive operation source ID and target memory address when exclusive READ access happens, remove entry when exclusive WRITE access happens
  - xRESP
    - EXOKAY means successful, but OKAY means exclusive fail
    - When exclusive write reply with OKAY, the memory won’t get updated
Ordering
- Write
  - W channel must follow the same order of AW channel
  - Different transaction IDs on W can be interleaved, but same ID must in order even for different transactions
- Read
  - No ordering between R and AR
  - Different transaction IDs on R can be interleaved, but same ID must in order
- Read and write don’t have ordering between each other
Alignment
- Unaligned start address only affects the first transfer in a transaction, all following transfer is aligned to AxSIZE to relax requirement on slave side
Interface attributes
- Write
  - Issuing capability: a master can issue how many outstanding transactions
  - Interleave depth: a slave can receive how many outstanding trasactions
- Read
  - Issuing capability: how many transactions a master can issue
  - Acceptance capability: how many transactions a slave can accept
  - Reordering depth: how many transactions a slave can transmit data

ARM Training Cortex Processor Behaviors

Mon, 18 Mar 2019 00:00:00 +0000

Simple Sequential Execution Model#

After optimization, the result should be exactly the same with “simple sequential execution model”.

Optimization: Instruction Fetching#

Fetch multiple instructions from memory
Branch
- Predictive fetch and execution
- Branch caches
- Return stack
  - The “link register” value is pushed into the stack, and used/popped when return
  - E.g. 4 entries of return stack, but my call stack is 5, then miss will happend. Then what to do? Discard all of the content probably will be the most obvious answer.
- Conditional branch prediction
  - Shift register of branch execution history, taken is 1’b1 and non-taken is 1’b0
  - Use the pattern to predict the next branch
  - Use branch monitor’s result to update this prediction table
- Misprediction
  - Predict into non-executable memory region, MMU needs to react
  - Some architecures require control bits to disable prediction

ARMv8 Architecture

Mon, 18 Mar 2019 00:00:00 +0000

ARM online training note

1. Introduction#

What is an architecture?#

Instruction set
Exception model
Memory model
Debug

ARMv8#

AArch32 vs AArch64
- AArch32: backward compatible to ARMv7
- AArch64: fixed 32-bit instruction, new exception model, 64-bit virtual address

Priviledge and security model#

4-level of privilege
- EL0 < EL1 < EL2 < EL3, larger the higher privilege
2 security modes

Mixture of AArch32 and AArch64#

Only 64-bit OS can host a mix of 32-bit and 64-bit apps
32-bit app can only be on lower EL level

2. ISA#

Register#

X0 to X30: 31 general purpose registers
- W0 to W30 are their 32-bit form
- Zero register: XZR and WZR
V0 to V31: floating point, SIMD, crypto operations
- Multiple view: B(8), H(16), S(32), D(64), Q(128)
System registers
- MSR / MRS: move from/to system register to/from generator purpose register

Data processing#

Flow control#

ARM also have implicitly flag registers that are results of comparisons

PCS (proceduare call standard)#

Parameter pass in: X0 - X7
Return value: X0 - X1
Must preserve: X19 - X29
Can corrupt: X0 - X18
Return address (LR): X30

Load and store#

LDR / STR W0, [X1, #12] (X1 is not changed)
- Pre-index: [X1, #12]! (X1 is changed then used)
- Post-index: [X1], #12 (X1 is used, then changed)

Floating point#

SIMD#

Lane = whole Vx register & element
Neon

Vectors#

Vn.xy
- n = register number
- x = number of elements
- y = size of the elements (B/H/S/D/Q)
Total vector length = 128-bit / 64-bit for instructions to work on a whole vector
Special instructions work on individual elements

3. Exception#

Synchronous = exception
Asynchronous = interrupt

Exception level (EL)#

PSTATE <=> SPSR#

PSTATE is the current state of the processor, and SPSR is the registers to save the PSTATE.

Interrupts and ARM GIC Architecture

Mon, 11 Mar 2019 00:00:00 +0000

Reference

Interrupt

Categorization#

Hardware vs. Software
- Hardware: usually caused by peripheral or other processors
  - IRQ: maskable interrupt
  - NMI: non-maskable interrupt
    - For highest priority tasks, like times, especially wathdog timers
      - Wathdog timer: a timer has to be reset by software on purpose periodically, otherwise it means the software has gone into some hanging situation and will trigger watchdog routine to recover or reboot.
  - IPI: inter-processor interrupt
- Software: caused by exception or special instructions that used to implement system calls
Interrupt vs inter-process communication signal
- Interrupt: mediated by the processor (hardware); handled by the kernel
- Signal: mediated by the kernel (through systeam call); handled by processes
  - Such as: SIGSEGV, SIGBUS, SIGILL, SIGFPE
Precise vs imprecise interrupt
- Precise interrupts has
  - PC and other architecture states are saved, so after interrupt handler is done the current process can resume
  - All instructions before the time point have fully executed, and no instructions beyond has been executed (or they are killed)
Triggering methods
- Physical interrupt
  - Level-triggered vs. Edge-triggered
- Message-signaled interrupts (or message-based interrupt as in ARM’s term)
  - Supported by PCI 2.2 and PCI-Express

MSI (Message Signaled Interrupts)#

Triggerred by write to a memory address
Can be converted from/to physical interrupt
In-band vs. out-of-band
- Dedicated interrupts wires are considered out-of-band, while MSI is in-band.
- Need hardware to convert MSI to physical interupts
Although use in-band network to transmit interrupt info, but it can only be used to descript the interrupt (such as source, priority) while cannot be used to carry data.
Pros & cons
- More scalable
  - Multi-sources to multi-processors
- Simpler, cheaper, more reliable (no interference noises)
- Not compatible with devices that need physical interrupts
- Need software support

Performance issue#

Livelocks

ARM GIC#

Categories#

LPI (locality-specific peripheral interrupt)	PPI (private peripheral interrupt)	SPI (shared peripheral interrupt)	SGI (software generated interrupt)
From peripheral to local PE (processing element)	From peripheral to a single, specific PE	From peripheral to distributor, then to PE that can accept this type of interrupt	From PE to PE, typically used for inter-processor communication
Edge-triggered behavior, message-based (???)	Edge-/level-triggered, need explicit deactivation	edge-/level-triggered, need explicit deactivation	Edge-triggered, need explicit deactivation
Can be routed with ITS	Cannot be routed with ITS	Cannot be routed with ITS	Cannot be routed with ITS
Non-secure	Secure or non-secure	Secure or non-secure	Secure or non-secure

Q: if need deactivation, then there has to be some ackknowledge mechanisms. what are they???

ARM's big.LITTLE Architecture

Wed, 30 Jan 2019 00:00:00 +0000

https://developer.arm.com/technologies/big-little

big.LITTLE is a practical example of SMP (Symmetric Multiprocessing). It combines high performance CPU cores and low power CPU cores in the same chip, connected using cache coherent interconnect, to achieve high peak performance within thermal bounds of the system when intense computational power is needed, as well as maximum energy efficiency when the device is in light usage mode most of the time. It’s a particular adaption to mobile devices usage.

Re-discover Hardware Security in Modern SoC

Wed, 30 Jan 2019 00:00:00 +0000

There is a big difference between how I used to understand hardware security and state-of-the-art security supported by hardware software co-design, after I watched some video talking about SEP (Security Enclave Processor) by Apple. It’s a key component in current iPhone to protect user data and password from being observed in any kind of hacking, including traditional side channel attack such as DPA (Dynamic Power Attack), debug channel attack, normal network attack, and etc.

Huwcha Accelerator architecture

Thu, 03 Jan 2019 00:00:00 +0000

From the reading of this paper, “The Hwacha Microarchitecture Manual, Version 3.8.1”, I found out that our Pygmy ES1 architecture is almost the same idea, just not as fancy.

We don’t have cache coherency, because we operate on unified physical memory space
The vector execution unit is not as fancy, just useless multithreading, no systolic
The prefetch is supposed to be done by DMA engine that is manually controlled by software

System architecture#

The vector accelerator only has L1 I$, no D$
- Don’t need to maintain cache coherency
- Lots of vector registers, 512 in total, each is 64x2x4=512-bit
- Wide bus connection to L2$, to provide higher bandwidth
Uncached TileLink between L1 I$ (both scalar processor and vector processors) and L2$
Cached TileLink between L1 D$ (only in scalar processor)
- L2$ maintains directory bits, which determines the states of corresponding cache line (JW: maybe something like MESI bits)
Operations of L2$, supported by TileLink protocol (“Productive Design of Extensible On-Chip Memory Hierarchies”)
- Sub-cache-block accesses
- Data prefetch requests
  - Read from DDR, don’t need to send back to the requester
- Atomic memory operations
  - ALU inside L2 cache banks

Decoupling#

Access/execute decoupling
Decoupled vector arch
Cache refill/access decoupling

Vector Command Queue (VCMDQ)#

Instruction fetch is handled by scalar processor, and then sent to VCMDQ
- There is explicity defined start vf and stop vstop instructions that flags the begin and end of vector instructions
  - JW: why is that necessary?

Vector Execution Unit (VXU)#

In a systolic style
- 4 banks in total, each bank has 256-entry 2x64-bit vector register file, as well as ALUs
- from the block diagram, these ALUs are only add/subtration/shift operations. Multi-cycle integer multiplier/divider, and floating point operations are outside of each bank and shared by all 4 banks via a crossbar.
  - JW: I don’t think this is good for ML applications. They are trying to make it more generic, so to improve on this, we could take the same micro-architecture but simplify it as well as putting more ML (basically MAC) into it.
- As operation flows through the banks, natually chain different operations together.
  - JW: the chaining is useful because we have limited shared function unit, such as IMUL/IDIV. For function units that are exclusive to bank, I don’t see the benifit.
Predicate is used for simple branch and etc.

Vector Memory Unit (VMU)#

It’s based TileLink protocol
Vector Load Unit (VLU)
- Opportunistic writeback mechanism: return as soon as the data is back, no re-order buffer.
- Simultaneously manage multiple operations to avoid artificial throttling of successive loads: too many requests will drive performance down
Vector Store Unit (VSU)
- Vector Store Data Queue (VSDQ)

Vector Runahead Unit (VRU)#

This block process all the vector load/store instructions in the prefetch queue from the scalar processor, generate address and send prefetch requests to L2$.
- Prefetch request doesn’t return data to requester.
- In ideal world, it will hide the memory access latency.
When from a cold start, it skips the first few load/store to run ahead of the VMU, until it reaches too far ahead, determined by the number of load/store operations that hasn’t been processed by VMU yet, it pauses.
- Cannot run too close, otherwise the memory access latency cannot be hide.
- Cannot run too further ahead, otherwise newly loaded data will force evict the data that’s been currently using by the VXU and cause even worse performance panity.
Also to limit using too much of the L2$ bandwidth, prefetch in L2$ only uses up 1/3 of available outstanding access (JW: I think it more likely the MSHR entries which handles any cache miss).

Multilane#

Hwacha support paramerized number of identical lanes, and these lanes work entirely decoupled from one another.
If the memory needed by each lane aligns to the cache line size, it will be the best, otherwise, there will be some waste of bandwidth to load unnecessary data to the lane.
JW: curious, how do they manage multilane?
- There is a master sequencer who knows the number of lanes out there as well as the vector length. It will dispatch the job to these lanes evenly.
JW: as it discussed in the paper, the lanes work on strip size, which is a unit-stride fasion. If there is any non-unit stride, not only it will waste the bandwidth, but also it will waste the computation resource. So I think it would be a simple dispatch based on unit-stride, the master sequencer will have to consider the stride as well.

Note of RISC-V Vector ISA Spec v0.6

Sat, 15 Dec 2018 00:00:00 +0000

Vector regfile#

32 of them, v0 to v31
Each is VLEN bits
Each can be divided into several elements
- The max element width is ELEN
- CSR vsew maps to SEW (standard element width) controls their width dynamically
- CSR vl controls the number of elements to operate on for vector instructions
Packing of shorter vector
- when SEW is smaller than ELEN, multiple SEW will be packed into one ELEN unit
  - Following little-endian rule
- ELEN units are packed into VLEN register also
  - Following little-endiam rule
Storage of longer vector
- If operand longer than SEW is needed, then
  - Even-numbered vector register holds the even-numbered elements
  - Odd-numbered vector register holds the odd-numberred elements
  - WHY? the author said it’s designed to simplify data path alignment
Register grouping via vlmul field
- Any operation applies on one vector register n will apply on all the other vector registers in the same group
- Depending on the vlmul, 00 = no group, 01 = v[n+16], 02 = v[n+8], 03 = v[n+4]
  - VLMAX will change accordingly, by double, quarod, 8 times
Reuse for floating-point regfile
- Floating-point registers reside at the LSB FLEN of the vector registers
- Lower precision floating-point types are NaN-boxed (MSB bits are set to 'b1)
- Loading floating-point data doesn’t change the upper bits where VLEN > FLEN

Vector operation#

Masking
- Masked off elements are skipped, and will not generate exceptions
- Use the LSB of each SEW in v0 as the masking bit
- 4 types in vm[1:0]: true, false, scalar, no-masking
Vector load/store
- Unit-stride
  - Starting address = base address in RS1 + imm offset
  - Fault-first version
    - WHY?
- Constant-stride
  - Base in RS1, stride in RS2, by an unit of byte
- Indexed (scatter-gather)
  - Base in RS1, indices in VRS2 (signed integers)
    - If SEW is larger than XLEN, LSB bits are taken
  - For store, there is an extra unordered-indexed load, in contrast to ordered-indexed load
    - JW: we can make unordered version to be higher performance
Vector arithmetic
- Widening operation with w suffix puts 2xSEW wide destination into an even-odd vector register pair
- Merge: use the mask field to merge 2 source operands
- Narrowing: convert multi-width vector into single-width vector
- Reduction: scalar <= vector op scalar
- Matrix multiplication support
  - Fused-multiply-add + reduction
- Mask
  - Mask population count
  - Find-first-set mask bit
  - viota: count the number of elements in v0 that’s masked or un-masked
    - Can be combined with scatter/gather instructions to perform vector compress/expand instructions
    - They are still discussing if this should be able to take any registers, not only v0
  - Set-before-frst mask bit
- Permutation
  - insert/extract
  - slide
  - gather

Exception#

CSR progress logs the element index that caused the trap
- With this, failed operations can resume from where exactly it traps exception
Precise vs imprecise traps
- Precise trap: slow, but support debug
- Imprecise trap: fast, but possibly obscure error conditions
- Implementations can choose

Ariane (PULP series high-performance core)

Sat, 01 Dec 2018 00:00:00 +0000

Ariane Document

Architecture note#

PC gen stage#

The fetching address for i-cache is always word-aligned.

Fetch stage#

Its fetch stage doesn’t have much decoding work to do, only the necessary one to generate next PC. And it relies on its branch prediction to give out next PC.
There is an internal FIFO with 2 entries to log the PC (and other meta-info) that was sent to i-cache, while waiting for its response.

Network-on-Chip Notes

Sat, 01 Dec 2018 00:00:00 +0000

NoC#

Clustering coefficient: the most intuitive explanation is the number of hops between two random nodes in the network.
Layers
- Physical layer
- Link layer
  - Transaction protocol: such as AXI
  - Seperated channels like AXI, to avoid dead-lock caused by depency problem
- Transport layer
  - Packet: header & payload
Flow control
- On link-level or end-to-end level
  - Link-level: every hop there is a notification, just like valid-ready protocol
  - End-to-end level: every transaction of data from sender to receiver has to have some kind of notification that the receiver notify the sender it has received the data successfully.
Advantages:
- Multiple transactions can complete in parallel in the network to utilize the routing resource.
- Layered structure makes it more clear, and each layer can be implemented independently.

Cache Coherence Notes

Thu, 01 Nov 2018 00:00:00 +0000

Coherence mechanism#

Snooping#

Every cache maintain its own cache state. And when it needs to access a shared address space, it sends snooping messages to all the other caches to either update or invalidate them.

Write invalidate: write operation will invalidate all the other shared copies. Others will have to read again from the next level of cache to use it again.
Write update: write operation will give the written data to the shared copies and update them accordingly.

One could add a snooping filter to filter out the exesive snooping traffic that doesn’t belong to current cache.

CPU Architecture Notes

Thu, 01 Nov 2018 00:00:00 +0000

Register renaming#

To eliminate the false and output data dependency by adding extra physical registers more than architectural registers.

Read-after-write (RAW) is true data dependency
Write-after-write (WAW) is output data dependency
Write-after-read (WAR) is false data dependency

Superscalar#

Dynamically issue multiple instructions in each cycle to increase IPC.

Normally need multi-port register files and ALU to avoid structural hazard.
Can be in-order or out-of-order

Re-order buffer#

For out-of-order execution CPU architecture, results are put into re-order buffer waiting for commit. These result can be forwarded to later instructions.

CPU Performance Test

Mon, 01 Jan 0001 00:00:00 +0000

Background#

i7-12700K = Intel Core i7-12700K (8 big cores each has 2 threads, 4 little cores each has 1 thread) running at 5GHz rpi4 = Raspberry Pi 4 Rev B, with 4x Cortex-A72 running at 1.8GHz (Broadcom BCM2711) rpi5 = Raspberry Pi 5, with 4x Coretex-A76 running at 2.4GHz (Broadcom BCM2712) am69 = TI AM69 starter kit, with 8x Cortex-A72 running at 2.0GHz (TI AM69) lpi4a = LiCheePi 4A, with 4x RISC-V RV64GCV running at 1.85GHz (Alibaba TH1520)

How to Evaluate NoC (Network-on-Chip)?

Mon, 01 Jan 0001 00:00:00 +0000

Modern SoCs heavily relies on NoC to connect interfaces and storage to compute. As the ML models grow larger and larger, the data delivery ability becomes more and more important to overall system performance.

While everyone is talking about computing resource such as systolic array and vector processors, NoC’s critical role sometimes is overseen. In this blog, I’m going to talk about the metrics to evaluate NoC in engineering practice.