Thanh Le / bench notes
EMBEDDED · BOARD ROCK3A · DONE

FreeRTOS on core 3 at 154 ms, Linux on the other three

Boot from SPI NOR, a 1 kHz loop through DDR training, a patched BL31 handover, and cutting the image the BootROM reads

ResultFreeRTOS takes core 3 at 0.154 s (was 3.90 s); Linux prompt at 2.49 s; 1 kHz loop worst lateness 0.25 µs
Last change2026-10-23: 472 packed to 111 KB, RTOS at 154 ms
Boardall stories on this board →

FreeRTOS on core 3 at 154 ms, Linux on the other three

Back to the ROCK 3A hub

Three NOR boot builds on one time axis

The figure is from my wiki, so the labels are in Vietnamese. Each row is one build; the green arrow is the moment FreeRTOS owns core 3, the pink bar under it is the real-time loop that runs from 0.4 ms.

Situation

I had FreeRTOS on four cores and a mainline Linux, but never both, and never quickly. On a robot the motor loop shouldn’t wait for Linux to finish booting. A normal RK3568 boot needs seconds before anything of mine runs: the DDR blob trains, the BootROM loads the next stage, TF-A, U-Boot, kernel.

Task

A hard real-time loop on core 3 from the very first instructions, even while DRAM is being trained. Then hand core 3 to FreeRTOS and boot Linux on cores 0 to 2 next to it, with no U-Boot. All of it from the 16 MiB SPI NOR, so the board boots by itself, and with the RTOS up as early as I can get it.

The plan: a stage between BL31 and Linux

The plan I drew before writing any of it (Vietnamese labels): BootROM, DDR blob, my tramp at EL3, BL31, then core 3 started into rtos_amp while Linux boots on cores 0 to 2 and attaches to it later with amp_ctl.ko.

Action

The loop has to run before DRAM exists. The only RAM free while the DDR blob runs is the second half of the 8 KB PMU SRAM. My first-stage 471 starts the STIMER, copies a tiny rt_loop there, wakes core 3 through the BootROM’s mailbox and returns. Core 3 then busy-waits 1 ms deadlines on CNTPCT. The first measurement said “no period lost”, and it was wrong: the DDR blob resets the STIMER twice (around 5.5 ms and 285 ms), and a loop that only compares against its deadline can’t see a counter that jumped back. The loop now treats any reading lower than the previous one as a reset and shifts its deadline by the same jump.

Core 3 runs a 1 kHz loop while the DDR blob trains

Core 3 in its 1 kHz loop from PMU SRAM, under the whole DDR training and download: 7820 periods in 7821 ms, worst 0.25 µs late. The red marks are the two STIMER resets (~5.5 ms and ~285 ms) that fooled my first version.

Booting from NOR ate a whole afternoon, most of it asking “why is it back in Maskrom”. U-Boot’s mkimage has two Rockchip header types, rksd and rkspi. rkspi pads every 2 KB to 4 KB for older SoCs. I picked rkspi because the name said SPI. The RK3568 BootROM silently ignores it and falls back to Maskrom. Comparing against U-Boot’s own u-boot-rockchip-spi.bin showed it uses rksd, at NOR offset 0x8000. I also stopped trusting the Rockchip SPL loader for writing the NOR (it once repeated a 32 KB chunk, and once wedged so hard only a power cut helped) and wrote a spi flash command into my own shell that erases, programs and reads back every byte.

Same two images, mkimage -T rksd vs -T rkspi

The afternoon’s mistake in one picture: the same two images packed with -T rksd (data back to back, boots) and with -T rkspi (2 KB of data in every 4 KB, the RK3568 BootROM ignores it and falls back to Maskrom). Idblock at NOR offset 0x8000.

For the handover, a small tramp at EL3 places the RTOS at 0x30000000 and fills a mailbox. I patched BL31 in three places, each of which would kill an RTOS that runs before Linux: it reloads the STIMER (the counter would jump to 0 under a running tick), it leaves Group 1 Non-secure interrupts off until Linux’s GIC driver turns them on, and its write of the request sat in core 0’s cache for 19 ms because core 3 reads PMU SRAM with its MMU off. With a cache clean after the write, core 3 sees the request 22 µs later. Linux then boots and amp_ctl just attaches: core 3 already runs '20261023-2156': attached.

The first NOR build gave FreeRTOS core 3 only at 3.9 s. The BootROM reads the whole second image from NOR, one data line at 24 MHz, about 2.8 MB/s, before running a byte of it. My 472 carried the 6.5 MB kernel, so the RTOS waited for a kernel it doesn’t need. I split it: the BootROM now reads a small 472 (366 KB), and a 3 KB nsloader I wrote reads the kernel later with quad-output reads and a CRC-32 on the fly. That alone brought the RTOS to 0.242 s.

Then 366 KB down to 111 KB. 136 KB of that 472 was all-zero pages (page tables, gaps), and 66 KB was the DTB, which nsloader can read later. mkimage.py now cuts everything into 64-byte blocks and writes a table of (src, dst, len) records; src = 0 means “write zeros”. Before writing the NOR I ran a Python replay of tramp over the 31 records and compared every piece byte for byte, because a bad record only shows as a board that doesn’t boot.

  0.101  out
  0.141  NOTICE:  BL31: core 3 asked to start the RTOS at 0x30000000
  1.495  nsloader: read 1350 ms (4755 KB/s), CRC ok; Linux
  2.487  linux_min: BusyBox shell (exit = kernel panic, reset the board)
  handover: asked at 153 ms, taken 829 us later, after 154 periods (154 ms since power-on)

Result

Milestone First NOR build Kernel split off Packed 472
Image the BootROM reads ~10.7 MB 366 125 B 110 832 B
FreeRTOS owns core 3 3.897 s 0.242 s 0.154 s
Linux prompt 4.888 s 2.561 s 2.487 s

The real-time loop starts 174 µs after my first instruction and keeps its 1 kHz period through DDR training and the whole download: 7820 periods in 7821 ms, worst lateness 0.25 µs, 0 periods late by more than 10 µs. The only gap on core 3 is the switch from the loop to FreeRTOS, 4.3 ms.

The 472 before and after packing, and the NOR map

The image the BootROM has to read, before and after packing: 366 125 B with 136 KB of zero pages (BootROM done at 141 ms), then 110 832 B as 31 records (40 ms). Below, the 16 MiB NOR to scale, with the 6.4 MB kernel at 0x100000 that nsloader reads later.

The blue LED tells me who owns core 3 without a terminal: an even 1 Hz is the loop, a double blink is FreeRTOS.

What I’d do next

Of the 154 ms, 101 ms is DDR training in Rockchip’s closed blob, and I don’t see a way around that. The Linux side is where the time is now: nsloader reads the NOR at 4.8 MB/s, while U-Boot read the eMMC at 91 to 125 MiB/s in HS200 on the same board. An eMMC driver in nsloader would cut most of the 1.35 s kernel read. For the RTOS itself I expect almost nothing from eMMC (my guess is 0.15 to 0.17 s, not measured).