RT Build: Firmware, OS and RT Kernel#

Prerequisite:: Physical Build: Hardware Setup complete
Next:: Software Build: Packages, Python, Drivers and Utilities

1. Scope #

The OS is Real-time Ubuntu 26.04 LTS: Ubuntu with Canonical’s PREEMPT_RT kernel (7. Real-time Kernel). PREEMPT_RT makes the kernel fully preemptible, converts spinlocks to sleeping rt-mutexes with priority inheritance, and runs IRQ handlers as schedulable threads. The result is a bounded worst-case latency.

This page takes the machine from bare hardware to a tuned RT platform: TCC enabled in firmware, hard CPU shielding, and background services stripped.

Design rules:

The RT kernel is the foundation, not an optimization. Every tuning step below assumes PREEMPT_RT is running.
No desktop environment. Ubuntu Server, no GNOME, no display manager, no snaps beyond the base set.
Nothing runs on shielded cores except instrument code. Shells, VNC and services are confined to housekeeping cores.

Note

$ prompts are omitted. Unless a block says otherwise, run as hsfei with sudo. Steps marked [reboot] require a restart before continuing.

2. Installation Media #

All drives are erased before starting; this is a full rebuild.

OS: Ubuntu Server 26.04 LTS (.iso, not Desktop)
Source: official Ubuntu download
Format: bootable USB (dd / Rufus / balenaEtcher)

3. Firmware: BIOS, TCC and Hardware RAID #

Do this before installing the OS. The RAID volume must exist for the installer to see it, and disabling SMT changes the core count used for shielding.

BIOS Update#

Flash the MIC-770 V3 to BIOS M770V3WWF60XH01. The files and Advantech’s “AMI BIOS update (UEFI)” guide are in FEI_Server_BIOS_Documents.zip.

Build the updater USB (on a Windows machine; the USB is wiped):
1. Extract FEI_Server_BIOS_Documents.zip and run rufus-2.18.exe from the MIC BIOS flash directory.
2. Select the USB under Device, set Partition Scheme to MBR for UEFI, tick only Quick Format, and press Start. The drive is empty when it finishes.
3. Copy the efi directory, all three .bin files and AfuEfix64.efi onto the USB, then eject it.
Plug the USB into the FEI server, power on and press Del for BIOS.
Boot tab: move UEFI: USB, Partition 1 to Option #1, then Save & Exit.
Let startup.nsh run (do not skip it). It drops to an EFI shell.

Flash the xH01 image:

AfuEfix64.efi M770V3WWF60XH01.bin /p /b /n /x

When the shell returns, power off with the power button and power on again.
Enter BIOS. Main tab: Project Version should read M770V3WWF60XH01.
Save & Exit ‣ Restore Defaults ‣ Load Optimized Defaults: Yes, then Save & Exit.

BIOS Settings#

Reboot and enter BIOS (Del).
Hyper-Threading / SMT / Logical Processors: Disabled. Deterministic execution requires one thread per physical core.
TCC Mode: Enabled. On Intel reference BIOS this is under Intel® Advanced Menu ‣ Time Coordinated Computing. If the option is hidden, consult the board vendor or set the underlying options manually per Intel’s TCC User Guide.
Double reboot. TCC settings are not fully applied until the second boot.

Note

TCC Mode subsumes the manual C-state work. It disables C-states and optimizes power-state and frequency transitions. Canonical measured average scheduling jitter dropping from ~100 µs to under 10 µs on an isolated core from this setting alone.

An isolated core whose periodic task finishes early idles for the rest of the cycle, and the idle subsystem drops it into a deep C-state with a long exit latency. That exit latency is the jitter. If TCC Mode is unavailable, disable C-states below C1 manually and set the power profile to Maximum Performance.

Hardware RAID 1 (Boot Volume)#

The two Samsung 990 Pro 1 TB NVMe drives form a RAID 1 array on the Broadcom MegaRAID 9520-2M2 controller.

Enter BIOS (Del) and open the MegaRAID controller’s configuration page (typically listed under the Advanced tab).
Create a RAID 1 virtual drive from the two 990 Pro NVMe drives.
Confirm the array reports Optimal.
The installer should then show a single ~1 TB device for the OS.

4. OS Installation (Ubuntu Server 26.04)#

Installation Parameters#

Language / Keyboard: English
Networking: Ethernet connected, no proxy. Leave DHCP for now; static addresses are applied in 6. Network Configuration (netplan).
Mirror: http://us.archive.ubuntu.com/ubuntu/ (default)
Server profile: minimized install.
OpenSSH: install it. Without sshd a headless build stops here.
Featured / Popular Snaps: select none. snapd timers are a jitter source.

Storage#

Root filesystem: the MegaRAID RAID 1 volume (Hardware RAID 1 (Boot Volume))
Data: 2 TB SSD, ext4, mounted at /data

Credentials#

Server name: hispecfei
User: hsfei (gets sudo)
Password: set during installation, not documented here

First Boot#

sudo apt update && sudo apt upgrade -y

5. Identity: Hostname, Users, Groups #

Hostname#

sudo hostnamectl set-hostname hispecfei

Map the loopback alias in /etc/hosts:

127.0.1.1   hispecfei

User and Groups#

hsfei, created at install, is the only account on this machine. It runs the instrument software and owns the hardware device nodes.

sudo groupadd -f hispecfei     # instrument / deployment group
sudo usermod -aG dialout,hispecfei hsfei

Note

Group changes require a logout/login (or newgrp dialout for the current shell). Confirm with id hsfei.

SSH Keys and Hardening#

Install your public key before relying on remote-only access:

ssh-copy-id hsfei@hispecfei

Then set in /etc/ssh/sshd_config:

PasswordAuthentication no
PermitRootLogin no
X11Forwarding yes
X11UseLocalhost yes

sudo systemctl restart ssh

X11Forwarding yes is required for 3. Remote GUI and Plot Access.

6. Network Configuration (netplan)#

Create /etc/netplan/01-hispecfei.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    # --- Management / site network ---
    MGMT_IFACE:
      dhcp4: no
      addresses: [192.168.29.107/24]
      routes:
        - to: default
          via: 192.168.29.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]

    # --- Archon fiber link (isolated, no gateway) ---
    enp202s0f0np0:
      dhcp4: no
      addresses: [10.0.0.10/24]
      mtu: 9000

Replace MGMT_IFACE with the real name from ip -br link. Apply:

sudo chmod 600 /etc/netplan/01-hispecfei.yaml
sudo netplan try            # auto-reverts in 120 s if you lose the link
sudo netplan apply

Warning

Always use netplan try first. A bad netplan file on a headless box means a physical trip to the machine.

Archon Host Entry#

Add 10.0.0.2 archon to /etc/hosts, connect the Archon to the fiber port labelled archon, and verify:

ping -c 3 archon
ip -br addr show enp202s0f0np0

The 10.0.0.0/24 link has no default route on purpose: instrument traffic stays off the management network. See archongui.rst for Archon-side configuration.

7. Real-time Kernel #

From 26.04, PREEMPT_RT is fully upstream and the real-time kernel is in the main Ubuntu archive. No Ubuntu Pro subscription is needed (earlier releases required one).

Install [reboot]#

sudo apt update
sudo apt install ubuntu-realtime
sudo reboot

If the machine comes back on the generic kernel, pick the -realtime entry under Advanced options for Ubuntu in the GRUB menu.

Verify#

uname -a                              # contains PREEMPT_RT, -realtime
cat /sys/kernel/realtime              # 1
chrt -m                               # SCHED_FIFO / SCHED_RR 1-99
grep -c . /proc/pressure/cpu          # PSI available

Note

Ubuntu Pro is optional. The build should have no Pro account, or one registered to a dedicated HISPEC email, never a personal or staff email. If Pro is attached but not needed (extended security updates, Livepatch), remove it with sudo pro detach. Never paste the Pro token into this document, a script, or shell history.

As-built Result#

Component	Value
CPU	Intel Core i5-13500E (13th Gen), 14 cores, 1 socket
Firmware	TCC-capable BIOS, TCC Mode enabled
OS	Ubuntu 26.04 LTS
Kernel	`7.0.0-38-realtime`
Kernel config	`PREEMPT_RT=y`, `INTEL_TCC=y`, `INTEL_TCC_COOLING=m`, `TSNEP=m`
Scheduler	`SCHED_FIFO` 1-99, `SCHED_RR` 1-99
PSI	CPU pressure available

8. CPU Shielding #

GRUB Kernel Parameters [reboot]#

Cores 0-5 are shielded from the scheduler, RCU callbacks and the timer tick. Edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet clocksource=tsc tsc=reliable nmi_watchdog=0 nosoftlockup isolcpus=domain,0-5 rcu_nocbs=0-5 nohz_full=0-5 irqaffinity=6-15 kthread_cpus=6-15"

Parameter	Purpose
`clocksource=tsc tsc=reliable`	Use the TSC; avoids HPET/ACPI read latency spikes
`nmi_watchdog=0`	Remove periodic NMI perf interrupts
`nosoftlockup`	Suppress soft-lockup warnings from long RT bursts
`isolcpus=domain,0-5`	Remove cores 0-5 from all scheduling domains
`rcu_nocbs=0-5`	Offload RCU callbacks from the isolated cores
`nohz_full=0-5`	Stop the tick when one task is runnable
`irqaffinity=6-15`	Direct hardware IRQs to housekeeping cores
`kthread_cpus=6-15`	Restrict kernel threads to housekeeping cores

Warning

The irqaffinity and isolcpus ranges must not overlap.

sudo update-grub
sudo reboot

# After boot
cat /proc/cmdline
cat /sys/devices/system/cpu/isolated       # 0-5
cat /sys/devices/system/cpu/nohz_full      # 0-5

IRQs still bound to a shielded core after reboot are typically PCIe devices. Rebinding them is not trivial or safe; they are left as-is and jitter has been confirmed within spec.

Disable irqbalance#

irqbalance redistributes interrupts at runtime and silently undoes irqaffinity:

sudo systemctl disable --now irqbalance

Confine systemd Services#

Keep every systemd service, present and future, off the shielded cores. Edit /etc/systemd/system.conf:

[Manager]
CPUAffinity=6-13

sudo systemctl daemon-reexec

Check C-states#

Confirm TCC (3. Firmware: BIOS, TCC and Hardware RAID) disabled the deep C-states:

for cpu in /sys/devices/system/cpu/cpu*/cpuidle/state*; do
    echo -n "$cpu: "; cat "$cpu"/name
    echo -n "  Exit latency: ";     cat "$cpu"/latency
    echo -n "  Disabled [1=yes]: "; cat "$cpu"/disable
done

Further Intel Optimizations (optional)#

If the latency baseline (11. Verification) is not tight enough, evaluate:

Cache Allocation Technology (CAT): partitions last-level cache so housekeeping workloads cannot evict the RT task’s working set.
Speed Shift / HWP: tunes frequency transitions on the isolated cores.

See Canonical’s Optimizing real-time performance on Intel CPUs.

9. Service Stripping #

Disable unneeded daemons, timers and update machinery. || true keeps the block safe when a unit is absent.

for svc in \
    irqbalance.service \
    cups.service cups-browsed.service \
    ModemManager.service \
    avahi-daemon.service avahi-daemon.socket \
    bluetooth.service \
    apt-daily.timer apt-daily-upgrade.timer \
    unattended-upgrades.service \
    fwupd-refresh.timer \
    man-db.timer \
    motd-news.timer \
    update-notifier-download.timer \
    update-notifier-motd.timer \
    sysstat-collect.timer \
    sysstat-summary.timer \
    sysstat-rotate.timer \
    multipathd.service \
    udisks2.service \
    plocate-updatedb.timer \
    apport.service \
    ubuntu-advantage.service ua-timer.timer
do
    sudo systemctl disable --now "$svc" 2>/dev/null || true
done

# Audit what is left
systemctl list-units --type=service --state=running
systemctl list-timers --all

Keep enabled: ssh, systemd-networkd, systemd-resolved and chrony. ubuntu-advantage and ua-timer are only needed when Ubuntu Pro is attached (7. Real-time Kernel); drop them from the loop in that case. apport is Ubuntu’s crash reporter and is not needed here. thermald and networkd-dispatcher are retained pending further RT/thermal testing.

Warning

With unattended-upgrades disabled, security patching is manual. Schedule a maintenance window.

10. RT Tooling and Scheduling Privileges #

sudo apt install -y util-linux rt-tests linuxptp stress-ng

rt-tests provides cyclictest for 11. Verification.

Allow hsfei to request real-time priority. Create /etc/security/limits.d/99-rt.conf:

hsfei    -    rtprio    95
hsfei    -    memlock   unlimited
hsfei    -    nice      -20

Log out and back in to apply. Alternatively grant the capability on chrt:

sudo setcap 'cap_sys_nice=eip' "$(command -v chrt)"

11. Verification #

Reboot, then record the results as the as-built baseline.

Check	Command	Expected
Hostname	`hostnamectl`	`hispecfei`
User / groups	`id hsfei`	`dialout`, `hispecfei`
RT kernel	`uname -a`	contains `PREEMPT_RT`
RT flag	`cat /sys/kernel/realtime`	`1`
Kernel cmdline	`cat /proc/cmdline`	matches 8. CPU Shielding
Isolated / tickless cores	`cat /sys/devices/system/cpu/{isolated,nohz_full}`	`0-5`
IRQ affinity	`cat /proc/irq/*/smp_affinity_list`	none confined to 0-5 (PCIe excepted)
irqbalance off	`systemctl is-enabled irqbalance`	`disabled` / not-found
systemd affinity	`grep CPUAffinity /etc/systemd/system.conf`	`6-13`
SMT disabled	`lscpu \| grep 'Thread(s) per core'`	`1`
C-states (TCC)	`cat /sys/devices/system/cpu/cpu0/cpuidle/state*/disable`	deep states `1`
Clocksource	`cat /sys/devices/system/clocksource/clocksource0/current_clocksource`	`tsc`
No desktop	`systemctl get-default`	`multi-user.target`
Management net	`ip -br addr`	`192.168.29.107/24`
Archon link	`ping -c 3 archon`	0% loss

Latency Baseline#

# 30 min, RT prio 80, one thread per shielded core
sudo cyclictest -m -S -p80 -i200 -h400 -D30m -a 0-5 > cyclictest_baseline.txt

# Repeat while stressing the housekeeping cores
stress-ng --cpu 8 --taskset 6-15 --timeout 30m &
sudo cyclictest -m -S -p80 -i200 -h400 -D30m -a 0-5 > cyclictest_loaded.txt

A large gap between loaded and idle max latency means something is still running on the isolated cores; recheck irqaffinity first.

12. Troubleshooting #

Symptom	Resolution
Installer shows two 1 TB drives	RAID 1 volume not created; revisit Hardware RAID 1 (Boot Volume).
Latency spikes under load	Check in order: (1) `irqaffinity` overlapping the shielded set; (2) `irqbalance` still running; (3) TCC Mode off / C-states enabled (look for jitter on an idle isolated core); (4) SMT still on; (5) unpinned processes on 0-5 (5. Real-Time Process Placement); (6) cache contention, evaluate Intel CAT.
RT kernel not booting after a kernel update	Select the previous kernel in GRUB. Out-of-tree modules need rebuilding against the new `uname -r`.
`chrt`: “Operation not permitted”	`limits.d` rtprio grant or `cap_sys_nice` missing (10. RT Tooling and Scheduling Privileges); re-login after editing limits.
Locked out after `netplan apply`	Physical console required. Always `netplan try` first.
Group membership not taking effect	Re-login, or `newgrp dialout` for the current shell.

13. Pending Tasks #

[ ] RAID 1 on /data: data drive is not yet mirrored.
[ ] Intel CAT / Speed Shift: evaluate if the latency baseline is not tight enough.
[ ] Patching policy: unattended-upgrades is disabled; define a manual maintenance window.

RT Build: Firmware, OS and RT Kernel#

1. Scope #

2. Installation Media #

3. Firmware: BIOS, TCC and Hardware RAID #

BIOS Update#

BIOS Settings#

Hardware RAID 1 (Boot Volume)#

4. OS Installation (Ubuntu Server 26.04)#

Installation Parameters#

Storage#

Credentials#

First Boot#

5. Identity: Hostname, Users, Groups #

Hostname#

User and Groups#

SSH Keys and Hardening#

6. Network Configuration (netplan)#

Archon Host Entry#

7. Real-time Kernel #

Install [reboot]#

Verify#

As-built Result#

8. CPU Shielding #

GRUB Kernel Parameters [reboot]#

Disable irqbalance#

Confine systemd Services#

Check C-states#

Further Intel Optimizations (optional)#

9. Service Stripping #

10. RT Tooling and Scheduling Privileges #

11. Verification #

Latency Baseline#

12. Troubleshooting #

13. Pending Tasks #

14. References #

Real-time Ubuntu#

Tuning#

Intel TCC#

Measurement#