[{"content":"There is one failure mode an A/B update system is not supposed to have, because avoiding it is the entire reason you pay for A/B: you ship an update, the device installs it, every status check returns green, yet the device quietly goes on running the old version. Not a visible rollback triggered by a crash loop. A fallback so clean that nothing upstream of the board ever changes color.\nI hit exactly that on a Raspberry Pi 5 gateway running RAUC1. The distance between \u0026ldquo;the pipeline reported success\u0026rdquo; and \u0026ldquo;the new kernel never executed once\u0026rdquo; came down to a single number in the bootloader\u0026rsquo;s configuration, one that sits one layer below everything the update pipeline knows how to verify.\nA kernel that grew The change that set it off was routine. I wanted portable eBPF tracing2 on the gateway, and portable tracing means BTF3, and BTF is generated by pahole4 from DWARF, so the kernel has to be built with debug info even though the running system never reads it. Yocto\u0026rsquo;s KERNEL_DEBUG5 keeps that DWARF in vmlinux, but vmlinux is not what ships. What lands in the FIT is linux.bin: vmlinux run through objcopy -O binary -S, and the -S strips the non-allocated sections, DWARF among them. The debug info never reaches the device. What survives is the .BTF section. The kernel\u0026rsquo;s BTF generator marks it allocable on purpose, so it lives through that strip and is present at runtime for CO-RE to read. That surviving metadata is the weight that came along: small, but enough to matter here. In the bundle the compressed kernel grew from about 18 MiB to 19.9 MiB. Unremarkable, and not the number that mattered.\nThe number that mattered is the one nobody watches: the decompressed size the bootloader has to land in RAM.\n$ dumpimage -T flat_dt -p 0 -o /tmp/k.gz fitImage $ gunzip -c /tmp/k.gz | wc -c 69048328 65.85 MiB, decompressed. Hold onto it.\nThe symptom, and where it wasn\u0026rsquo;t The bundle built, signed, and installed without complaint. rauc status marked slot B good, the hashes matched, the boot order flipped to B A. The device rebooted onto slot A. Three more reboots, still A. On paper the update had landed; in practice the new slot had never run.\nThe reflex is to suspect whatever changed, and what changed was the RAUC bundle. That reflex is wrong here, and the serial console is where it becomes obvious. The FIT loaded from the boot partition, its signature verified against the baked-in key, the kernel subimage hash verified. Then bootm began decompressing the kernel into its load address and stopped:\nUncompressing Kernel Image to 1314c40 Error: inflate() returned -5 Image too large: increase CONFIG_SYS_BOOTM_LEN inflate() returning -5 is zlib\u0026rsquo;s Z_BUF_ERROR6: the output buffer filled before the stream was done. Every integrity check upstream of this line had already passed. The failure is downstream of all of them, in the one operation none of them model: decompression into a fixed buffer.\nOne number, one layer down That buffer is CONFIG_SYS_BOOTM_LEN. The upstream rpi_arm64_defconfig never sets it, so the build inherits the Kconfig default7 of 0x4000000, 64 MiB. The decompressed kernel is 65.85 MiB. The arithmetic isn\u0026rsquo;t subtle, and the fix isn\u0026rsquo;t either: set the ceiling explicitly, with headroom.\nCONFIG_SYS_BOOTM_LEN=0x8000000 128 MiB leaves the kernel sitting at half the ceiling, room for drift before this returns. One line. The interesting question is not how to fix it; it\u0026rsquo;s why nothing between the build server and the board caught a kernel that had outgrown the bootloader shipped alongside it.\nWhy the pipeline couldn\u0026rsquo;t see it The bundle carries two things that have to agree: a kernel and a bootloader. They are built independently, and nothing enforces the one assumption they share: that the decompressed kernel fits the buffer the bootloader was compiled with. The kernel side assumed 128 MiB of headroom; the bootloader side shipped 64. Both halves were internally correct. Their contract was violated, and no stage in the pipeline is written to check a contract that spans them.\nIt couldn\u0026rsquo;t have been caught by the checks that exist, because every one of them operates on bytes at rest. The bundle signature, the FAT-side SHA, the FIT config signature, the kernel hash all verify that the right bytes arrived intact. None of them can answer \u0026ldquo;will these bytes decompress into the buffer this particular bootloader has.\u0026rdquo; That property is only observable at bootm time, on the device, on the first boot of the new slot, the last place you want to discover it.\nAnd A/B did its job, which is the part that stings. Redundancy exists to make a bad install survivable, so A/B treated a good install with a latent incompatibility identically: slot B failed its three attempts, and slot A, whose kernel and bootloader were last written as a mutually consistent pair, won the fallback and booted normally. From the fleet\u0026rsquo;s point of view the device was healthy. The only two places the truth surfaced were /proc/cmdline reporting rauc.slot=A after an install that targeted B, and the serial log holding the diagnostic. Neither is wired into anything that pages you.\nIt surfaced on a dev image, on the bench, where the cost was an evening and a serial cable. The same silent fallback on a fielded production unit is a different bill: a device that reports a successful update and quietly keeps running the old build is the kind of thing you rediscover as a commissioning engineer\u0026rsquo;s flight to a remote site, to reflash a board that swore it was fine. Catching it here was luck of timing, not design, which is most of why it\u0026rsquo;s worth writing down.\nWhat I\u0026rsquo;d actually change The Kconfig line closes this instance and nothing more. A fix that relies on remembering to raise a buffer every time the kernel gains weight isn\u0026rsquo;t a fix; it\u0026rsquo;s a scheduled repeat. Three things generalize out of this, in ascending order of how much they\u0026rsquo;d have helped.\nThe smallest is that the CO-RE debug metadata, the DWARF and BTF info that produced the growth, has no business in the image the fleet builds by default. It\u0026rsquo;s bench equipment: needed for portable tracing while you\u0026rsquo;re developing against the kernel, dead weight in every image that isn\u0026rsquo;t.\nThe larger one is the coupling between kernel and bootloader, currently implicit and enforced by nothing. Make it explicit where it\u0026rsquo;s cheap to check: a build step that decompresses the kernel out of the FIT and fails the build if it wouldn\u0026rsquo;t clear the bootloader\u0026rsquo;s configured ceiling, with margin. That turns this class of bug from a field fallback into a red build. The build server is where it costs minutes; the device in the field is where it costs a debugging session and a serial cable.\nThe uncomfortable one is the monitoring. A pipeline that reports success at every layer while the device fails to boot the thing it just installed is not observability, it\u0026rsquo;s decoration. The signal exists: \u0026ldquo;installed to B, booted from A\u0026rdquo; is a computable fact the moment the device comes back up. It simply isn\u0026rsquo;t treated as one. Surfacing that mismatch as a first-class alert is more work than a Kconfig line, and it\u0026rsquo;s the change that would actually have caught this without anyone reading a serial log.\nThe first of those is the part I actually changed. The BTF/CO-RE debug metadata now lives in its own opt-in kernel fragment, gated off by default and kept deliberately separate from the eBPF tracing runtime, so you can turn on bpf, kprobes and ftrace for everyday tracing without dragging in the DWARF that inflates the image, and you only pay the size when you explicitly ask for a CO-RE lab build. The default profile, and every production image, never carries it and never comes near the ceiling. The BOOTM_LEN bump stays regardless, as the floor under the dev builds that do opt in.\nThe other two haven\u0026rsquo;t shipped, and the gap between them is the point. The build-time fit-check still doesn\u0026rsquo;t exist, though the bundle did grow an adjacent safeguard since: it now self-heals a set of boot-critical U-Boot environment variables from a canonical file carried inside the signed bundle. That closes a real failure mode, a stale boot script stranding an OTA\u0026rsquo;d kernel, but not this one: it corrects an environment, not a kernel that won\u0026rsquo;t decompress. And the monitoring is still exactly where it was: \u0026ldquo;installed to B, booted from A\u0026rdquo; remains a fact the system could compute on every boot and doesn\u0026rsquo;t. Which is the uncomfortable payoff of listing these in ascending order of leverage: the change that would have caught this earliest is the one still not built.\nUntil that changes, the only thing keeping a big kernel and a small buffer out of the same bundle is a convention. Conventions are exactly what failed here the first time.\nUpdate (v0.5.0, July 2026). This was the boot chain as it stood in May 2026. A later change, shipped as v0.5.0, re-architected it: the signed-FIT flow was split and the non-FIT boot path removed. The CONFIG_SYS_BOOTM_LEN fix here still stands; the machinery around it has moved. That rework is a story for another post.\nGateway Yocto BSP (open source): github.com/umair-as/rpi5-iot-gateway — U-Boot config, RAUC hooks, and kernel fragments under meta-iot-gateway/.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBPF Portability and CO-RE — Andrii Nakryiko\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nKernel BTF documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\npahole — dwarves project\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nYocto Variables Glossary — KERNEL_DEBUG\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nzlib.h — Z_BUF_ERROR definition\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nU-Boot boot/Kconfig — CONFIG_SYS_BOOTM_LEN\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://umair-as.github.io/blog/posts/uboot-bootm-len-silent-ab-fallback/","summary":"A debug-info kernel overshoots U-Boot\u0026rsquo;s decompression buffer by 1.85 MiB. The bundle verifies at every layer, RAUC reports success, and the board silently falls back to the old slot, because the one property that mattered is only observable at bootm time on the device.","title":"When the Kernel Outgrew the Bootloader: A Clean Install That Never Booted"},{"content":"RAUC HTTPS streaming with mTLS had been working on the rpi5-iot-gateway — a Yocto-built Raspberry Pi 5 with a Thread border router, MQTT broker, and a self-hosted RAUC OTA backend — since March, on file-based client keys: /etc/ota/device.key on the rootfs, referenced from system.conf\u0026rsquo;s [streaming] section, paired with a CA-signed device.crt. That stack was good enough to get over-the-air installs running over a real WAN, but it left the private key recoverable from any rootfs image — an SD card pulled from a fielded device, a backup, a misplaced flash dump.\nThis post is the next step: moving that streaming key off the filesystem and into the TPM2 via PKCS#11. The Yocto recipe wiring was straightforward — the URI handed to RAUC\u0026rsquo;s streaming subprocess was not.\nWhy TPM-back the streaming key A file-based device.key on the rootfs is recoverable from any image of that rootfs — an SD card pulled from a fielded device, a debug image left on a workbench, a backup blob in a build artifact. The recovered key is enough to impersonate the gateway against the OTA backend indefinitely, and a CA revocation only closes the window after the abuse is detected. Moving the key into a TPM2 object accessed via PKCS#11 binds it to this physical device\u0026rsquo;s TPM: the same image read on a different machine yields a useless certificate.\nThis is the convergent pattern across embedded mTLS clients: AWS IoT Greengrass v2, Azure IoT Edge, and Mender all support PKCS#11-backed TPM or HSM keys for the same reason. Any client that loads its key through OpenSSL\u0026rsquo;s PKCS#11 engine — as RAUC does — lands on the same libp11 URI parser this post is about.\nThe honest limitation: a live device with the PIN file readable to its owner process remains a signing oracle. A root-level runtime compromise can drive the TPM to sign anything at request rate. This migration addresses rootfs-at-rest, not runtime compromise — the full threat-model triage and the CRA mapping land in the planned hardening series.\nThe provisioning side, briefly Before any of this matters at install time, three things have to be in place on the device. First, a TPM2 PKCS#11 token store at /var/lib/tpm2_pkcs11, initialised with tpm2_ptool and populated with the rauc-client-key keypair generated inside the TPM. Second, the ota system user — created by an iotgw-ota-user Yocto recipe (system account, group ota, no home, /bin/false shell) — which is the sandbox-user RAUC drops the streaming subprocess to. Third, the OTA material under /etc/ota/: device.crt and ca.crt for mTLS, and pkcs11-pin (mode 0640 root:ota) holding the PIN that unlocks the PKCS#11 token at runtime.\nA oneshot ota-certs-provision service handles all of this on every boot. It sources cert material from the boot partition (factory provisioning), falls back to /data/ota/certs/ (persisted from prior provisioning), and then runs a PKCS#11 preflight that opens the token, unlocks it with the PIN file, and signs a probe message. If the preflight passes, RAUC has everything it needs by the time the OTA service starts. The preflight is also where this post\u0026rsquo;s bug was hidden in plain sight — but that comes later.\nThe four moving parts The migration touches four things in the Yocto layer: a distro toggle (IOTGW_RAUC_STREAMING_KEY_MODE) that selects file vs pkcs11; the rauc-conf recipe template that substitutes the chosen tls-key value into system.conf; a systemd drop-in that exports PKCS11_MODULE_PATH and TPM2_PKCS11_STORE to the streaming subprocess; and the upstream RAUC meson option pkcs11_engine, which has to be enabled so RAUC links libp11 as the OpenSSL PKCS#11 engine.\nThe full wiring is at the bottom of this post. It is worth seeing the failure first.\nFirst boot in PKCS#11 mode: server not responding Day one of the new mode. The URI in local.yml looked like every TPM2 PKCS#11 example online — token, object, type, the module path, the PIN source:\npkcs11:token=iotgw;object=rauc-client-key;type=private;module-path=/usr/lib/pkcs11/libtpm2_pkcs11.so;pin-source=file:/etc/ota/pkcs11-pin A local-bundle install (rauc install /tmp/foo.raucb) worked. HTTPS streaming from the same server, with the same bundle, failed every time. The top-level error was failed to configure streaming: server not responding, which implied a network or TLS connectivity problem.\nIt wasn\u0026rsquo;t.\nFrom journalctl -u rauc -b during rauc install https://...:\nApr 23 05:36:45 iot-gateway rauc[1090]: Remote URI detected, streaming bundle... Apr 23 05:36:45 iot-gateway rauc[1090]: sending config request to nbd server: { \u0026#39;key\u0026#39;: \u0026lt;\u0026#39;pkcs11:token=iotgw;object=rauc-client-key;type=private; module-path=/usr/lib/pkcs11/libtpm2_pkcs11.so; pin-source=file:/etc/ota/pkcs11-pin\u0026#39;\u0026gt;, ...} Apr 23 05:36:45 iot-gateway rauc[1193]: The private key ID is not a valid PKCS#11 URI Apr 23 05:36:45 iot-gateway rauc[1193]: The PKCS#11 URI format is defined by RFC7512 Apr 23 05:36:45 iot-gateway rauc[1193]: The private key was not found at: pkcs11:token=iotgw;object=rauc-client-key;type=private; module-path=/usr/lib/pkcs11/libtpm2_pkcs11.so;pin-source=file:/etc/ota/pkcs11-pin Apr 23 05:36:45 iot-gateway rauc[1193]: PKCS11_get_private_key returned NULL Apr 23 05:36:45 iot-gateway rauc[1090]: Installation 7836ac74 failed: Failed to stream bundle: failed to configure streaming: server not responding The server not responding line is RAUC\u0026rsquo;s generic failure when the streaming subprocess can\u0026rsquo;t configure the connection. The actual failure is two lines earlier in the subprocess output — key loading failed before any TLS handshake started.\nTwo PIDs, two error sources The two PIDs in that log matter. rauc[1090] is the main RAUC daemon. rauc[1193] is the NBD streaming subprocess it spawned. RAUC\u0026rsquo;s HTTPS streaming path runs in a separate sandboxed process that drops privileges to the user named in system.conf\u0026rsquo;s sandbox-user — on this gateway, the dedicated ota user (UID 997, GID 996). The subprocess inherits the environment of rauc.service, which includes the 10-tpm2-pkcs11-store.conf drop-in:\nApr 23 06:14:43 iot-gateway rauc-nbd[1827]: nbd server running as UID 997, GID 996 Privilege drop worked. Module path environment was inherited. The failure was in URI parsing, inside libp11, after the subprocess had everything it needed to load the key.\nThe diagnostic point: the lines The private key ID is not a valid PKCS#11 URI and The PKCS#11 URI format is defined by RFC7512 come from libp11\u0026rsquo;s URI parser, not from RAUC. RAUC reports the top-level failure (server not responding); libp11 reports the actual reason at the subprocess PID. If you read only the daemon\u0026rsquo;s output, you never see the reason.\nFirst fix attempt: RFC7512 query form RFC7512 partitions PKCS#11 URI attributes into two groups: path attributes (token, object, type, id, slot\u0026hellip;) and query attributes (pin-source, pin-value, module-name\u0026hellip;). A reasonable first hypothesis was that placement was the issue — pin-source and module-path are both query-shaped attributes, and the URI as written had them mixed in with path attributes after a semicolon, not after a ?. So:\npkcs11:token=iotgw;object=rauc-client-key;type=private?module-path=/usr/lib/pkcs11/libtpm2_pkcs11.so\u0026amp;pin-source=file:/etc/ota/pkcs11-pin Same error. Same not a valid PKCS#11 URI. Placement wasn\u0026rsquo;t the problem.\nRoot cause: libp11 and p11-kit disagree on module-path module-path is not part of RFC7512 at all. It is a p11-kit extension, defined in p11-kit\u0026rsquo;s URI documentation, that lets a URI itself name a PKCS#11 module file when no module-loading framework is configured. p11-kit accepts it. libp11 does not — it rejects URIs containing attributes it doesn\u0026rsquo;t recognise, with the exact error strings seen above.\nlibp11\u0026rsquo;s accepted attribute set is the RFC7512 standard query attributes (pin-source and pin-value are both defined there, in §2.4). module-path is not. RAUC\u0026rsquo;s streaming subprocess loads the key via OpenSSL\u0026rsquo;s ENGINE interface with libp11 as the engine, so the strict parser is the one that runs at install time.\nThe kicker: with PKCS11_MODULE_PATH already exported in rauc.service.d/10-tpm2-pkcs11-store.conf, libp11 already knows which module to load. module-path in the URI was both unsupported by the parser and redundant given the environment.\nThe fix Strip module-path from the URI. Use only standard RFC7512 attributes plus pin-source:\ntls-key=pkcs11:token=iotgw;object=rauc-client-key;type=private;pin-source=file:/etc/ota/pkcs11-pin From the validated run:\nApr 23 06:14:43 iot-gateway rauc-nbd[1827]: nbd server running as UID 997, GID 996 Apr 23 06:14:43 iot-gateway rauc-nbd[1827]: nbd server received configuration: { \u0026#39;key\u0026#39;: \u0026lt;\u0026#39;pkcs11:token=iotgw;object=rauc-client-key;type=private; pin-source=file:/etc/ota/pkcs11-pin\u0026#39;\u0026gt;, ...} Apr 23 06:14:43 iot-gateway rauc-nbd[1827]: nbd server received total size 555044310 Apr 23 06:18:25 iot-gateway rauc[1813]: Installation 87af83d1 succeeded No key loading errors. The NBD subprocess opened the TPM2-backed key, established the mTLS connection, and the install completed. pin-source=file: reads the PIN from /etc/ota/pkcs11-pin at runtime — the PIN does not appear in system.conf, in environment variables, or in the journal.\nThe diagnostic gap The PKCS#11 preflight check that runs as part of OTA provisioning passed cleanly with the broken URI:\nApr 23 05:31:08 iot-gateway ota-certs[575]: [INFO] PKCS#11 preflight passed for token=\u0026#39;iotgw\u0026#39; key=\u0026#39;rauc-client-key\u0026#39; It was never going to catch this, and the reason is worth sitting with. The preflight signs a probe with pkcs11-tool, addressing the key by token and object label with the module passed explicitly:\npkcs11-tool --module /usr/lib/pkcs11/libtpm2_pkcs11.so \\ --token-label iotgw --login --pin \u0026#34;$pin\u0026#34; \\ --label rauc-client-key --sign --mechanism RSA-PKCS ... That proves the key exists, the PIN unlocks it, and the TPM will sign. What it never touches is the streaming URI. module-path lives only in that URI, and the URI is parsed by exactly one thing: libp11, inside RAUC\u0026rsquo;s streaming subprocess, at install time. A passing preflight gives no signal about whether the streaming path can load the same key — not because it runs a more forgiving parser, but because it never parses that URI at all. The install wrapper doesn\u0026rsquo;t close the gap either: its connectivity preflight skips the key check entirely for a pkcs11: tls-key, deferring to RAUC.\nThe fix has two shapes: validate the URI against the actual streaming consumer at provisioning time, or refuse to emit URIs the streaming consumer can\u0026rsquo;t parse. The recipe-level guard below does the second — cheaper, and it catches the bug at build time instead of first-boot.\nHow it\u0026rsquo;s wired in the layer The toggle and the URI live in the distro include (meta-iot-gateway/conf/distro/include/iotgw-common.inc):\nIOTGW_RAUC_STREAMING_KEY_MODE ?= \u0026#34;file\u0026#34; # file | pkcs11 IOTGW_RAUC_PKCS11_BACKEND ?= \u0026#34;tpm2\u0026#34; # Backend-specific presets. The mode/backend selector resolves the bare # IOTGW_RAUC_PKCS11_TLS_KEY and IOTGW_RAUC_PKCS11_MODULE from the active # backend\u0026#39;s *_TPM2 values below. IOTGW_RAUC_PKCS11_TLS_KEY_TPM2 ?= \u0026#34;pkcs11:token=iotgw;object=rauc-client-key;type=private;pin-source=file:/etc/ota/pkcs11-pin\u0026#34; IOTGW_RAUC_PKCS11_MODULE_TPM2 ?= \u0026#34;/usr/lib/pkcs11/libtpm2_pkcs11.so\u0026#34; IOTGW_TPM2_PKCS11_STORE ?= \u0026#34;/var/lib/tpm2_pkcs11\u0026#34; The rauc-conf recipe selects between device.key and the PKCS#11 URI based on the mode, then substitutes into the system.conf template:\nIOTGW_RAUC_STREAMING_TLS_KEY = \u0026#34;${@bb.utils.contains(\u0026#39;IOTGW_RAUC_STREAMING_KEY_MODE_EFFECTIVE\u0026#39;, \u0026#39;pkcs11\u0026#39;, d.getVar(\u0026#39;IOTGW_RAUC_PKCS11_TLS_KEY\u0026#39;) or \u0026#39;\u0026#39;, \u0026#39;/etc/ota/device.key\u0026#39;, d)}\u0026#34; # ... sed -e \u0026#34;s|@TLS_KEY@|${IOTGW_RAUC_STREAMING_TLS_KEY}|g\u0026#34; \\ ${WORKDIR}/iotgw-system.conf \u0026gt; ${D}${sysconfdir}/rauc/system.conf The template is otherwise identical between modes:\n[streaming] sandbox-user=ota tls-cert=/etc/ota/device.crt tls-key=@TLS_KEY@ tls-ca=/etc/ota/ca.crt In pkcs11 mode, @TLS_KEY@ becomes a PKCS#11 URI. In file mode, it stays /etc/ota/device.key.\nThe rauc_%.bbappend installs a systemd drop-in under /etc/systemd/system/rauc.service.d/10-tpm2-pkcs11-store.conf only when IOTGW_RAUC_PKCS11_USES_TPM2 = \u0026quot;1\u0026quot;:\n[Service] Environment=TPM2_PKCS11_STORE=@IOTGW_TPM2_PKCS11_STORE@ Environment=PKCS11_MODULE_PATH=@IOTGW_RAUC_PKCS11_MODULE@ PKCS11_MODULE_PATH is what libp11 reads at runtime to load libtpm2_pkcs11.so. It is also what makes module-path redundant in the URI — set it here, and the parser never needs to be told which module to use.\nThe same bbappend handles the upstream meson option. RAUC\u0026rsquo;s rauc.inc defaults pkcs11_engine to true on 1.15.x; the bbappend explicitly disables it when the streaming key is file-based, to avoid pulling libp11 into builds that don\u0026rsquo;t use PKCS#11:\nEXTRA_OEMESON:append = \u0026#34;${@bb.utils.contains(\u0026#39;IOTGW_RAUC_STREAMING_KEY_MODE_EFFECTIVE\u0026#39;, \u0026#39;pkcs11\u0026#39;, \u0026#39;\u0026#39;, \u0026#39; -Dpkcs11_engine=false\u0026#39;, d)}\u0026#34; RDEPENDS:${PN}-service:append = \u0026#34;${@bb.utils.contains(\u0026#39;IOTGW_RAUC_STREAMING_KEY_MODE_EFFECTIVE\u0026#39;, \u0026#39;pkcs11\u0026#39;, \u0026#39; openssl-engines libp11\u0026#39;, \u0026#39;\u0026#39;, d)}\u0026#34; In PKCS#11 mode, RAUC links against openssl-engines and libp11. In file mode, neither package is in the image.\nThe build-time guard sits in rauc-conf-iotgw_1.0.bb. It refuses to render system.conf if the URI contains module-path=, or if it lacks both pin-source and pin-value:\ndo_install() { if [ \u0026#34;${IOTGW_RAUC_STREAMING_KEY_MODE_EFFECTIVE}\u0026#34; = \u0026#34;pkcs11\u0026#34; ] \u0026amp;\u0026amp; \\ [ \u0026#34;${IOTGW_RAUC_PKCS11_USES_TPM2}\u0026#34; = \u0026#34;1\u0026#34; ]; then case \u0026#34;${IOTGW_RAUC_STREAMING_TLS_KEY}\u0026#34; in *\u0026#34;module-path=\u0026#34;*) echo \u0026#34;ERROR: TPM2 PKCS#11 streaming key URI must not include module-path when PKCS11_MODULE_PATH service env is used.\u0026#34; \u0026gt;\u0026amp;2 echo \u0026#34; Current: ${IOTGW_RAUC_STREAMING_TLS_KEY}\u0026#34; \u0026gt;\u0026amp;2 exit 1 ;; esac case \u0026#34;${IOTGW_RAUC_STREAMING_TLS_KEY}\u0026#34; in *\u0026#34;pin-source=file:/etc/ota/pkcs11-pin\u0026#34;*|*\u0026#34;pin-value=\u0026#34;*) ;; *) echo \u0026#34;ERROR: TPM2 PKCS#11 streaming key URI must include pin-source=file:/etc/ota/pkcs11-pin (preferred) or pin-value=\u0026lt;PIN\u0026gt;.\u0026#34; \u0026gt;\u0026amp;2 exit 1 ;; esac fi # ... sed substitution into system.conf } Around 15 lines, and it catches the only failure mode in this stack that has no clean runtime fallback.\nThe migration shipped in PR #49 — \u0026ldquo;ota/rauc: gated PKCS#11+crypt OTA flow with TPM2 streaming fix\u0026rdquo;. The PKCS#11 path is opt-in; file-key streaming remains the default. To switch a build to TPM2-backed streaming:\n# kas/local.yml local_conf_header: iotgw: | IOTGW_RAUC_STREAMING_KEY_MODE = \u0026#34;pkcs11\u0026#34; IOTGW_RAUC_PKCS11_BACKEND = \u0026#34;tpm2\u0026#34; IOTGW_RAUC_PKCS11_TLS_KEY = \u0026#34;pkcs11:token=iotgw;object=rauc-client-key;type=private;pin-source=file:/etc/ota/pkcs11-pin\u0026#34; Anything else in that URI (notably module-path=) fails the recipe.\nA wider lesson on daemons and subprocesses When a daemon spawns a sandboxed subprocess and the subprocess fails, the daemon\u0026rsquo;s top-level error is often a generic timeout. RAUC reports server not responding whether the failure was a real network problem, a TLS handshake error, a sandbox setup failure, or — as here — a parse error in a URI the subprocess never got to use. The first place to look in journalctl -u \u0026lt;unit\u0026gt; is not the line with the word failed; it is the highest PID in the same time window. The shortest-lived process is usually the one that actually broke.\nThe module-path attribute was one of two walls this migration hit. The other showed up before it and cut deeper: the sandboxed streaming helper couldn\u0026rsquo;t reach the TPM at all, and Unable to load module (null) was the only clue it left. That\u0026rsquo;s the next part in this series.\nUpdate (v0.5.0, July 2026). This was the layout as of April 2026. The TPM2 PKCS#11 token store has since moved from /var/lib/tpm2_pkcs11 to /data/tpm2_pkcs11, so the OTA identity survives the switch to a volatile /var and persists across A/B updates. The URI, the module path, and the module-path/pin-source handling are unchanged.\nReferences RAUC — Streaming Bundles — [streaming] configuration, the NBD subprocess model RAUC — PKCS#11 Support RFC 7512 — The PKCS#11 URI Scheme — defines the standard attribute set; pin-source and pin-value are in §2.4 libp11 — engine_pkcs11 — the URI parser used by OpenSSL\u0026rsquo;s ENGINE backend p11-kit URI specification — defines module-path as a p11-kit extension, not RFC7512 tpm2-pkcs11 — TPM2 PKCS#11 module AWS IoT Greengrass v2 — Hardware security integration — equivalent PKCS#11 plugin pattern in a different stack PR #49 in rpi5-iot-gateway — gated PKCS#11+crypt OTA flow with the TPM2 streaming fix and recipe guard ","permalink":"https://umair-as.github.io/blog/posts/rauc-streaming-key-tpm2-pkcs11/","summary":"Migrating RAUC\u0026rsquo;s mTLS client key from a file on disk to a TPM2-backed PKCS#11 URI. The recipe wiring was straightforward; the URI was not. module-path is a p11-kit URI extension that libp11 — the parser RAUC\u0026rsquo;s streaming path actually uses — rejects outright.","title":"Moving RAUC's HTTPS Streaming Key into TPM2 via PKCS#11"},{"content":"Moving RAUC\u0026rsquo;s streaming key into the TPM took two debugging sessions on two unrelated failures. The first part was a module-path attribute the URI parser rejected. This is the other one, and it cut deeper: the sandboxed streaming user couldn\u0026rsquo;t reach the TPM at all, and the error said almost nothing about why.\nThe streaming OTA setup had been working for months: RAUC\u0026rsquo;s rauc-nbd subprocess authenticates to the bundle server over mTLS, streams the bundle directly into the inactive slot, and hands off to RAUC for signature verification and slot activation. The TLS client key lived at /etc/ota/device.key, file permissions handled access, and sandbox-user=ota in system.conf kept the streaming helper running as a dedicated system user with no shell, no home directory, and minimal privileges.\nMoving that key into the TPM was the next step — store the private key in a TPM2 key object on the SLB9672 via tpm2-pkcs11, and reference it through a PKCS#11 URI instead of a file path. One line in system.conf changed:\n[streaming] sandbox-user=ota tls-cert=/etc/ota/device.crt tls-key=pkcs11:token=iotgw;object=rauc-client-key;type=private tls-ca=/etc/ota/ca.crt Same bundle, same server, same network. The install failed on the first attempt:\nApr 21 12:26:33 iot-gateway rauc-nbd[1887]: nbd server running as UID 985, GID 984 Apr 21 12:26:33 iot-gateway rauc-nbd[1887]: nbd server configuring for URL: https://192.168.0.193:8443/bundles/iot-gw-image-dev-bundle-full-fit.raucb Apr 21 12:26:33 iot-gateway rauc[1887]: Unable to load module (null) Apr 21 12:26:33 iot-gateway rauc[1887]: PKCS11_get_private_key returned NULL Apr 21 12:26:33 iot-gateway rauc-nbd[1887]: request failed: failed to load private key from crypto engine (retrying 1/5) Five retries, identical output each time. The parent process gave up:\nApr 21 12:26:38 iot-gateway rauc[632]: Installation 348ed464 failed: Failed to stream bundle \\ https://192.168.0.193:8443/bundles/iot-gw-image-dev-bundle-full-fit.raucb: \\ failed to configure streaming: server not responding The first wrong read Unable to load module (null) is from libp11, the OpenSSL engine RAUC uses for PKCS#11 operations. When libp11 gets a null module path, the engine never opened a token or attempted a key lookup — no CKR_* error, no PIN prompt, no token-not-found. The PKCS#11 module itself didn\u0026rsquo;t load.\nThe setup already had a drop-in for rauc.service:\n[Service] Environment=RAUC_PKCS11_MODULE=/usr/lib/pkcs11/libtpm2_pkcs11.so The RAUC docs mention RAUC_PKCS11_MODULE as a way to name the PKCS#11 module when you can\u0026rsquo;t use p11-kit. The error looked like a module-discovery failure, so that env var was the obvious first place to look. I removed the drop-in, restarted rauc, ran the install again. Same PID, same (null), same five retries.\nThat killed the env var hypothesis — and reading the docs more carefully showed why it was never going to matter. RAUC_PKCS11_MODULE is consulted only on RAUC\u0026rsquo;s signing path; the reference marks it \u0026ldquo;signing only\u0026rdquo;. The streaming subprocess loads the client key through libp11\u0026rsquo;s own module discovery and never reads that variable. The drop-in had been doing nothing from the start — a signing knob aimed at a streaming problem.\nWhat the sandbox user actually sees If p11-kit is involved and libp11 still gets null back, p11-kit is loading the module but getting nothing usable from it. The right move was to reproduce exactly what rauc-nbd does: run p11-kit as the ota user and watch what happens.\n$ su -s /bin/sh ota -c \u0026#39;p11-kit list-modules 2\u0026gt;\u0026amp;1\u0026#39; ERROR: Could not mkdir \u0026#34;/nonexistent/.tpm2_pkcs11\u0026#34;, error: No such file or directory ERROR: Cannot open database: unable to open database file WARNING: ESYSDB backend was not initialized. WARNING: Getting tokens from fapi backend failed. ERROR:tcti:tcti-device.c:454:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: Permission denied ERROR:tcti:tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 ERROR:tcti:tcti-device.c:454:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: Permission denied ERROR:tcti:tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 ERROR:tcti:tctildr-dl.c:269:tctildr_get_default() No standard TCTI could be loaded ERROR:tcti:tctildr.c:430:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI ERROR: Could not initialize tpm ctx: 0x5 Two independent failures, both happening before the module produces anything usable.\nNo TPM device access. tpm2-tss — the library tpm2-pkcs11 uses to talk to the hardware — tries device nodes in a fixed order: /dev/tpmrm0 (the resource manager, preferred for all userspace use), then /dev/tpm0 (direct access), then a software TPM socket on 127.0.0.1:2321. Their permissions on this system:\ncrw-rw---- 1 root iotgwtpm 251, 65536 Apr 21 12:19 /dev/tpmrm0 crw------- 1 root root 10, 224 Apr 21 12:19 /dev/tpm0 /dev/tpmrm0 requires membership in iotgwtpm. /dev/tpm0 is root-only. The software TPM isn\u0026rsquo;t running. The ota user\u0026rsquo;s actual group membership:\nuid=985(ota) gid=984(ota) groups=984(ota) Not in iotgwtpm. Never had been. tpm2-tss exhausts all three paths, fails to open any device, and returns an error. The PKCS#11 module cannot reach the hardware.\nNo database path. tpm2-pkcs11 maintains a SQLite database of provisioned tokens and key objects. Without a TPM2_PKCS11_STORE environment variable pointing elsewhere, it defaults to ~/.tpm2_pkcs11/. The ota user in /etc/passwd:\nota:x:985:984:OTA Update Daemon:/nonexistent:/bin/false Home is /nonexistent — the standard placeholder for system users created with --no-create-home. The directory does not exist and is never created. tpm2-pkcs11 attempts to mkdir ~/.tpm2_pkcs11/ inside it, fails immediately, and exits without an open store.\nEither failure alone would have been fatal. Together, they leave the module unable to talk to the hardware and with nowhere to load token metadata from. p11-kit loads libtpm2_pkcs11.so successfully, but the module\u0026rsquo;s initialisation fails silently and returns no slots, no tokens, no usable state. libp11 receives nothing and reports Unable to load module (null) — not because the shared library is missing, but because the module initialised into an empty, broken state and returned null.\nWhy this was invisible with the file key The two access paths look like this side by side:\n# file key rauc (root) → rauc-nbd (ota, UID 985) → open /etc/ota/device.key # PKCS#11 key rauc (root) → rauc-nbd (ota, UID 985) → p11-kit → tpm2_pkcs11 → /dev/tpmrm0 → ~/.tpm2_pkcs11/store.db For the file key, the sandbox user needed read permission on one file — and it had that. The sandboxing was correct and working. Switching to PKCS#11 silently expanded the access requirements: a device node gated by group membership, a writable path for library-managed state. Neither was wired to the ota user because when the file key was the only option, neither was needed. The sandbox didn\u0026rsquo;t break — the new key backend\u0026rsquo;s prerequisites were just never accounted for.\nThe fix Two changes, each closing one of the two failures.\nGive ota membership in iotgwtpm. That group is created by the iotgw-tpm-policy recipe specifically to gate /dev/tpmrm0; adding ota to it hands tpm2-tss a device node it can open, which unblocks the TCTI chain. The ota user is built by the OTA layer\u0026rsquo;s useradd recipe, so this is a change to the set of groups it\u0026rsquo;s created with — nothing more.\nPoint TPM2_PKCS11_STORE at a real directory in the rauc.service drop-in:\n[Service] Environment=TPM2_PKCS11_STORE=/var/lib/tpm2_pkcs11 That overrides the default ~/.tpm2_pkcs11/, which for a /nonexistent-home system user can never be created. /var/lib/tpm2_pkcs11 is a system-owned directory that exists and is reachable by the service user. The token and key object still have to be provisioned there with tpm2_ptool — a one-time setup step, not a permissions problem.\nWith both access issues resolved, the module initialises cleanly and rauc\u0026rsquo;s error shifts from Unable to load module (null) to a token-lookup failure: a different error, a solvable one, and proof that the module is finally working.\nThe one-line documentation gap The RAUC docs have one sentence covering this:\nWhen using TLS client certificates, you need to ensure that the key (or PKCS#11 token) is accessible to the streaming sandbox user.\nFor a file key, \u0026ldquo;accessible\u0026rdquo; means read permission on one file. For a TPM-backed PKCS#11 token it means: membership in the group that gates the TPM device node, a reachable path for the PKCS#11 library\u0026rsquo;s token database, and a provisioned token with the key object present. The surface area is significantly larger, and none of it appears in the error output when something is missing — just a null module pointer and a generic load failure, with no pointer to which prerequisite was not met.\nUpdate (July 2026). This was the layout in April 2026. Since then the OTA user moved to a dedicated iotgw-ota-user recipe, its iotgwtpm membership is applied as a rootfs post-process step rather than through useradd directly, and the token store moved from /var/lib/tpm2_pkcs11 to /data/tpm2_pkcs11 so the OTA identity survives the switch to a volatile /var. The access requirements the post describes are unchanged.\nReferences RAUC — Streaming Bundles — the sandboxed rauc-nbd streaming subprocess and sandbox-user RAUC — PKCS#11 Support — RAUC_PKCS11_MODULE (signing only) and p11-kit discovery tpm2-pkcs11 — the PKCS#11 module, its SQLite token store, and TPM2_PKCS11_STORE tpm2-tss — the TCTI device-probe order (/dev/tpmrm0 → /dev/tpm0 → swtpm socket) libp11 — engine_pkcs11 — the OpenSSL PKCS#11 engine RAUC\u0026rsquo;s streaming path loads the key through Gateway Yocto BSP (open source): github.com/umair-as/rpi5-iot-gateway — the ota user, iotgw-tpm-policy, and RAUC streaming wiring under meta-iot-gateway/ ","permalink":"https://umair-as.github.io/blog/posts/rauc-pkcs11-tpm-streaming-sandbox/","summary":"Switching RAUC\u0026rsquo;s streaming mTLS client key from a file path to a PKCS#11 URI produced a one-line error with no obvious cause. The real problem was not the module path, not the engine, and not a missing PIN — it was the sandbox user\u0026rsquo;s group membership and what tpm2-pkcs11 needs to initialise.","title":"RAUC Streaming OTA: When the Sandbox User Can't Reach the TPM"},{"content":"Telegraf is a good fit for edge observability: one static binary, a plugin system that covers system metrics, MQTT, Modbus RTU, and InfluxDB output in a single config file. Adding it to the rpi5-iot-gateway — a Yocto-built RPi5 system running a Thread border router, MQTT broker, and RAUC OTA stack — looked like a straightforward packaging task.\nNo Telegraf recipe exists in meta-openembedded. The OE layer index has entries in four layers — meta-luneos, meta-webosose, meta-ossystems-base, and meta-influx — but none go above 1.21.4, and Telegraf 1.31 changed plugin registration in a way that makes those recipe structures wrong, not just stale. That meant writing from scratch, which surfaced two non-obvious build issues.\nThe harder problem was credentials. The image rootfs is a squashfs image replaced wholesale on every RAUC OTA update — any credential baked into the rootfs is either shared across every device running that build, or silently wiped on the next upgrade. The standard fallback, EnvironmentFile=, exposes secrets in /proc/$PID/environ and core dumps — a trade-off that\u0026rsquo;s acceptable on a server you can audit, not on a fleet of devices you can\u0026rsquo;t. This post covers the recipe, the two build failures, and the credential model that the image layout actually requires.\nFour layers, none past 1.21.4 Telegraf is a Go module, so the recipe inherits go-mod. The binary entry point is cmd/telegraf. A dedicated system user handles isolation:\nGO_IMPORT = \u0026#34;github.com/influxdata/telegraf\u0026#34; PV = \u0026#34;1.31.0\u0026#34; SRCREV = \u0026#34;fbfaba054e62413b6a0a90372281e687d9ff1238\u0026#34; SRC_URI = \u0026#34; \\ git://github.com/influxdata/telegraf.git;protocol=https;nobranch=1;destsuffix=${BPN}-${PV}/src/${GO_IMPORT} \\ file://telegraf.service \\ file://telegraf.conf \\ file://telegraf.tmpfiles \\ \u0026#34; inherit go-mod systemd useradd GO_INSTALL = \u0026#34;${GO_IMPORT}/cmd/telegraf\u0026#34; GROUPADD_PARAM:${PN} = \u0026#34;-f --system telegraf; -f --system dialout\u0026#34; USERADD_PARAM:${PN} = \u0026#34;--system -d /var/lib/telegraf -m -s /sbin/nologin \\ --gid telegraf --groups dialout telegraf\u0026#34; That gets the build wired. First bitbake telegraf failed immediately.\n300 plugins, two conflicting symbols runtime/cgo(.text): symbol __cgoenqueue defined in both vendor/github.com/Azure/azure-kusto-go/... vendor/github.com/snowflakedb/gosnowflake/... duplicate symbol error Two vendored packages — azure-kusto-go and gosnowflake — both expose a C symbol __cgoenqueue generated by their shared cel-go/antlr4 dependency. The linker sees two definitions and stops.\nThe root cause is how Telegraf 1.31 changed plugin registration. Before 1.31, each plugin category had a single all.go that imported every plugin unconditionally. From 1.31 onward, each plugin has its own file gated by a build constraint:\n//go:build !custom || inputs.cpu package all import _ \u0026#34;github.com/influxdata/telegraf/plugins/inputs/cpu\u0026#34; Without the custom tag, !custom is true and all 300+ plugins compile. Two of them drag in conflicting vendored symbols via the same generated C code.\nTelegraf ships a custom_builder tool1 that reads a config file and prints exactly which build tags are needed. Running --dry-run against the gateway telegraf.conf gave the tag list:\nTELEGRAF_PLUGIN_TAGS = \u0026#34;custom,inputs.cpu,inputs.disk,inputs.internal,\\ inputs.mem,inputs.modbus,inputs.mqtt_consumer,inputs.net,inputs.processes,\\ inputs.system,inputs.temp,outputs.influxdb,parsers.json,secretstores.systemd\u0026#34; GOBUILDFLAGS:append = \u0026#34; -tags ${TELEGRAF_PLUGIN_TAGS}\u0026#34; Adding a plugin later means appending its tag here and adding the corresponding stanza to telegraf.conf. custom_builder --dry-run is the source of truth — don\u0026rsquo;t guess.\nThe same linker, a different package pair Build tags cleared the first error. The second came from go.bbclass defaults.\ngo.bbclass appends -buildmode=pie by default.2 Telegraf\u0026rsquo;s upstream Makefile builds with CGO_ENABLED=0 and no PIE — all selected plugins in this profile are pure Go. Enabling PIE against CGO-disabled code caused the Go generic linker to produce another duplicate symbol error, from a different package pair in the antlr4/cel-go dependency tree. Same pattern: two packages generating the same C symbol from shared generated code. The symptom kept surfacing from different package combinations until the flags matched upstream.\nMatching upstream\u0026rsquo;s release build:\nCGO_ENABLED = \u0026#34;0\u0026#34; GOBUILDFLAGS:remove = \u0026#34;-buildmode=pie\u0026#34; # -linkshared requires CGO — clear it for a pure-Go build GO_LINKSHARED = \u0026#34;\u0026#34; One more go.bbclass default to override: do_compile[network] = \u0026quot;0\u0026quot;. Go module resolution for this recipe happens at compile time rather than fully during do_fetch, so network access is needed:\ndo_compile[network] = \u0026#34;1\u0026#34; After these changes the binary builds cleanly.\nNeither the rootfs nor the environment The rootfs constraint is straightforward: squashfs is replaced wholesale on every OTA update, so anything baked in either gets wiped on upgrade or is identical across every device running that image. Neither is acceptable for service credentials.\nEnvironment variables are ruled out too, and less obviously so. If you reach for EnvironmentFile=-/etc/default/telegraf with credentials stored as KEY=value pairs, they end up readable in /proc/$PID/environ:\nMQTT_PASSWORD=... INFLUXDB_PASSWORD=... Any process running as root — audit daemons, monitoring agents, anything with ptrace access — can read them. They also appear in core dumps. An Environment= drop-in in a systemd unit has the same exposure: the value serialises into the unit state under /run/systemd/ and is visible via systemctl show.\nThe solution is LoadCredential= (systemd 250+)3, which creates a per-unit tmpfs at service activation — isolated from the environment, not visible in /proc, not in argv. Combined with Telegraf 1.28\u0026rsquo;s secretstores.systemd plugin4, credentials never appear in the config file either. A provisioning script fills the credstore at first boot; the values survive OTA because /etc is an overlay fs with its upper layer on the persistent data partition (mmcblk0p5, ext4). OTA replaces the squashfs lower layer but leaves the data partition intact.\nThe full mechanics of LoadCredential= — how the tmpfs is structured, the permission model, how to provision it at manufacturing time, and how it composes with SetCredentialEncrypted= — are worth a dedicated post. That\u0026rsquo;s coming. For now, the relevant config:\n# telegraf.service (excerpt) LoadCredential=mqtt_username:/etc/credstore/telegraf.service.mqtt_username LoadCredential=mqtt_password:/etc/credstore/telegraf.service.mqtt_password LoadCredential=influxdb_username:/etc/credstore/telegraf.service.influxdb_username LoadCredential=influxdb_password:/etc/credstore/telegraf.service.influxdb_password # telegraf.conf (excerpt) [[secretstores.systemd]] id = \u0026#34;systemd\u0026#34; prefix = \u0026#34;\u0026#34; [[outputs.influxdb]] urls = [\u0026#34;${INFLUXDB_URL}\u0026#34;] database = \u0026#34;${INFLUXDB_DATABASE}\u0026#34; username = \u0026#34;@{systemd:influxdb_username}\u0026#34; password = \u0026#34;@{systemd:influxdb_password}\u0026#34; [[inputs.mqtt_consumer]] servers = [\u0026#34;tcp://localhost:1883\u0026#34;] username = \u0026#34;@{systemd:mqtt_username}\u0026#34; password = \u0026#34;@{systemd:mqtt_password}\u0026#34; topics = [\u0026#34;sensors/+/data\u0026#34;] data_format = \u0026#34;json\u0026#34; INFLUXDB_URL and INFLUXDB_DATABASE are non-secret defaults from EnvironmentFile=-/etc/default/iotgw-observability. The @{systemd:key} references resolve at runtime from $CREDENTIALS_DIRECTORY.\nThe guard that ran as the wrong user The credstore files ship as zero-byte placeholders, mode 0600, owned root:root. Before provisioning runs on a fresh device, Telegraf must not start — bad credentials cause a restart loop and journal noise.\nThe first cut used ExecCondition= with test -s checks in [Service]. It looked correct. The service skipped cleanly on a fresh device — exactly the behaviour we wanted. It was only when testing the fix on device that test -s kept returning failure even after the credstore files were populated. The reason: ExecCondition= runs under the service\u0026rsquo;s User= — in this case telegraf. The credstore directory is 0700 root:root. The telegraf user has no execute permission on it, so test -s returns EACCES regardless of file content. The guard worked initially only because the files happened to be empty — both empty files and inaccessible paths look the same to test -s.\nConditionFileNotEmpty=5 is the right directive — it runs in PID 1\u0026rsquo;s context (root), is designed exactly for this check, and belongs in [Unit] not [Service]. The fix:\n--- a/telegraf.service +++ b/telegraf.service @@ -3,6 +3,10 @@ Description=Telegraf metrics agent After=network.target mosquitto.service Wants=mosquitto.service +ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_username +ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_password +ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_username +ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_password StartLimitIntervalSec=5min @@ LoadCredential=influxdb_password:/etc/credstore/telegraf.service.influxdb_password -# Skip service start until required credentials are non-empty. -ExecCondition=/bin/sh -c \u0026#39;test -s /etc/credstore/telegraf.service.mqtt_username\u0026#39; -ExecCondition=/bin/sh -c \u0026#39;test -s /etc/credstore/telegraf.service.mqtt_password\u0026#39; -ExecCondition=/bin/sh -c \u0026#39;test -s /etc/credstore/telegraf.service.influxdb_username\u0026#39; -ExecCondition=/bin/sh -c \u0026#39;test -s /etc/credstore/telegraf.service.influxdb_password\u0026#39; ExecStart=/usr/bin/telegraf --config /etc/telegraf/telegraf.conf [Unit] ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_username ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_password ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_username ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_password When any file is absent or empty, systemd marks the unit condition-failed and skips it cleanly — no Restart= cycle, no journal errors:\ntelegraf.service: Skipped due to unmet condition checks. Condition check resulted in Telegraf metrics agent being skipped. Once the provisioning unit writes non-empty values and exits, the next start attempt clears all conditions.\nOn device With the fixed service file deployed, journal on first start with test credentials in the credstore:\nsystemd[1]: Started Telegraf metrics agent. telegraf[897]: Loading config: /etc/telegraf/telegraf.conf telegraf[897]: Available plugins: 11 inputs, 0 aggregators, 0 processors, 2 parsers, 1 outputs, 1 secret-stores telegraf[897]: Loaded inputs: cpu disk internal mem mqtt_consumer net processes system temp telegraf[897]: Loaded secretstores: systemd telegraf[897]: Loaded outputs: influxdb telegraf[897]: Tags enabled: host=iot-gateway role=iot-gateway site=plant-a telegraf[897]: [outputs.influxdb] When writing to [http://127.0.0.1:8086]: dial tcp 127.0.0.1:8086: connect: connection refused telegraf[897]: [telegraf] Error running agent: starting input inputs.mqtt_consumer: not Authorized Loaded secretstores: systemd confirms credentials were read from $CREDENTIALS_DIRECTORY. The MQTT not Authorized error means Telegraf reached the broker and attempted authentication with the credential — the secret store flow worked. The failures are expected: these are test credentials and the local InfluxDB instance isn\u0026rsquo;t running.\nsystemctl show confirms no credentials in the service environment:\nEnvironment= EnvironmentFiles=/etc/default/iotgw-observability (ignore_errors=yes) The environment file only carries non-secret defaults:\nINFLUXDB_URL=http://127.0.0.1:8086 INFLUXDB_DATABASE=gateway_data To verify nothing leaks into the process environment or /proc, running the same LoadCredential= setup as the telegraf service under User=telegraf:\n$ ls -la $CREDENTIALS_DIRECTORY dr-xr-x---+ 2 root root 80 Apr 23 05:35 . drwxr-xr-x 3 root root 60 Apr 23 05:35 .. -r--r-----+ 1 root root 14 Apr 23 05:35 mqtt_password -r--r-----+ 1 root root 14 Apr 23 05:35 mqtt_username $ env | grep -iE \u0026#34;pass|secret|mqtt|influx\u0026#34; (none) $ cat /proc/self/environ | tr \u0026#39;\\0\u0026#39; \u0026#39;\\n\u0026#39; | grep -iE \u0026#34;pass|secret|mqtt|influx\u0026#34; (none) Credentials exist only as files in the per-unit tmpfs — mode 0500 on the directory, 0400 on each file, owned by the service user. They are not in the environment, not in argv, and not in /proc/$PID/environ. The tmpfs is created at service activation and destroyed when the service exits.\nLoadCredential= requires systemd 250. The RPi5 BSP on Scarthgap ships systemd 255, so that\u0026rsquo;s not a constraint here. On BSPs below 250, SetCredential= (available from 247) bakes a value into the unit file at build time — no help for device-specific secrets. Below that there\u0026rsquo;s no clean answer from systemd alone.\nFull source for the recipe, service file, and tmpfiles config is in meta-iot-gateway.\nTelegraf custom_builder — upstream docs\u0026#160;\u0026#x21a9;\u0026#xfe0e;\ngo-mod.bbclass — OpenEmbedded source\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nsystemd.exec(5) — LoadCredential=\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nsecretstores.systemd plugin — Telegraf docs\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nsystemd.unit(5) — ConditionFileNotEmpty=\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://umair-as.github.io/blog/posts/telegraf-yocto-recipe/","summary":"No OE recipe for Telegraf exists upstream. What looked like a straightforward packaging task turned into a Go linker collision, a PIE/CGO conflict, and a credential problem that ruled out the obvious solution entirely.","title":"Packaging Telegraf for Yocto: Build Tags, a Linker Collision, and Credentials That Can't Live in the Rootfs"},{"content":"Part 1 ended with otbr-agent — the OpenThread Border Router daemon — running as a dedicated system user with three network capabilities and a systemd-analyze security score of 4.1 OK. The service unit carried a hardening block we didn\u0026rsquo;t explain:\nroot@iot-gateway:~# systemctl cat otbr-agent | grep -A20 \u0026#34;# Capabilities\u0026#34; # Capabilities and hardening AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE NoNewPrivileges=true ProtectSystem=full ProtectHome=true PrivateTmp=true PrivateDevices=false ProtectKernelTunables=true ProtectKernelModules=true ProtectKernelLogs=true ProtectControlGroups=true ProtectClock=true RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK RestrictNamespaces=true RestrictSUIDSGID=true LockPersonality=true This post walks through each directive: what it does, what it restricts, and where the tradeoffs are for a border router specifically. One — PrivateDevices — had to be set to false. The others held.\nSetup note: Validated on a Yocto-built image on a Raspberry Pi 5 (systemd 255, Linux 6.18). Kernel version, systemd version, and how your dependencies are built all affect which directives are safe to enable. The drop-in test pattern used throughout — write a /run/systemd/system/\u0026lt;service\u0026gt;.d/test.conf, daemon-reload, restart, verify — is how to check any directive on your own system before committing it.\nWhat systemd-analyze security actually measures Before walking the directives, it helps to understand what the tool is actually scoring.\nsystemd-analyze security \u0026lt;service\u0026gt; evaluates the service unit against a rubric of hardening options. Each missing or weakly-set directive contributes exposure points to the total score. Lower is better:\n0.0 → very secure 0.1 – 2.9 → OK 3.0 – 5.9 → medium exposure 6.0 – 8.9 → exposed 9.0+ → UNSAFE A ✓ means the directive is set and provides the expected restriction. A ✗ means it\u0026rsquo;s absent or set permissively — but ✗ is a signal, not a verdict. A border router that manages network interfaces will always have PrivateNetwork= marked ✗ because isolating the network namespace would break the thing it exists to do. That\u0026rsquo;s a deliberate gap, not a missed directive.\nThe useful question isn\u0026rsquo;t \u0026ldquo;how do I get to zero?\u0026rdquo; It\u0026rsquo;s \u0026ldquo;do I understand every ✗ in my output?\u0026rdquo;\nHere\u0026rsquo;s what it looks like on this gateway, abbreviated:\nroot@iot-gateway:~# systemd-analyze security otbr-agent.service --no-pager NAME DESCRIPTION EXPOSURE ✗ PrivateDevices= Service potentially has access to hardware devices 0.2 ✓ PrivateTmp= Service has no access to other software\u0026#39;s temporary files ✓ ProtectHome= Service has no access to home directories ✓ ProtectKernelLogs= Service cannot read from or write to the kernel log ring buffer ✓ ProtectKernelModules= Service cannot load or read kernel modules ✓ RestrictNamespaces=~user Service cannot create user namespaces ✓ RestrictNamespaces=~net Service cannot create network namespaces ✓ LockPersonality= Service cannot change ABI personality ✓ RestrictSUIDSGID= SUID/SGID file creation by service is restricted ✗ PrivateNetwork= Service has access to the host\u0026#39;s network 0.5 ✗ SystemCallFilter=~@privileged Service does not filter system calls 0.2 ✗ UMask= Files created by service are world-readable by default 0.1 ... → Overall exposure level for otbr-agent.service: 4.1 OK :-) Every ✗ is accounted for by the end of this post.\nThe hardening directives Walking through each directive above — what it restricts and where the tradeoffs are. The systemd.exec(5) man page has the full semantics; links to the relevant anchors are in each section.\nProtectSystem=full Remounts /usr, /boot, and /efi read-only inside the service\u0026rsquo;s mount namespace. The process can\u0026rsquo;t modify system binaries or bootloader files. strict would make the entire file system hierarchy read-only and would require a complete audit of every path the service writes to at runtime — that audit is deferred. full covers the primary threat surface without that cost.\nProtectHome=true No access to /home, /root, or /run/user. Standard for any daemon — a border router has no business reading home directories.\nPrivateTmp=true The service gets an isolated /tmp and /var/tmp namespace, separate from the host and all other services. Rules out a class of attacks where one service leaves crafted files in shared temp space for another to consume.\nPrivateDevices=false — the forced exception This is the one directive that had to stay off.\nPrivateDevices=true mounts a minimal private /dev that contains only a safe subset of device nodes: null, zero, full, random, urandom, tty. It explicitly excludes real hardware devices — including /dev/net/tun.\nOTBR creates the wpan0 Thread interface by opening /dev/net/tun and issuing a TUNSETIFF ioctl. Without the TUN device node, startup fails immediately:\niot-gateway otbr-agent[4235]: [NOTE]-AGENT---: Radio URL: spinel+hdlc+uart:///dev/otbr-rcp?uart-baudrate=460800 iot-gateway otbr-agent[4235]: [C] Platform------: Init() at hdlc_interface.cpp:153: No such file or directory iot-gateway systemd[1]: otbr-agent.service: Main process exited, code=exited, status=5/NOTINSTALLED iot-gateway systemd[1]: otbr-agent.service: Failed with result \u0026#39;exit-code\u0026#39;. Verified by enabling it via a /run/systemd/system/ drop-in on the device:\nmkdir -p /run/systemd/system/otbr-agent.service.d/ cat \u0026gt; /run/systemd/system/otbr-agent.service.d/test.conf \u0026lt;\u0026lt; EOF [Service] PrivateDevices=true EOF systemctl daemon-reload \u0026amp;\u0026amp; systemctl restart otbr-agent The RCP serial device (/dev/otbr-rcp, symlinked to /dev/ttyACM1) also disappears from the private /dev. Both failures happen at the same point.\nPrivateDevices=false is the correct setting for this configuration. The TUN device requirement is fundamental to how OTBR creates the wpan0 interface; no alternative was validated against this build.\nProtectKernelTunables=true Prevents the process from writing to /proc/sys at runtime — no sysctl modifications from within the service. This is safe for OTBR provided IPv6 forwarding is configured at boot, not by the daemon.\nOn this image, net.ipv6.conf.all.forwarding=1 is set via a sysctl.d fragment applied during early boot by systemd. OTBR never needs to set it at runtime. If your image relies on OTBR to configure forwarding dynamically, this directive needs to come off.\nProtectKernelModules, ProtectKernelLogs, ProtectControlGroups, ProtectClock All safe for OTBR without exception:\nProtectKernelModules=true — the service cannot load or unload kernel modules ProtectKernelLogs=true — no access to /proc/kmsg or the kernel log ring buffer ProtectControlGroups=true — the cgroup filesystem is read-only; the service can\u0026rsquo;t modify its own or any other cgroup ProtectClock=true — blocks clock_settime, settimeofday, and related calls; NTP is managed by systemd-timesyncd, not OTBR RestrictAddressFamilies= RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK Any socket() call for a family not in this list is blocked by a seccomp BPF rule that systemd generates and loads before ExecStart runs. The four allowed families come directly from the strace output in Part 1:\nFamily Used for AF_UNIX D-Bus, local IPC, the agent\u0026rsquo;s own Unix domain socket AF_INET / AF_INET6 REST API, TREL, multicast, ICMPv6 AF_NETLINK Route management via netlink sockets (NETLINK_ROUTE) AF_PACKET is notably absent — raw packet socket access at the Ethernet layer is not needed. Excluding it is correct and intentional.\nRestrictNamespaces=true Blocks unshare(), clone() with namespace flags, and setns() for all namespace types. The implementation is seccomp BPF — each namespace type gets its own filter program. This is where the 15 seccomp filters visible in Part 1\u0026rsquo;s /proc/status output come from:\nSeccomp: 2 Seccomp_filters: 15 RestrictNamespaces, RestrictAddressFamilies, LockPersonality, and RestrictSUIDSGID together account for all 15. None of them appear as SystemCallFilter= in systemd-analyze security output — they\u0026rsquo;re a side effect of sandbox directives, not an explicit syscall allowlist.\nLockPersonality=true, RestrictSUIDSGID=true LockPersonality=true prevents personality() syscall — the process can\u0026rsquo;t change the kernel execution domain (e.g. switch to Linux32 ABI emulation). RestrictSUIDSGID=true prevents the service from creating setuid/setgid files. Both are standard; neither has OTBR-specific caveats.\nNoNewPrivileges=true with ambient capabilities NoNewPrivileges=true ensures that any binary executed via execve() within the service — including child processes — cannot gain capabilities or privileges beyond what the bounding set allows. Combined with AmbientCapabilities, the interaction is:\nAmbient set grants capabilities at startup without setcap on the binary Bounding set caps what can ever be held NoNewPrivileges prevents execve from escalating beyond the bounding set With all three set consistently to the same capability list, this is belt-and-suspenders: the bounding set already prevents escalation, NoNewPrivileges makes that guarantee hold for child processes too. Correct default for any non-root service with ambient capabilities.\nStateDirectory= and ReadWritePaths= StateDirectory=thread ReadWritePaths=/var/lib/thread ReadWritePaths=/run StateDirectory=thread tells systemd to create /var/lib/thread owned by the otbr user before the service starts. OTBR persists Thread network dataset here across reboots.\nProtectSystem=full makes /usr, /boot, and /efi read-only — it does not restrict /run. RuntimeDirectory=otbr creates /run/otbr with correct ownership before the process starts, which is all the service actually needs. ReadWritePaths=/run is redundant here and can be dropped; it\u0026rsquo;s in the unit as a leftover from earlier iterations when the sandbox boundary wasn\u0026rsquo;t fully mapped.\nThe remaining gaps ✗ PrivateNetwork= Service has access to the host\u0026#39;s network 0.5 ✗ RestrictAddressFamilies=~AF_(INET|INET6) Service may allocate Internet sockets 0.3 ✗ SystemCallFilter=~@privileged Service does not filter system calls 0.2 ✗ SystemCallFilter=~@clock Service does not filter system calls 0.2 ✗ UMask= Files created by service are world-readable by default 0.1 PrivateNetwork= — a border router manages host network interfaces by design. Running in a private network namespace would isolate it from wlan0, wpan0, and the infrastructure network it exists to bridge. This gap cannot be closed.\nRestrictAddressFamilies=~AF_(INET|INET6) — AF_INET and AF_INET6 are in the allowed set because OTBR needs them. systemd\u0026rsquo;s scoring flags any service that can allocate internet sockets as exposed, regardless of whether those sockets are required. The allowed family list is as tight as it can be.\nSystemCallFilter= — an explicit syscall allowlist would tighten the remaining attack surface significantly. It\u0026rsquo;s not set here because building a correct allowlist requires profiling the full runtime syscall set — strace under normal operation, error paths, and restart scenarios. A wrong allowlist causes silent failures that are harder to debug than a missing hardening directive. Deferred to a future iteration.\nUMask= — low severity. Files created by the service default to world-readable. Setting UMask=0027 would restrict this. It\u0026rsquo;s a cosmetic open item; OTBR doesn\u0026rsquo;t create files that require tighter default permissions.\nThe score root@iot-gateway:~# systemd-analyze security otbr-agent.service --no-pager ... → Overall exposure level for otbr-agent.service: 4.1 OK :-) 4.1 OK. Every ✗ in the output is accounted for: one is a hard operational requirement (PrivateNetwork), one is a documented tradeoff (SystemCallFilter), two are required socket families that systemd scores as exposed regardless of intent (RestrictAddressFamilies), and one is cosmetic (UMask). None are oversights.\nThat closes the OTBR service. The pattern — read the directive, test it on hardware, account for every ✗ — is the same work for every other daemon on this gateway. When another service has an interesting story, it\u0026rsquo;ll make it here.\nReferences systemd.exec(5) — execution environment systemd-analyze security seccomp(2) — Linux man page seccomp BPF — kernel documentation EU Cyber Resilience Act — European Commission Part 1 — Running OTBR as Non-Root: Finding the Capability Floor ","permalink":"https://umair-as.github.io/blog/posts/otbr-systemd-hardening/","summary":"Part 1 got OTBR running as a non-root user with three capabilities. Part 2 covers the hardening block in the service file — what each directive does, why one had to be an exception, and how to use systemd-analyze security as a decision tool rather than a score to chase. Part 2 of 2.","title":"Hardening OTBR: Reading a 4.1 systemd-analyze Score"},{"content":"A while back I was hardening the services on my Raspberry Pi 5 IoT gateway — a Yocto-built image running a Thread border router, MQTT broker, and a handful of other daemons. One of the design requirements I\u0026rsquo;d set for myself early on: every long-running service runs as a dedicated non-root user. Least privilege, isolated runtime directories, no ambient root access sitting around waiting to be exploited. The kind of posture the EU\u0026rsquo;s Cyber Resilience Act will eventually require you to justify on paper, but worth doing regardless.\nMost daemons were straightforward — a USERADD_PARAM in the recipe, a User= line in the service file, done. OpenThread Border Router was not.\nOTBR ships expecting to run as root. The CMake build system makes this explicit:\nset(OTBR_AGENT_USER \u0026#34;root\u0026#34; CACHE STRING \u0026#34;set the username running otbr-agent service\u0026#34;) set(OTBR_AGENT_GROUP \u0026#34;root\u0026#34; CACHE STRING \u0026#34;set the group using otbr-agent client\u0026#34;) Override flags exist — -DOTBR_AGENT_USER and -DOTBR_AGENT_GROUP — but the documentation stops there. It doesn\u0026rsquo;t tell you what the non-root user actually needs. All test scripts use sudo. The Docker container runs as root without user switching. The escape hatch exists; it\u0026rsquo;s just unlabeled.\nThe first error Create the user, update the service, restart:\notbr-agent[1423]: Failed to create TUN device: Operation not permitted EPERM on TUNSETIFF. Expected. The question is: what else will fail once we fix this? The brute-force approach — add capabilities until it starts — leaves you with more than you need and no understanding of why. Better to map the requirements first.\nTracing capabilities from source Rather than iterating blind on the device, I used DeepWiki\u0026rsquo;s ot-br-posix index with Claude Code to trace through the OTBR source and identify every call site that requires elevated privileges — complete list upfront, no hardware iteration.\nWhat it actually needs Portability note: Some decisions here are specific to this image — a nftables-only kernel (CONFIG_IP6_NF_IPTABLES=n), a read-only rootfs, and otbr-web sharing the same system user as otbr-agent. The two core capability requirements (CAP_NET_ADMIN, CAP_NET_RAW) were required in this build and are expected for any standard OTBR configuration — but may vary with non-default build flags. The third (CAP_NET_BIND_SERVICE) and the firewall approach are deployment-specific.\nTwo capabilities are required unconditionally by otbr-agent. A third depends on your deployment. The source analysis makes this clear before touching the device.\nCAP_NET_ADMIN This covers most of what OTBR does with the network stack:\nOperation Source location Open /dev/net/tun + TUNSETIFF ioctl src/host/posix/netif_linux.cpp:96-99 Netlink route socket (AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) src/utils/socket_utils.cpp:65 SO_BINDTODEVICE socket option src/host/posix/netif.cpp:348 MRT6_INIT, MRT6_ADD_MIF (multicast routing) src/host/posix/multicast_routing_manager.cpp:202 Netfilter queue socket src/backbone_router/nd_proxy.cpp:418 The TUN device is the most fundamental — that\u0026rsquo;s how OTBR creates wpan0. Without CAP_NET_ADMIN, you never get past startup.\nCAP_NET_RAW Two raw socket operations, both in the infrastructure interface code:\nsrc/host/posix/netif.cpp:334 AF_INET6, SOCK_RAW, IPPROTO_ICMPV6 src/host/posix/infra_if.cpp:278 AF_INET6, SOCK_RAW, IPPROTO_ICMPV6 These are for ICMPv6 — neighbor discovery, router advertisements — the machinery that makes Thread devices appear as proper IPv6 nodes on the infrastructure network.\nCAP_NET_BIND_SERVICE (deployment-specific) otbr-agent\u0026rsquo;s REST API runs on port 8081 — above 1024, so the agent itself doesn\u0026rsquo;t need this. The web UI (otbr-webui) binds port 80, and that does. On this image both services run under the same otbr user, so CAP_NET_BIND_SERVICE is carried in the bounding set to cover both. If you run otbr-webui as a separate user or redirect port 80 via nftables, you can drop it from the agent entirely. It is not a core requirement of otbr-agent.\nThe external command problem Capabilities cover the direct syscalls. But OTBR also execs external tools at runtime via system():\nip -6 route commands — dua_routing_manager.cpp ip6tables — nd_proxy.cpp ipset — firewall set management There are two problems here, and they\u0026rsquo;re independent.\nThe first is image-specific: this kernel has no legacy iptables stack (CONFIG_IP6_NF_IPTABLES=n). The ip6tables calls in nd_proxy.cpp would fail regardless of privilege — the binary simply isn\u0026rsquo;t there. If you\u0026rsquo;re on a distribution with the full xtables stack, you won\u0026rsquo;t hit this, but you\u0026rsquo;ll still hit the second problem.\nThe second is architectural: firewall state should be initialized once before the daemon starts, not managed at runtime by a long-running process. Child processes spawned via system() do inherit ambient capabilities from the parent — with CAP_NET_ADMIN in the ambient set, ip route operations would work without root. But ip6tables and ipset rule setup belongs in initialization, not scattered through a daemon\u0026rsquo;s runtime path. It\u0026rsquo;s fragile, hard to audit, and makes the service harder to reason about.\nBoth issues point to the same solution: take firewall management out of OTBR\u0026rsquo;s hands entirely. Front-load the rules in an init script that runs once as root before the daemon starts.\nsystemd\u0026rsquo;s ExecStartPre=+ is the mechanism. The + prefix is a per-command root override — it runs that specific command as root regardless of the service\u0026rsquo;s User= setting. The main ExecStart (without +) still runs as otbr. So the privilege split is clean: root work happens once during initialization, the long-running process never has root.\nThe socket directory problem OTBR creates its Unix domain sockets in /run by default:\n/run/openthread-wpan0.sock /run/openthread-wpan0.lock Two things make this impossible on this gateway. First, the rootfs is read-only (ext4, ro) — but that\u0026rsquo;s actually fine here because /run is a separate tmpfs mount (rw). The real problem is permissions: /run itself is drwxr-xr-x (root-owned, no write for others), and the otbr user gets a hard Permission denied trying to create anything there:\n$ touch /run/test-otbr touch: cannot touch \u0026#39;/run/test-otbr\u0026#39;: Permission denied Every other daemon on the system has its own subdirectory — /run/avahi-daemon/, /run/mosquitto/, etc. — owned by its respective user. OTBR needs the same treatment.\nThe problem is the socket path is compiled in, hardcoded in two places:\n// third_party/openthread/repo/src/posix/platform/openthread-posix-daemon-config.h #define OPENTHREAD_POSIX_CONFIG_DAEMON_SOCKET_BASENAME \u0026#34;/run/openthread-%s\u0026#34; This needed a patch — same change in both the web service client and the platform config header, because if they\u0026rsquo;re out of sync, the web UI can\u0026rsquo;t connect to the daemon socket:\n-#define OPENTHREAD_POSIX_CONFIG_DAEMON_SOCKET_BASENAME \u0026#34;/run/openthread-%s\u0026#34; +#define OPENTHREAD_POSIX_CONFIG_DAEMON_SOCKET_BASENAME \u0026#34;/run/otbr/openthread-%s\u0026#34; Then tmpfiles.d creates the directory at boot:\n# /usr/lib/tmpfiles.d/otbr.conf d /run/otbr 0750 otbr otbr - The configuration System user and device groupadd --system otbr useradd --system --home-dir /var/lib/otbr \\ --shell /sbin/nologin \\ --gid otbr --groups dialout \\ otbr In a Yocto recipe, this is handled by USERADD_PARAM. The CMake flags wire into the build via EXTRA_OECMAKE += \u0026quot;-DOTBR_AGENT_USER=otbr -DOTBR_AGENT_GROUP=otbr\u0026quot; — this makes the installed systemd unit reference the correct user without patching it manually.\nThe dialout group gives access to the RCP serial device. A udev rule assigns the symlink:\nSUBSYSTEM==\u0026#34;tty\u0026#34;, KERNEL==\u0026#34;ttyACM*\u0026#34;, GROUP=\u0026#34;dialout\u0026#34;, MODE=\u0026#34;0660\u0026#34;, SYMLINK+=\u0026#34;otbr-rcp\u0026#34; Service unit [Unit] Description=OpenThread Border Router Agent Requires=dbus.socket avahi-daemon.service dev-otbr\\x2drcp.device After=dbus.socket avahi-daemon.service dev-otbr\\x2drcp.device [Service] User=otbr Group=otbr SupplementaryGroups=dialout EnvironmentFile=-/etc/default/otbr-agent Environment=OTBR_SOCKET_DIR=/run/otbr RuntimeDirectory=otbr RuntimeDirectoryMode=0750 # These run as root (note the + prefix) ExecStartPre=+/usr/libexec/otbr/otbr-ipset-init ExecStartPre=+/bin/sh -c \u0026#39;rm -f /run/otbr/openthread-wpan0.sock; \\ touch /run/otbr/openthread-wpan0.lock; \\ chown otbr:otbr /run/otbr/openthread-wpan0.lock; \\ chmod 660 /run/otbr/openthread-wpan0.lock\u0026#39; ExecStart=/usr/sbin/otbr-agent $OTBR_AGENT_OPTS KillMode=mixed Restart=on-failure RestartSec=5 # Core: CAP_NET_ADMIN CAP_NET_RAW — required by otbr-agent # Deployment-specific: CAP_NET_BIND_SERVICE — needed here because otbr-web shares this user AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE NoNewPrivileges=true [Install] WantedBy=multi-user.target The service file also carries a hardening block — ProtectSystem, PrivateDevices, RestrictAddressFamilies, and a dozen others. Some of them required deliberate exceptions for OTBR specifically, and each directive is worth explaining properly. That\u0026rsquo;s Part 2.\nThe firewall init script otbr-ipset-init replaces what upstream OTBR would do via ip6tables at runtime. It runs once as root via ExecStartPre=+, sets up the ipset tables OTBR expects to exist, and installs the equivalent nftables rules. The NAT block has an explicit fallback — not all kernel configs include nf_nat, and the service shouldn\u0026rsquo;t fail hard on a device where NAT isn\u0026rsquo;t needed:\n#!/bin/sh set -e THREAD_IF=\u0026#34;${THREAD_IF:-wpan0}\u0026#34; INFRA_IF=\u0026#34;${INFRA_IF:-wlan0}\u0026#34; ipset create -exist otbr-ingress-deny-src hash:net family inet6 ipset create -exist otbr-ingress-deny-src-swap hash:net family inet6 ipset create -exist otbr-ingress-allow-dst hash:net family inet6 ipset create -exist otbr-ingress-allow-dst-swap hash:net family inet6 nft delete table inet otbr 2\u0026gt;/dev/null || true nft -f - \u0026lt;\u0026lt;EOF table inet otbr { chain mangle_prerouting { type filter hook prerouting priority mangle; policy accept; iifname \u0026#34;$THREAD_IF\u0026#34; meta mark set 0x1001 } chain filter_forward { type filter hook forward priority filter; policy accept; oifname \u0026#34;$INFRA_IF\u0026#34; accept iifname \u0026#34;$INFRA_IF\u0026#34; accept } } EOF nft -f - \u0026lt;\u0026lt;EOF || echo \u0026#34;WARNING: nftables NAT unavailable, running without masquerade\u0026#34; \u0026gt;\u0026amp;2 table ip otbr_nat { chain nat_postrouting { type nat hook postrouting priority srcnat; policy accept; meta mark 0x1001 oifname \u0026#34;$INFRA_IF\u0026#34; masquerade } } EOF D-Bus policy OTBR claims a D-Bus name (io.openthread.BorderRouter.*). The default policy only allows root to own service names. This one is easy to miss — D-Bus registration happens late in startup, after the TUN device, sockets, and capabilities are all working. The agent exits silently.\n\u0026lt;policy user=\u0026#34;otbr\u0026#34;\u0026gt; \u0026lt;allow own_prefix=\u0026#34;io.openthread.BorderRouter\u0026#34;/\u0026gt; \u0026lt;allow send_interface=\u0026#34;io.openthread.BorderRouter\u0026#34;/\u0026gt; \u0026lt;allow send_interface=\u0026#34;org.freedesktop.DBus.Properties\u0026#34;/\u0026gt; \u0026lt;allow send_interface=\u0026#34;org.freedesktop.DBus.Introspectable\u0026#34;/\u0026gt; \u0026lt;/policy\u0026gt; Verifying on the device strace confirmation Before locking down the unit file, confirm with strace that the three capabilities are sufficient. Stop the service, run the ipset init manually (it normally runs via ExecStartPre=+), then launch under capsh:\nroot@iot-gateway:~# systemctl stop otbr-agent root@iot-gateway:~# /usr/libexec/otbr/otbr-ipset-init root@iot-gateway:~# capsh --user=otbr \\ --inh=\u0026#34;cap_net_admin,cap_net_raw,cap_net_bind_service\u0026#34; \\ --addamb=\u0026#34;cap_net_admin,cap_net_raw,cap_net_bind_service\u0026#34; \\ -- -c \u0026#34;strace -f -e trace=socket,bind,ioctl \\ /usr/sbin/otbr-agent -I wpan0 -B wlan0 \\ --rest-listen-address 0.0.0.0 \\ spinel+hdlc+uart:///dev/otbr-rcp?uart-baudrate=460800 trel://wlan0\u0026#34; 2\u0026gt;\u0026amp;1 | \\ grep -E \u0026#34;socket|bind|TUNSETIFF|EPERM|error\u0026#34; The -f follows forked threads — important since otbr-agent spawns several that each do their own socket setup. Output from the device:\nsocket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC, NETLINK_ROUTE) = 14 bind(14, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000101}, 12) = 0 socket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, NETLINK_ROUTE) = 16 bind(16, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000101}, 12) = 0 ioctl(17, TUNSETIFF, 0xffffca8e61e0) = 0 socket(AF_INET6, SOCK_RAW|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_ICMPV6) = 18 socket(AF_INET6, SOCK_RAW|SOCK_CLOEXEC, IPPROTO_ICMPV6) = 21 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 22 bind(22, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0 No EPERM anywhere. Every call maps back to the source analysis: TUNSETIFF and netlink sockets → CAP_NET_ADMIN; both IPPROTO_ICMPV6 raw sockets → CAP_NET_RAW. CAP_NET_BIND_SERVICE doesn\u0026rsquo;t appear because the REST API binds above port 1024 — it only matters for otbr-web on port 80.\nCapability sets Check what the running process actually holds:\ncat /proc/$(pgrep -x otbr-agent)/status | grep Cap capsh --decode=$(cat /proc/$(pgrep -x otbr-agent)/status | grep CapEff | awk \u0026#39;{print $2}\u0026#39;) Three capabilities: cap_net_bind_service, cap_net_admin, cap_net_raw. Nothing else.\nKnown rough edges A few warnings appear in the logs that don\u0026rsquo;t affect operation:\n[W] P-Netif-------: ADD [U] fe80::107f:217:fc6f:63bf failed (InvalidArgs) [W] P-Netif-------: Failed to process event, error:InvalidArgs OTBR attempting to add a link-local address that already exists on the interface. The interface still comes up correctly.\nBefore and after Before: otbr-agent as root, full capabilities, no sandbox.\nAfter: a system user with no shell, three network capabilities, ExecStartPre=+ handling the root-only initialization work, and a runtime directory the process can actually write to.\nThe socket patch took longer to track down than it should have — the web service client and the platform config header need matching paths, and a mismatch produces a silent connection failure, not a crash. Everything after that — the hardening block, the directives that had to stay off, the systemd-analyze security score — is a different kind of work: not \u0026ldquo;will it run?\u0026rdquo; but \u0026ldquo;how much surface are you actually exposing?\u0026rdquo;\nThat\u0026rsquo;s the next article.\nFull source for the Yocto recipe, service files, and init script is in meta-iot-gateway.\nReferences OpenThread Border Router — official docs ot-br-posix source index — DeepWiki ot-br-posix on GitHub capabilities(7) — Linux man page CAP_NET_ADMIN, CAP_NET_RAW, CAP_NET_BIND_SERVICE — kernel credentials docs systemd.exec(5) — execution environment systemd-analyze security tmpfiles.d(5) seccomp(2) — Linux man page seccomp BPF — kernel documentation nftables wiki Migrating from iptables to nftables ipset man page EU Cyber Resilience Act — European Commission EU Cyber Resilience Act — community resource ","permalink":"https://umair-as.github.io/blog/posts/running-otbr-as-non-root/","summary":"OpenThread Border Router assumes it runs as root. Getting it down to a least-privilege non-root user means figuring out exactly what it needs — which turned out to be a source code problem, not a trial-and-error problem. Part 1 of 2.","title":"Running OTBR as Non-Root: Finding the Capability Floor"}]