Packaging Telegraf for Yocto: Build Tags, a Linker Collision, and Credentials That Can't Live in the Rootfs

Telegraf is a good fit for edge observability: one static binary, a plugin system that covers system metrics, MQTT, Modbus RTU, and InfluxDB output in a single config file. Adding it to the rpi5-iot-gateway — a Yocto-built RPi5 system running a Thread border router, MQTT broker, and RAUC OTA stack — looked like a straightforward packaging task.

No Telegraf recipe exists in meta-openembedded. The OE layer index has entries in four layers — meta-luneos, meta-webosose, meta-ossystems-base, and meta-influx — but none go above 1.21.4, and Telegraf 1.31 changed plugin registration in a way that makes those recipe structures wrong, not just stale. That meant writing from scratch, which surfaced two non-obvious build issues.

The harder problem was credentials. The image rootfs is a squashfs image replaced wholesale on every RAUC OTA update — any credential baked into the rootfs is either shared across every device running that build, or silently wiped on the next upgrade. The standard fallback, EnvironmentFile=, exposes secrets in /proc/$PID/environ and core dumps — a trade-off that’s acceptable on a server you can audit, not on a fleet of devices you can’t. This post covers the recipe, the two build failures, and the credential model that the image layout actually requires.

Four layers, none past 1.21.4

Telegraf is a Go module, so the recipe inherits go-mod. The binary entry point is cmd/telegraf. A dedicated system user handles isolation:

GO_IMPORT = "github.com/influxdata/telegraf"
PV = "1.31.0"
SRCREV = "fbfaba054e62413b6a0a90372281e687d9ff1238"

SRC_URI = " \
    git://github.com/influxdata/telegraf.git;protocol=https;nobranch=1;destsuffix=${BPN}-${PV}/src/${GO_IMPORT} \
    file://telegraf.service \
    file://telegraf.conf \
    file://telegraf.tmpfiles \
"

inherit go-mod systemd useradd

GO_INSTALL = "${GO_IMPORT}/cmd/telegraf"

GROUPADD_PARAM:${PN} = "-f --system telegraf; -f --system dialout"
USERADD_PARAM:${PN} = "--system -d /var/lib/telegraf -m -s /sbin/nologin \
                       --gid telegraf --groups dialout telegraf"

That gets the build wired. First bitbake telegraf failed immediately.

300 plugins, two conflicting symbols

runtime/cgo(.text): symbol __cgoenqueue defined in both
  vendor/github.com/Azure/azure-kusto-go/...
  vendor/github.com/snowflakedb/gosnowflake/...
duplicate symbol error

Two vendored packages — azure-kusto-go and gosnowflake — both expose a C symbol __cgoenqueue generated by their shared cel-go/antlr4 dependency. The linker sees two definitions and stops.

The root cause is how Telegraf 1.31 changed plugin registration. Before 1.31, each plugin category had a single all.go that imported every plugin unconditionally. From 1.31 onward, each plugin has its own file gated by a build constraint:

//go:build !custom || inputs.cpu

package all

import _ "github.com/influxdata/telegraf/plugins/inputs/cpu"

Without the custom tag, !custom is true and all 300+ plugins compile. Two of them drag in conflicting vendored symbols via the same generated C code.

Telegraf ships a custom_builder tool¹ that reads a config file and prints exactly which build tags are needed. Running --dry-run against the gateway telegraf.conf gave the tag list:

TELEGRAF_PLUGIN_TAGS = "custom,inputs.cpu,inputs.disk,inputs.internal,\
inputs.mem,inputs.modbus,inputs.mqtt_consumer,inputs.net,inputs.processes,\
inputs.system,inputs.temp,outputs.influxdb,parsers.json,secretstores.systemd"

GOBUILDFLAGS:append = " -tags ${TELEGRAF_PLUGIN_TAGS}"

Adding a plugin later means appending its tag here and adding the corresponding stanza to telegraf.conf. custom_builder --dry-run is the source of truth — don’t guess.

The same linker, a different package pair

Build tags cleared the first error. The second came from go.bbclass defaults.

go.bbclass appends -buildmode=pie by default.² Telegraf’s upstream Makefile builds with CGO_ENABLED=0 and no PIE — all selected plugins in this profile are pure Go. Enabling PIE against CGO-disabled code caused the Go generic linker to produce another duplicate symbol error, from a different package pair in the antlr4/cel-go dependency tree. Same pattern: two packages generating the same C symbol from shared generated code. The symptom kept surfacing from different package combinations until the flags matched upstream.

Matching upstream’s release build:

CGO_ENABLED = "0"
GOBUILDFLAGS:remove = "-buildmode=pie"
# -linkshared requires CGO — clear it for a pure-Go build
GO_LINKSHARED = ""

One more go.bbclass default to override: do_compile[network] = "0". Go module resolution for this recipe happens at compile time rather than fully during do_fetch, so network access is needed:

do_compile[network] = "1"

After these changes the binary builds cleanly.

Neither the rootfs nor the environment

The rootfs constraint is straightforward: squashfs is replaced wholesale on every OTA update, so anything baked in either gets wiped on upgrade or is identical across every device running that image. Neither is acceptable for service credentials.

Environment variables are ruled out too, and less obviously so. If you reach for EnvironmentFile=-/etc/default/telegraf with credentials stored as KEY=value pairs, they end up readable in /proc/$PID/environ:

MQTT_PASSWORD=...
INFLUXDB_PASSWORD=...

Any process running as root — audit daemons, monitoring agents, anything with ptrace access — can read them. They also appear in core dumps. An Environment= drop-in in a systemd unit has the same exposure: the value serialises into the unit state under /run/systemd/ and is visible via systemctl show.

The solution is LoadCredential= (systemd 250+)³, which creates a per-unit tmpfs at service activation — isolated from the environment, not visible in /proc, not in argv. Combined with Telegraf 1.28’s secretstores.systemd plugin⁴, credentials never appear in the config file either. A provisioning script fills the credstore at first boot; the values survive OTA because /etc is an overlay fs with its upper layer on the persistent data partition (mmcblk0p5, ext4). OTA replaces the squashfs lower layer but leaves the data partition intact.

The full mechanics of LoadCredential= — how the tmpfs is structured, the permission model, how to provision it at manufacturing time, and how it composes with SetCredentialEncrypted= — are worth a dedicated post. That’s coming. For now, the relevant config:

# telegraf.service (excerpt)
LoadCredential=mqtt_username:/etc/credstore/telegraf.service.mqtt_username
LoadCredential=mqtt_password:/etc/credstore/telegraf.service.mqtt_password
LoadCredential=influxdb_username:/etc/credstore/telegraf.service.influxdb_username
LoadCredential=influxdb_password:/etc/credstore/telegraf.service.influxdb_password

# telegraf.conf (excerpt)
[[secretstores.systemd]]
  id = "systemd"
  prefix = ""

[[outputs.influxdb]]
  urls = ["${INFLUXDB_URL}"]
  database = "${INFLUXDB_DATABASE}"
  username = "@{systemd:influxdb_username}"
  password = "@{systemd:influxdb_password}"

[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  username = "@{systemd:mqtt_username}"
  password = "@{systemd:mqtt_password}"
  topics = ["sensors/+/data"]
  data_format = "json"

INFLUXDB_URL and INFLUXDB_DATABASE are non-secret defaults from EnvironmentFile=-/etc/default/iotgw-observability. The @{systemd:key} references resolve at runtime from $CREDENTIALS_DIRECTORY.

The guard that ran as the wrong user

The credstore files ship as zero-byte placeholders, mode 0600, owned root:root. Before provisioning runs on a fresh device, Telegraf must not start — bad credentials cause a restart loop and journal noise.

The first cut used ExecCondition= with test -s checks in [Service]. It looked correct. The service skipped cleanly on a fresh device — exactly the behaviour we wanted. It was only when testing the fix on device that test -s kept returning failure even after the credstore files were populated. The reason: ExecCondition= runs under the service’s User= — in this case telegraf. The credstore directory is 0700 root:root. The telegraf user has no execute permission on it, so test -s returns EACCES regardless of file content. The guard worked initially only because the files happened to be empty — both empty files and inaccessible paths look the same to test -s.

ConditionFileNotEmpty=⁵ is the right directive — it runs in PID 1’s context (root), is designed exactly for this check, and belongs in [Unit] not [Service]. The fix:

--- a/telegraf.service
+++ b/telegraf.service
@@ -3,6 +3,10 @@ Description=Telegraf metrics agent
 After=network.target mosquitto.service
 Wants=mosquitto.service
+ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_username
+ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_password
+ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_username
+ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_password
 StartLimitIntervalSec=5min
@@  LoadCredential=influxdb_password:/etc/credstore/telegraf.service.influxdb_password
-# Skip service start until required credentials are non-empty.
-ExecCondition=/bin/sh -c 'test -s /etc/credstore/telegraf.service.mqtt_username'
-ExecCondition=/bin/sh -c 'test -s /etc/credstore/telegraf.service.mqtt_password'
-ExecCondition=/bin/sh -c 'test -s /etc/credstore/telegraf.service.influxdb_username'
-ExecCondition=/bin/sh -c 'test -s /etc/credstore/telegraf.service.influxdb_password'
 ExecStart=/usr/bin/telegraf --config /etc/telegraf/telegraf.conf

[Unit]
ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_username
ConditionFileNotEmpty=/etc/credstore/telegraf.service.mqtt_password
ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_username
ConditionFileNotEmpty=/etc/credstore/telegraf.service.influxdb_password

When any file is absent or empty, systemd marks the unit condition-failed and skips it cleanly — no Restart= cycle, no journal errors:

telegraf.service: Skipped due to unmet condition checks.
Condition check resulted in Telegraf metrics agent being skipped.

Once the provisioning unit writes non-empty values and exits, the next start attempt clears all conditions.

On device

With the fixed service file deployed, journal on first start with test credentials in the credstore:

systemd[1]: Started Telegraf metrics agent.
telegraf[897]: Loading config: /etc/telegraf/telegraf.conf
telegraf[897]: Available plugins: 11 inputs, 0 aggregators, 0 processors, 2 parsers, 1 outputs, 1 secret-stores
telegraf[897]: Loaded inputs: cpu disk internal mem mqtt_consumer net processes system temp
telegraf[897]: Loaded secretstores: systemd
telegraf[897]: Loaded outputs: influxdb
telegraf[897]: Tags enabled: host=iot-gateway role=iot-gateway site=plant-a
telegraf[897]: [outputs.influxdb] When writing to [http://127.0.0.1:8086]: dial tcp 127.0.0.1:8086: connect: connection refused
telegraf[897]: [telegraf] Error running agent: starting input inputs.mqtt_consumer: not Authorized

Loaded secretstores: systemd confirms credentials were read from $CREDENTIALS_DIRECTORY. The MQTT not Authorized error means Telegraf reached the broker and attempted authentication with the credential — the secret store flow worked. The failures are expected: these are test credentials and the local InfluxDB instance isn’t running.

systemctl show confirms no credentials in the service environment:

Environment=
EnvironmentFiles=/etc/default/iotgw-observability (ignore_errors=yes)

The environment file only carries non-secret defaults:

INFLUXDB_URL=http://127.0.0.1:8086
INFLUXDB_DATABASE=gateway_data

To verify nothing leaks into the process environment or /proc, running the same LoadCredential= setup as the telegraf service under User=telegraf:

$ ls -la $CREDENTIALS_DIRECTORY
dr-xr-x---+ 2 root root 80 Apr 23 05:35 .
drwxr-xr-x  3 root root 60 Apr 23 05:35 ..
-r--r-----+ 1 root root 14 Apr 23 05:35 mqtt_password
-r--r-----+ 1 root root 14 Apr 23 05:35 mqtt_username

$ env | grep -iE "pass|secret|mqtt|influx"
(none)

$ cat /proc/self/environ | tr '\0' '\n' | grep -iE "pass|secret|mqtt|influx"
(none)

Credentials exist only as files in the per-unit tmpfs — mode 0500 on the directory, 0400 on each file, owned by the service user. They are not in the environment, not in argv, and not in /proc/$PID/environ. The tmpfs is created at service activation and destroyed when the service exits.

LoadCredential= requires systemd 250. The RPi5 BSP on Scarthgap ships systemd 255, so that’s not a constraint here. On BSPs below 250, SetCredential= (available from 247) bakes a value into the unit file at build time — no help for device-specific secrets. Below that there’s no clean answer from systemd alone.

Full source for the recipe, service file, and tmpfiles config is in meta-iot-gateway.