0014: Provision Log Sinks Instead of Relaying Logs

Status

Accepted

Settles how supervised workloads reach the host logger. Owned in detail by C0 Observability § The log stream and the spawn contract in C0 Agent § Supervision.

Amendment note (2026-08). Rewritten in place as the Installer design settled where the log destination is configured. Three changes to the decision as first accepted: the file class is gone — private now names whatever fd 2 already is for an unprovisioned process, and the rotation argument behind its removal is recorded below; macOS is no longer a second class, running syslog against a real daemon like every other platform, which makes syslog the single production sink everywhere; and the resulting host dependency is stated in full, including what it asks of the socket's owner and what is accepted where that cannot be met. The FLOR_* spawn variables became FLORWL_* in the same pass. This text is the integrated current decision.

Context

flor agent supervises the platform workloads on a node (Agent), and every one of them — plus the agent itself — must reach the host's logger with its lines attributable to a workload name. Only the agent is registered as a service; its children are ordinary forked processes with no OS identity of their own.

The original C0 design made the agent a relay: it captured each child's stderr over a pipe, re-emitted every line tagged with the workload name onto its own stderr, and let the service wrapper carry that to journald or a file. Five problems accumulated against it:

• Log delivery became a function of agent liveness. Rust sets SIGPIPE to SIG_IGN, so children survive an agent crash but write into a broken pipe and go silently blind — at precisely the moment their logs are wanted.
• The supervisor acquired a line parser. Partial reads, oversized lines, a panic backtrace with no trailing newline, and re-parsing the sd-daemon <N> priority prefix in order to preserve it — all in the hot path of every log line, undoing work the child had just done.
• Flooding was bounded by stalling the supervisor. Pipe backpressure does bound a noisy child, by blocking the process that owns sync and restart.
• Per-workload filtering degraded to grep. With every child sharing the agent's service identity, the host logger sees one stream; a name glued to the front of a line is not a filter.
• The message boundary was lost. A pipe carries bytes, so a relay re-frames by newline — and a multi-line error stack is several messages by the time it reaches the sink. This turned out to be the decisive constraint, and it decides the transport below as much as it decides the relay.

Decision

The agent provisions log sinks; it never relays log bytes. Before fork it opens a sink for the child and installs it as fd 2. Three classes, and the split that matters is not per-OS but who arranged the sink:

syslog is the only class normal operation uses, and it is the same class on every supported platform. The agent provisions no other: private is what a lone process gets when nobody provisioned anything for it, and shared is what an agent run with --log-stderr hands down for attached debugging. So the class list is not a per-OS matrix — there is one production sink, and two shapes that only appear when a human is watching.

Two rules follow, and they are the load-bearing part:

The agent chooses the sink; a child cannot redirect its own output. The child holds a descriptor it did not open, so attribution by sink is enforced. Attribution within a sink is content the child writes — the syslog identifier and the rendered workload name alike — and is therefore cooperative; on private there is nothing to attribute, because one process holds the sink alone. flor's own binaries are trusted for that content, as they are for sane levels and for not flooding. A third-party workload will be trusted for none of it, and needs no change to this contract to be confined.

The contract is "fd 2 is your log sink"; what is behind it is the agent's choice. Today that is the OS. For an untrusted tenant it can become a filtering logger process per workload — rate-limiting, sanitizing, and rewriting the identifier to what the agent says it is, which is what restores enforced attribution exactly where it will matter — with nothing on the child's side changing.

Because neither the sink's framing nor "is this fd shared with my siblings" is observable from the fd, the agent states the class in FLORWL_LOG_SINK, and every process announces it at startup. A process started by hand has nobody to tell it and defaults to private; it probes for nothing, since a probe answers a different question than the one being asked. All three values may be set by hand, private included, so the default is sayable rather than only reachable by absence.

Rationale

Four options were weighed against the single question of who must be in the data path.

Relay (the previous design) buys one OS-integration point and pays with the four problems above. Docker is the one mature supervisor that does relay — because it promises pluggable log drivers, so something must sit in the stream and dispatch. flor has one production destination, the same on every platform, and does not earn that cost.

One service unit per workload was already rejected for supervision reasons (Agent § Supervision mechanics) — multiplied platform backends, split rollback state. It solves logging as a side effect of a change made for other reasons, and those reasons still say no.

Each workload calling a native logging API (sd_journal_send, os_log) gives the best queryability and costs a second output format — structured fields rather than a rendered line — plus FFI on macOS. It also opens its own connection rather than riding fd 2, so it cannot catch a workload's unframed output and would need a second sink beside it. Deferred, not foreclosed.

Provisioning (chosen) takes the supervisor out of the data path, and it is what the two closest analogues already do. systemd hands each unit its own stream fd and lets journald attribute; kubelet is not in the path at all, and container logs are attributed by their path under /var/log/pods/…. The shared class is what daemontools-family supervisors and docker compose up produce when run in a terminal, and it is the reason the class is named for sharing rather than for a console: a foreground agent with its output redirected to a file shares that file with its children just as a tty is shared.

Why there is no file sink

An earlier form of this decision gave each workload a file — <workload>.log, opened by the agent — and made that the private class. The class is gone rather than merely unused, and the reason deserves recording, because a directory of files looks like the obvious fallback for any platform without a syslog socket, and will look that way again on Windows.

A file the agent owns cannot be rotated. The child inherits fd 2 at fork and holds it for its whole life. Rotation renames or truncates the path, and every child goes on writing into an inode that no longer has a name — the agent cannot fix this, because it cannot reach a descriptor it has already handed away. Reopening is the child's act, so a file sink works only with cooperating workloads: a SIGHUP-and-reopen convention, or a reopen command over a per-workload control channel. Both put a new obligation into the supervised-program contract — the surface a tenant workload must implement from B2+ — and the value of that contract is that it is small and demands nothing a workload can silently get wrong. This obligation is exactly the kind that fails quietly: a workload that ignores the reopen keeps logging into a deleted inode until the disk fills, and nothing reports it.

The socket has none of this. Retention, rotation, and the size cap belong entirely to the daemon behind it, and flor owns no log file anywhere — which is also why the installer ships no log directory and no rotation config. That is the load-bearing reason for requiring a socket rather than a directory, and it is what reduced the production class list to one.

It also sharpens the kubelet comparison above rather than resting on it: a per-container file works there because the container runtime holds the write end and can reopen it — someone is in the data path, which is precisely what this ADR removed. Having taken the supervisor out of the path, flor cannot own a file; the two choices are consistent with each other and cannot be mixed.

private keeps its name for something narrower: whatever fd 2 already is for a process nobody provisioned for. That may well be a file — a developer's 2> vertex.log — but it was opened by their shell, is rotated by nobody, and lasts as long as the debugging session. A file in the picture is then the operator's deliberate act, never flor's.

Why syslog rather than journald's stdout stream

The obvious Linux sink is the journald stdout-stream fd, which pre-attributes at connect time and would have made attribution enforced rather than cooperative. It was rejected because it terminates a record at every \n: a multi-line error stack becomes one journal entry per line, each needing its own priority prefix, and the message boundary flor's writer maintains is destroyed on arrival. Measured on a live journald: three entries for a three-line message on the stream, one entry with the newlines intact for the same message sent as a syslog datagram.

That forces a choice, and it generalizes past this decision: among journald's transports, only the line-splitting stream can pre-attribute a descriptor — every boundary-preserving transport, the native field protocol included, carries attribution as sender-chosen content. Message integrity and enforced attribution cannot both be had. Integrity wins: it is what makes an error stack readable and what keeps peer-injected text from fabricating a record, whereas attribution is cooperative only among flor's own trusted binaries and is restored by the filter process on the day it is not.

Naming the class for the transport rather than for journald is deliberate: a syslog socket is answered by rsyslog, syslog-ng or busybox on hosts with no systemd, which is ordinary on embedded Linux, and the same descriptor and framing work against all of them. What varies behind it is not the record boundary — that is the datagram — but multi-line rendering and the size cap, which are the daemon's.

The daemon then supplies handling that would otherwise be flor's: it parses the <N> frame into a priority and strips it, files an unframed write at its default priority, and rate-limits, dropping rather than blocking.

Why macOS takes the same class

macOS is not a second class. It runs the syslog class against a real syslog daemon — rsyslog or syslog-ng, routine via Homebrew — with the socket path named in config.toml (Installer § Logging). What is deliberately excluded is /var/run/syslog: Apple's socket is a compatibility shim into unified logging whose info and debug records are memory-only unless enabled per-subsystem, so flor's default INFO record would appear to be delivered and silently fail to persist, and there is no per-workload filter to replace journalctl -t — every vertex is the same flor process. Selecting it would produce a broken arrangement that looks like it works, which is the one failure mode this design spends everywhere else to avoid.

That leaves three ways to log on macOS, and the choice reads differently once there is no file class to retreat to. Apple's shim drops records silently. Calling os_log natively means FFI, its default redaction of %s arguments, and a second output format for one OS — worth doing eventually, not for the first milestone. One documented host dependency wins, because it is the only option that keeps one class, one line format, and one code path on every platform, and because it fails loudly: with nothing answering the socket, the agent exits at startup instead of discarding its own diagnostics — 69 (EX_UNAVAILABLE) rather than 78, since a daemon that is not up yet is a condition the next restart may find resolved.

The general form is worth stating, since Windows will ask the same question in C1: this ADR's answer to a platform with no syslog socket is a native structured sink (os_log, journald's field protocol, the Event Log) — never a directory of agent-owned files, which would reintroduce the rotation problem the socket exists to avoid.

Consequences

Easier. Per-workload filtering with follow is native and identical on every platform (journalctl -t <workload> -f, or the equivalent against whatever daemon answers). A workload keeps logging across an agent crash. No process sits in the path copying log bytes, so a flooding workload cannot stall supervision. Retention and rotation are wholly the daemon's, so flor ships no log directory, no rotation config, and no reopen convention. Confinement for future tenant workloads has a place to go that does not disturb the contract.

Harder — a new host dependency, and it holds for the whole run. flor now requires something listening on a syslog socket, and refuses to start without it (69, retried; a destination of the wrong socket type is 78 and is not). On Linux that is free; on macOS it is a documented setup step (user guide); on a platform with no syslog socket at all it is native-sink work, never a file fallback. This is the price of deleting the file class, and it is paid at startup with a clear error rather than at 3am with a missing record.

The dependency does not end at startup, and this is the honest limit of the decision: a child cannot be re-provisioned. The property that makes attribution enforceable — the child holds a descriptor it did not open — is exactly what stops anyone fixing that descriptor later, so a daemon which unlinks and recreates its socket on restart leaves the agent and every workload connected to a dead peer. It is the rotation problem in its other clothes: deleting the file class removed the frequent, automatic form of it and left the rare, deliberate one.

What closes it is a property of the socket, not of flor: the system supervisor should own it. A listening socket held by the service manager and passed to the daemon is created once and outlives every restart behind it — the inode is stable, so no descriptor goes stale, and datagrams sent while the daemon is down queue in the socket buffer rather than failing. systemd's syslog.socket is precisely this, which is why the default Linux path never exhibited the problem; launchd offers the same shape through a plist's Sockets key and launch_activate_socket(), for a daemon that asks for it. So the requirement C0 states is a system-owned socket where the platform provides one, and accepted log loss across daemon restarts where it does not — a host arrangement flor can ask for and check, not a mechanism it has to build. The residual case is a daemon binding its own path; recovery there is restarting the scope, and the cost is records, not service.

What C1 may add, neither committed: the agent noticing its own failing writes, stopping its workloads and exiting 69 so the wrapper brings the scope back with fresh sinks, or a service-manager dependency where one exists (systemd's PartOf=). The general cure, for the day untrusted tenants make accepted loss unacceptable, is the filter process this ADR already reserves — with a flor-owned logger between child and daemon, reopening becomes that process's job and needs nothing from the workload, which is what makes it the right answer there rather than an extra hop.

Harder, elsewhere. Two provisioning paths, and the sink class must be injected because it cannot be inferred. The private and shared classes have no daemon behind them and therefore no rate limit — bounded through B2 only because every supervised workload is flor's own and carries the flood-safe Layer, and tolerable because both classes are attended debugging. Where the daemon is not journald, "which file did my logs land in" is answered by reading that daemon's config, which is the concrete argument for a flor logs reader.

Weakened, in the threat model. Attribution is no longer enforced against a lying child; only the choice of sink is. The invariant this ADR can carry is the narrower one stated above, and the strong form arrives with the filter process, alongside the milestone that first needs it. Nothing built on the weaker form has to change when it does.

Changed, in the line itself. The message body is one format on every class — timestamp, level, component, spans, message, fields — but two things vary with the sink, and both are consequences of this decision rather than of taste. The syslog class gains a <N><workload>: frame per message, which the daemon consumes; its priority half is machine-facing and not a replacement for the level word, since journalctl's default output shows no priority beyond coloring and a levelless message would read worse. The shared class gains the workload name in the body, because it is the one class whose sink supplies no attribution at all.

Strengthened, against log forgery. On syslog the record boundary is the datagram, so peer-controlled text carrying a newline stays inside the entry it arrived in and cannot fabricate a record — a property the line-framed classes do not have and the rejected stdout stream would not have given either. It makes message framing, not escaping, the structural answer. Because syslog is now the production class on every platform rather than Linux's alone, the property holds everywhere flor actually runs, and private/shared carry the residual hole only where a human is already watching the stream. The reserved native sinks would carry framing of their own, so nothing regresses when one lands.

Unchanged. The self-stamped clock, the level conventions, and the counter layer.