Observability

Posture

C0 observability is node-local by constraint (Scope · Maintenance): every flor process logs to stderr, the host's system logger stores the stream, and nothing leaves the node — no collector, no upload, no dashboards. The upload path arrives as a metrics push to the coordinator's node-facing service (C1+/B1, sketched under Collection).

Within that constraint, this page covers the scope's five maintenance events — loss of connectivity, crashes and abnormal terminations, slowdown of traffic flows, excessive resource usage, security issues — for all of flor's own running components: the vertex, the agent, the coordinator, and (in its small, operator-tool way) retectl. Tenant workloads are out of scope: the supervised-program contract provisions their log sink, and everything else about their observability is their own business.

Two consumers set the bar for every choice below: an operator reading the host log during an incident (journalctl, or whatever the host's syslog daemon writes), and a pilot user exporting logs to attach to a bug report (user-guide obligations). The future third consumer — the Coordinator's Monitor step (B1 sketch, where node observations feed the control loop) — consumes the counter layer through a transport that does not exist yet; C0's job is only to make sure the counters will not need re-instrumenting when it does.

Two signals

C0 emits exactly two, and the boundary between them is deliberate:

• Metrics — OpenMetrics counters in a per-process registry, incremented explicitly in code. Aggregate, cheap, always on. Their eventual consumers are the operator, and the Coordinator's control loop from B1.
• The event stream — log lines carrying span context, written to stderr. Detailed, situational, node-local, retained by the host logger.

What the two do share is a vocabulary: the same token means the same thing whether it labels a metric or fields an event, which is what lets one bounded/unbounded rule govern both.

The event stream is built on the tracing crate, and this page uses its four concepts throughout — worth fixing here, because the design below leans on all of them:

Spans here provide local context only — which connection, which sync, which request an event belongs to. They are not distributed traces: nothing correlates them across nodes, by design (see Collection). Why tracing, what it replaces, and when an event must be a tracing:: one rather than a log:: one are settled under Facade.

The classification rule

Within the event stream, whether something becomes a line or a counter increment is decided by one rule, applied again whenever a new event is added later:

1. Anything a peer or the network can trigger at wire rate is a counter. A per-occurrence line would turn a rogue-connection flood — precisely a security event we must observe — into a second, self-inflicted DoS of the log. Counters have constant cost per event and never suppress; the detail (which address, which cert error) may additionally go into a flood-safe line.
2. State transitions and lifecycle moments are log lines, edge-triggered. A workload started, a set activated, a peer became unreachable, a peer recovered. Never per-attempt: a vertex retrying a dial logs the transition into failure once and the recovery once; the attempts themselves are counter increments. This matches the reconnect-tolerance convergence model of the agent's lifecycle — retry loops are normal operation and must not narrate themselves.
3. Instantaneous quantities are sampled at readout, not stored. Open-connection count, per-connection RTT and loss (from the QUIC stack's own stats), resource usage. They exist at the moment of a snapshot; between snapshots there is nothing to maintain.

The flood-safe Layer

Rule 1's "additionally" is enforced by a Layer, not by rate-limiting code at each call site — one implementation, one place to test, no per-site state to get wrong. It is a C0 component.

The key is (callsite, one bounded field). A callsite is tracing's static identity for a single macro invocation in the source — one per source location, created once at compile time, the same for every event that line ever emits. So for this denial:

// core/transport/endpoint/verifier.rs:212 — one callsite
tracing::warn!(
    target: LOG_TARGET,
    initiator = %id,
    acl = "ingress",
    "Connection denied by ACL"
);

the key is (verifier.rs:212, "spiffe://acme.rete/user/mallory"): at most one line per window (order of ten seconds) for that line of code about that initiator, then suppression with a count flushed at window end (suppressed 412 similar). Keying per callsite keeps one flooding site from muting unrelated ones; adding the field means a legitimate denial for user/alice still appears while user/mallory floods.

initiator is this family's choice, not the rule. Which field joins the callsite is per event family, resolved by a documented precedence in the Layer — initiator where the event has one, else reason, else the callsite alone. A handshake failure has no authenticated identity to key on, so it keys on reason; the closed enum keeps it bounded just as an identity would.

Whatever is chosen must be bounded — that is the actual invariant. Socket addresses and anything else attacker-controlled must never enter the key: the suppression map is state, and state keyed on attacker-controlled values is itself a DoS vector. The unbounded values still appear in the line that does get through, as fields; they just never become map keys.

It applies to tracing:: events only. Records bridged from the log facade all share one static callsite per level — every log::warn! in the codebase is the same callsite — so they cannot be rate-limited per site and pass through unlimited. That is exactly why flood-prone events must be tracing:: events with fields: a path that can flood is converted, not exempted.

Vocabulary

Metric labels and event fields draw on one vocabulary. Two invariants hold globally; everything else is per-context.

Bounded values may be metric labels; unbounded values are event fields only. This is not a metrics convention imposed on the design — it falls out of the architecture. Florete distinguishes names (identities: stable, operator-authored, few) from locators (layer-private addresses: ephemeral, many, never names of a layer's users). Names are labelable; locators are not.

A token names an axis; each metric or event family declares its own closed value set. result on a sync is unchanged/applied/failed; on a coordinator request it is ok/not_modified/rejected/error. There is no global enum and no attempt to unify them — the sets are documented with the family that uses them. What is global: values are always closed enums, never free text and never a formatted error message. A reason built from {e} would blow up cardinality on the first novel error string; the human-readable error belongs in the event's message, the closed code in reason.

Common core — two axes, used everywhere with per-family values: result (how an operation ended) and reason (why it failed).

Vertex

Agent: workload (bounded), cause — crash/sync, signer (the signing principal on a verified artifact — bounded), set_version (monotonic, unbounded over time → field only).

Coordinator: operation — artifact_fetch/publish, node (bounded), result.

Three naming decisions worth their reasons:

• initiator / target / signer, not principal. A bare principal is ambiguous: Florete has TLS principals (user, service, node, vertex) and signing-only plane principals (management-plane, control-plane). Naming by role in the operation rather than by credential mechanism resolves it and reuses terms the scope already defines. tls_principal is reserved for the rare event that genuinely cannot distinguish the role.
• link_peer…, not peer. The prefix names the relationship and layer; the suffix names the representation. So link_peer is the peer vertex's identity and link_peer_sock_addr its UDP locator where one exists — over radio links, or in a mesh overlay, there is no socket address and the identity stands alone. peer unqualified would also have collided with the northbound TCP client, which is a different thing entirely (client_sock_addr).
• operation, not endpoint or route. endpoint already means the QUIC TransportEndpoint in the data plane, and route would collide with network routing. operation also describes intent rather than URL shape, so the coordinator's paths can change without touching a metric.

Event coverage

The five maintenance events, the signal that actually reveals each, and where the signal comes from. "Designed here" means new instrumentation this page specifies; "supervision design" means it already falls out of the agent page; "delegated" means the OS provides it and flor deliberately does not duplicate it.

Three honest gaps, stated rather than papered over:

• Slowdown has no baseline in C0. Nothing records "normal", so nothing can detect "slow" — detection needs collection and history, which is the B1 Monitor's job. What C0 provides is the raw material for a human: byte counters and RTT samples in every snapshot, retained by the host logger, so an operator diagnosing "it feels slow" can compare now against an hour ago by searching the log for stats.
• Memory leaks are trends, not moments. The OS shows current usage (delegated); the trend only exists across observations. Same answer: OS accounting now, automated trend detection when collection exists.
• flor's visibility starts at the QUIC handshake. A pre-handshake packet flood (garbage UDP at the socket) is dropped inside the QUIC stack and largely below flor's ability to attribute; interface-level counters belong to the OS. C0 sees its consequences (handshake failures, resource pressure), not the packets.

Resource usage is delegated deliberately (not by omission): systemd accounts CPU and memory per service unit out of the box, launchd/macOS has ps and Activity Monitor, and the user guide documents reading them. The known gap — the whole scope shares one unit, so per-child split needs ps rather than systemctl status — is pilot-tolerable. Self-sampled per-process gauges join the snapshot when a collector exists to consume them (reserved).

Counters: the metrics-ready layer

Counters are Prometheus-model metrics from day one — only the transport is deferred. Each process owns a typed metric registry (the prometheus-client crate — the official Prometheus Rust client: typed counter/gauge families keyed by derived label structs, an explicit registry object that fits the DI style, OpenMetrics text encoding as a plain function call, and none of the exporter/server machinery C0 must not build). What C0 fixes is the part that is expensive to change later — the data model:

• Naming follows Prometheus conventions: flor_<component>_ prefix (flor_vertex_, flor_agent_, flor_coordinator_ — the coordinator's config sits outside the flor contract, but it is still flor's own binary and shares the namespace), _total suffix on counters, base units in names. Tenant workloads never appear in this namespace.
• Labels are bounded by construction, per the vocabulary rule: identities are operator-authored and small, failure reasons are closed enums. Socket addresses, ports, connection IDs — anything unbounded — never become labels; they appear only as event fields.
• No histograms in C0. Bucket boundaries want tuning against a real consumer, and a histogram serialized into a log stream is noise. Counters plus sampled RTT lines cover pilot diagnosis; histograms are an additive registry change when the push transport lands.
• Stored vs. sampled. The registry holds only what must survive between readouts — monotonic counters. Instantaneous quantities (rule 3 above) are read live at snapshot time; when a transport exists they become on-demand collectors against the same registry, which is the standard pattern, not a redesign.

The starter set — normative for naming style, deliberately small, grown by need:

A cardinality note on the pair: metrics carrying both initiator and target produce a cross-product. At C0 scale (≤20 principals each way) that is a few hundred series — fine — but it is the one place cardinality could creep, so carrying both is a per-metric decision rather than a blanket rule. flor_vertex_handshake_failures_total deliberately carries neither: a failed handshake often has no authenticated identity to attribute, which is why the attempted one (when there is any) belongs in the event's fields, not the metric's labels.

Sampled at readout, not stored: open connections, per-connection RTT and loss estimates (the QUIC endpoint's own connection stats), active workload state (already in status).

What is deliberately not built in C0: no /metrics listener, no push, no scrape, no metrics database, no new sockets. The C1+/B1 transport is a push to the coordinator, for the same reason B1's config watch is client-initiated (Coordinator § Reserved, not built): nodes already dial the coordinator, so a push reuses that connection, whereas a pull/scrape would invert the topology and make the collector responsible for reaching every node — a fan-out the poll model already avoids. Whether that push is Prometheus remote-write or periodic OpenMetrics text is a C1 decision that the registry serves either way. See Reserved, not built.

Readout

Every process periodically snapshots its own counters into its own log stream. With per-workload control channels reserved rather than built, the vertex has no channel to hand counters to the agent in C0 — its log stream is its counter surface, and the system logger's retention is what turns point-in-time counters into history.

• Format: one OpenMetrics-style sample per line, under the component stats — stats: flor_vertex_bytes_total{initiator="user/alice",target="billing",dir="to_target"} 1048576. Greppable at 2am (journalctl --grep stats on Linux, grep stats on the macOS file, then by metric name), zero invented syntax, and trivially machine-convertible because each line is the industry exposition format.
• Cadence: changed-only every 300s (FLOR_STATS_INTERVAL seconds overrides; 0 disables). Changed-only keeps a quiet node's log quiet; an unchanged counter's last value is in an earlier snapshot, which is what retention is for. The interval is a node-local knob, not operator intent — an artifact field is reserved for the day fleet-wide tuning matters.
• Full snapshot on demand: SIGUSR1 dumps the complete registry plus sampled quantities immediately — the incident readout, no waiting for the timer. Unix-only, which C0 is; Windows arrives in C1 together with the control channel that supersedes this.
• Full snapshot at drain: the SIGTERM handler emits a final complete snapshot before exit, so a clean stop never loses the tail.

flor agent status reports agent-owned state only. Over agent.sock, structured: per-workload state (running / crash-looping / stopped, pid, uptime, restart counts, last exit code and time), sync state (last result and time, active set version), and the agent's own counters. It does not proxy vertex counters in C0 — that requires the per-workload control channel; when the channel lands, aggregation into status is additive. Until then the operator's incident pair is flor agent status (is it up, is it syncing) plus the host log (what is it doing).

The log stream

Ownership: every process writes its own stderr, and the agent provisions what is behind it. The agent installs a sink on each child's fd 2 at spawn and never sees a log byte (ADR-0014), so a child keeps logging across an agent crash. The agent chooses the sink and a child cannot redirect its own output — attribution by sink is enforced, because the child holds only a descriptor it did not open. Attribution within a sink shared by siblings is cooperative: the syslog identifier and the rendered workload name are both content a child writes, and a child that lied would still be writing into the sink it was given. On the private class the two coincide, since the sink is the attribution. Multi-rete hosts separate scopes by service instance (flor-agent@<scope>); within a scope, the sink separates workloads.

Three sink classes, not per-OS formats. What a line must contain depends on what the sink adds — and on whether siblings share it. The format switches on the class, never on a compiled-in OS assumption. syslog is the only class normal operation uses, on every platform; the other two exist for running something by hand:

The agent provisions only syslog in C0. private is what a lone process gets when nobody provisioned anything for it — one workload run straight from a shell to debug it — and its defining property is that no sibling shares the sink, so nothing needs distinguishing in the line. shared is the opposite: siblings do share it, so the name must be rendered. The distinction is about sharing, not about files: a private sink is often a terminal.

The syslog class is named for the transport. What it requires is a local syslog socket accepting datagrams, which is as ordinary on an embedded Linux box running busybox syslogd as on a systemd server — /dev/log is the portable path, a symlink to journald's own socket where journald is what answers. Behind it the daemon differs, and so do two guarantees: the record boundary holds everywhere (one datagram is one record, by construction), while multi-line rendering and the size cap are the daemon's. journald keeps embedded newlines in the message and accepts well over 100 KB; rsyslog escapes control characters into #012 by default and caps at a few KB; busybox caps lower still. So the class promises one record per message everywhere, and readable multi-line only where the daemon obliges.

The <N> prefix is per message, not per line, and does not replace the level word. One frame per write is what buys the boundary: the whole event, however many physical lines it spans, is one datagram and therefore one record at one priority. The prefix is machine-facing and consumed by the daemon — journald parses it into PRIORITY and strips it, so it never reaches the stored message — whereas the level word is content, and journalctl's default output shows no priority at all beyond coloring the line. Dropping the word would leave the common view levelless. Exactly one representation of the level survives in the message text on every class; the frame is the extra that makes -p warning work.

The workload name is rendered only where the sink does not supply it. The syslog frame carries it as the identifier, so rendering it into the message body would duplicate what any export already shows — and journalctl -t vertex is a per-workload filter, which a name inside the text would not be. On private there is nothing to disambiguate: one process, one sink. A shared sink supplies nothing at all, so there the rendered name is the attribution, and it is what keeps a combined capture readable.

The syslog severity mapping is narrow on purpose. flor frames under facility daemon and uses four of syslog's eight severities:

Nothing above err is ever emitted. crit, alert and emerg are the severities hosts wire to operator attention — rsyslog's stock configuration broadcasts emerg to every logged-in user — and no failure flor can observe on one node warrants that. A component that cannot proceed is an ERROR; the escalation is the operator's to configure, not ours to claim.

The collapse at the bottom is the second reason the level word stays in the message: DEBUG and TRACE share debug, so the syslog priority alone cannot tell them apart and -p debug selects both. The word is what distinguishes them, on every class.

Unframed writes still land. Anything that reaches fd 2 without flor's frame — a panic backtrace, a stray println!, a library writing to the descriptor directly — arrives as a datagram the daemon files at its default priority with no identifier. Nothing is lost, only attribution, and flor's own binaries close even that with a panic hook that routes the payload through the logger. This is why the sink is a datagram socket on fd 2 rather than a private connection opened by the logger: one sink catches everything the process emits.

The class is injected, never inferred. Whether siblings share an fd is not observable from the fd, so nothing probes for it. The agent — which either provisioned the sink or passed its own down — states the answer in FLORWL_LOG_SINK (the spawn environment). A process started by hand has nobody to tell it, and unset means private — the class whose defining property is precisely that nothing was arranged for it. An operator who wants a hand-run process framed for syslog sets FLORWL_LOG_SINK=syslog alongside whatever they attached to fd 2 — explicit, like every other class selection.

All three values are accepted by hand, the default one included. FLORWL_LOG_SINK=private means exactly what unset means, and it exists because absence is a poor way to say something deliberately: it cannot be written in the VAR=value cmd prefix form, so a shell that already exports syslog — a dev environment file, a shell descended from the agent — can be corrected only by an unset that outlives the one command it was meant for. A test matrix over the three classes hits the same wall in the same place. Making the default sayable costs one match arm and removes a special case rather than adding one. What survives is the narrower and more useful rule: the agent hands out two of the three, never private, because under an agent a sink was always chosen. An unrecognised value is a startup error, not a quiet fall back to the default — guessing is how a process ends up framing for a sink it does not have.

The agent takes its own class from configuration, not from its surroundings. It uses the configured socket and provisions the same for its children. Two failures are hard errors, not downgrades — a daemon that cannot log durably has lost the ability to explain its own failures, so it refuses to start instead — but they exit differently, because they are not the same kind of problem:

The second is worth naming because it looks benign: a stream connects, writes succeed, and every record boundary silently disappears, turning one flood into an unreadable smear.

The syslog socket must be there at start, and stay the same socket. This is the arrangement's one standing host dependency. Children receive a connected descriptor they did not open and cannot replace — the property that makes attribution enforceable is the same property that makes the sink unfixable from the inside — so a daemon that unlinks and recreates its socket on restart leaves the agent and every workload writing into a dead peer. Nothing in C0 detects it. Recovery is restarting the scope — systemctl restart flor-agent@<scope> or the launchd equivalent — which tears the workloads down through the ordinary stop path and re-provisions each one as the agent re-forks it. An agent crash is not recovery here: children deliberately outlive it (ADR-0014), so they keep their broken descriptors.

The fix is not ours to write, and it already exists: let the system supervisor own the socket. Where the service manager holds the listening socket and hands it to the daemon, the socket is created once at boot and outlives every restart of the thing behind it — the inode never changes, so no descriptor anywhere goes stale, and datagrams written while the daemon is down wait in the socket buffer instead of failing. This is not an exotic arrangement: it is exactly what systemd's syslog.socket does for /dev/log, which is why the default Linux path has never had this problem and why rsyslog and syslog-ng ship units that take their socket from systemd rather than binding it themselves. So C0 states a requirement rather than a hazard:

The configured destination should be a socket the system supervisor owns. Where it is, a syslog daemon can restart freely and flor never notices. Where it is not — a daemon binding its own path — flor accepts the loss of records across that daemon's restarts, until the scope is restarted.

On Linux the requirement is satisfied by doing nothing: /dev/log is socket-activated on every systemd host. On macOS the mechanism exists — launchd's Sockets key holds a SockPathName and passes it to the daemon through launch_activate_socket(), the direct analogue of sd_listen_fds() — but it takes a daemon that asks for it, and whether Homebrew's rsyslog or syslog-ng build does is unverified and needs checking before the user guide promises it. Until it is, macOS is the second tier, knowingly: the scale of the failure is a logging gap and not a service outage — flor keeps forwarding traffic — and a syslog daemon on a pilot laptop is restarted rarely and deliberately.

Two directions are recorded for C1, neither chosen: making the agent notice its own failing writes and shut down — stopping its workloads first, exactly as it does for any other stop, then exiting 69 so the wrapper brings the scope back with fresh sinks throughout; this is portable and reuses the exit path above rather than adding one, and the deliberate teardown is the whole point, since orphaned children would otherwise keep the descriptors the restart exists to replace. Or declaring the dependency to the service manager, where systemd's PartOf= propagates a syslog restart into ours for one directive and launchd offers no equivalent. A proper rotation story is further out and is the reserved filter process's to carry: replacing a live child's fd means either handing it a new descriptor over the per-workload control channel — cooperation a third-party workload cannot be trusted to implement — or putting a flor-owned logger process back in the path for exactly those workloads, which is the confinement that ADR already reserves.

The agent makes one choice, not two. Where its own records go and what it provisions for its children are the same decision, because only two combinations are ever wanted: syslog for both in production, and the agent's own stderr for both when a human is watching. Nothing needs the agent on syslog while its children go to a terminal. So the surface is one flag, flor agent run --log-stderr, and it is off by default:

The choice is configuration; the flag is only its interactive form. Nobody should retype --log-stderr all day, so it resolves in the usual order — the flag, else FLOR_LOG_DESTINATION in the environment, else [log] destination in config.toml, else the built-in default:

[log]
destination = "unix:/dev/log"   # or "stderr"

The value is one string rather than a { kind, path } pair, and the reason is not that flat beats structured. An environment variable is flat, so any structured setting must be flattened before it can be overridden, and the established answer is one variable per leaf, named mechanically from the key path — Spring Boot's SPRING_DATASOURCE_URL, .NET's Logging__LogLevel__Default, Viper's dot-to-underscore replacer. What that convention rejects is serializing a record — several independently-meaningful settings — into one variable, and the test for whether something is a record is whether anyone would ever want to override one part without the other. Spring's own answer is the instructive one: the datasource URL is a single leaf while username and password are separate ones, because deployments routinely change credentials without changing the host. A log destination fails that test in the opposite direction — unix: alone selects nothing, and /dev/log alone does not say what speaks it. Kind and locator are one fact, so the string is a leaf, not a flattened record, and FLOR_LOG_DESTINATION=stderr / FLOR_LOG_DESTINATION=unix:/run/rsyslog.sock are the entire surface.

Which is why the shape has such conventional company. PostgreSQL's log_destination takes stderr, syslog, csvlog, eventlog; systemd's StandardOutput= takes journal, null, file:PATH, append:PATH, fd:NAME; every *_URL and *_PROXY variable in general use is the same primitive. A closed set of kinds, some carrying a locator after a colon, is an ordinary scalar with a parser — the thing the convention is protecting, not the thing it warns about.

The test that matters is whether it extends, and the destinations already on the horizon are the ones with no locator at all. The native structured sinks — journald's field protocol, macOS os_log, Windows' Event Log — need no path: they are journald, oslog, eventlog, bare kinds the grammar takes without a new variable or a new shape. Note that journald would not be a synonym for unix:/dev/log even though journald answers both — one is the syslog transport carrying flor's <N> frame, the other the native field protocol; different framing, different sink class, correctly different values. Remote syslog extends the same way (udp://host:514, or tls://host:6514 for RFC 5425), and a null discard is there for free if anything ever wants it.

And the form has a stated stopping point, which is what keeps it from drifting into the blob the convention warns about. When a destination needs more than a locator — a TLS syslog target wanting a CA bundle, a client cert, a queue depth — those are independently-overridable settings, and the answer is not a longer string. It is a named table in config.toml that the destination selects by name, with the extra knobs as leaves of their own. C0 has no such destination and may well never need one; recording the exit is what makes the single string a bounded decision rather than an open-ended one.

FLOR_LOG_DESTINATION and FLORWL_LOG_SINK are different things, and the words say so. A destination is where the agent sends its records, chosen by whoever runs the host; a sink class is what shape a descriptor has, told to a child that cannot choose. The prefixes already separate flor's own configuration from the workload contract; the nouns keep them from being read as the same setting.

Two things follow from the table. The agent injects two of the three classes — private is not something it ever hands out, because it always chose a sink; the value is nonetheless sayable by hand, for exactly the case where nobody did. And the flag is not a class name: the class vocabulary describes what a sink is, while the agent is choosing between production and attached debugging, so naming the flag after a class would offer combinations that do not exist. --log-stderr says what actually changes, and follows sshd -e — the same flag, for the same reason, in the same words.

Detecting the situation instead was considered and rejected. A tty on fd 2 is a real observable, but it answers the wrong question: it correlates with "a human is watching" without meaning it, and the failure it invites is the expensive direction — a service that happens to have a terminal would log there and lose everything when the terminal closes, which is precisely the silent-loss case the hard errors above exist to prevent. The reverse mistake is loud and self-correcting: a developer running the agent on a machine with no syslog daemon gets an exit and a message naming the flag and the setting. Established daemons land in the same place — sshd -e, PostgreSQL's log_destination — because the destination of a daemon's own log is an operator's decision, not something to infer. The one thing flor does read from the fd is color, a true property of the sink that changes nothing about where records land.

Color is orthogonal and resolved locally, by is_terminal() on fd 2, because tty-ness is a true property of a process's own sink whoever else writes there.

Deployment in a Docker container requires a choice. Container has no /dev/log, and the answer is a choice, not a detection. Two arrangements work, and which one is right depends on whether the host or the runtime is meant to own the logs. Bind-mounting the host's syslog socket keeps the syslog class and every guarantee above. Otherwise flor agent run --log-stderr in the image: the agent passes its own fd 2 — the pipe the container runtime is already reading — down to its workloads, and the runtime's log driver collects the combined stream, which is the one place a pluggable-driver relay genuinely earns its keep. The cost is the shared class's: attribution moves into the line, and per-workload filtering becomes grep over docker logs. Either way it is stated in the image's command, not sniffed from /.dockerenv — the same no-inference rule as everywhere else, and the reason a container behaves the same whoever runs it.

The class is announced, never inferred by the reader — each process names it in its startup line, the same discipline the clock source follows. It decides where the log went, what it is framed by, and which of this page's guarantees hold, none of which an operator should have to deduce from a running system.

Flood handling belongs to the daemon, where there is one. journald rate-limits per service, dropping with a Suppressed N messages record rather than blocking the writer; other syslog daemons have their own. The private and shared classes have none, so a flooding workload grows a file or backs up a pipe. Bounded through B2 because every supervised workload is flor's own and carries the flood-safe Layer; beyond that the fix is a filtering logger process per workload, which the spawn contract already admits without changing the workload's side.

A message is one write. The logger renders a whole event — every physical line of it — into one buffer and emits it with a single write on unbuffered stderr, so nothing splits a message at its own newlines and no other thread of the process interleaves. That last guarantee is the standard library's stderr lock rather than the kernel's, which is why the writer stays std::io::stderr and not a raw fd 2: a hand-rolled writer would expose every multi-threaded process to tearing its own messages.

Every class caps a message, and the smallest cap wins. On shared, a pipe is atomic only up to PIPE_BUF — 4096 bytes on Linux, but 512 on macOS, POSIX's floor and therefore ours; above it a concurrent writer can land inside a message. On syslog the cap is the daemon's datagram limit, generous under journald and a few KB under rsyslog or busybox; beyond it the write fails rather than tears. Only private has no cap, since an O_APPEND write of any length lands whole. So 512 bytes is the size budget for a single message, and an error stack is the one construct that routinely exceeds it: an oversized Report is rendered as several complete messages, each attributed in its own right. A message that can be torn — or silently rejected — is not readable.

Two rules for readers follow, and they are why flor logs exists: a record begins at a timestamp and a level, never at a \n, and per-workload filtering is a sink property (above), not a grep.

macOS needs a real syslog daemon, and Apple's own is not one. /var/run/syslog is a compatibility shim into unified logging whose semantics differ exactly where it matters: info and debug are memory-only unless enabled per-subsystem, so flor's default INFO record would silently fail to persist, and there is no per-workload filter to replace journalctl -t — every vertex is the same flor process. Calling os_log natively instead means FFI, its default redaction of %s arguments, and a second line format for one OS. So C0 asks a macOS host to run rsyslog or syslog-ng (Homebrew makes this routine) and points config.toml at its socket — one class, one format, one path on every platform, at the cost of one documented setup step (Installer § Logging). Native structured sinks stay reserved, not built.

Sanitization and the debug console

flor's lines carry peer-controlled text — hostnames, SPIFFE IDs, remote error strings — so escape sequences and control characters inside a message are neutralized at write time. Write-time is the load-bearing part: it is what makes the stored log safe for whoever reads it later, including on the private and shared classes, whose readers offer nothing (a terminal or tail -f renders whatever bytes are there, unlike journalctl, which by default abbreviates a field containing unprintable characters as blob data). This is a display defense, and ESC, BEL, BS, FF and DEL are what it covers today.

Newlines are a separate problem, and how bad it is depends on the class. A peer-controlled \n reaching a line-framed sink forges a record: on private and shared, the reader's only frame is the line, so an injected newline plus a plausible timestamp and level is a complete fabricated entry. On syslog it cannot be — the record boundary is the datagram, so peer text stays inside the entry it arrived in and can at most append lines to a genuine record. Message framing is therefore the structural fix, and escaping is the fallback for classes that lack it; the same holds for the native structured sinks.

Where framing does not close it, the rule is: a newline in data is escaped, a newline flor writes is structure. CR and LF join the neutralized set for every value a message or field carries, which is what makes the reader rule above ("a record begins at a timestamp and a level") true rather than merely conventional. The one multi-line construct flor emits on purpose is an error stack, and this is why it must be rendered by the logger rather than interpolated into a message: rendered, its newlines are flor's own structure and the peer-authored strings inside it are still values; interpolated with {e:?} it is one value, and the rule flattens it.

C0 ships without either half, and that is a deliberate deferral rather than an oversight: the production class is syslog, where framing already closes the hole, and private and shared are attended debugging in a controlled environment. The same gap exists in every tracing-based program, since the stock sanitizer covers escape sequences and not newlines.

One relaxation, and it is explicit. error_stack renders a Report's styling into the message string, so a colored error stack and a sanitized message are mutually exclusive. A developer running a reproducer in a controlled environment wants the color; a node in the field must never have it. The switch is FLOR_LOG_UNSANITIZED=1, and it is:

• honored only when fd 2 is a tty — the one sink with no storage behind it, so no durable log is poisoned;
• a runtime choice, not a build one — a release-optimized binary in a lab is exactly the case that needs it, and a debug build on a node is exactly the case that must not have it;
• announced at startup at WARN, the same discipline the clock source follows — a relaxed process says so, and it is never inferred.

Every other sink sanitizes unconditionally, at every level, in every build.

Timestamps and the clock

Every line carries a timestamp flor wrote; the sink's own stamp is never the authoritative one. Two independent reasons, and the first already binds in C0:

• The private and shared classes supply nothing. A terminal capture or a container's combined stream carries no time but flor's own, so self-stamping is what makes such a capture readable at all. It is also what makes a time-bounded export possible without journalctl --since: a lexically sortable ISO-8601 prefix is what grep/awk can slice.
• Rete time is coming, and no sink can produce it. B2's network clock names log timestamps as a rete-time consumer — cross-node correlation at ~10 ms, which is most of what a timestamp is for in a distributed system. Rete time is a per-rete virtual clock the agent maintains as coefficients over the host's monotonic clock, and that design's hard rule is that the host clock is never disciplined: a multi-rete node needs one clock per trust domain, and stepping the host's clock is the host owner's business, not any rete's. A sink stamps from the host clock by definition, so only the emitting process can stamp in rete time.

The stamp therefore comes from a clock abstraction, not a direct wall-clock read at the call site — a one-line indirection in C0 that is the whole of the migration later. In C0 that clock is the host wall clock, rendered ISO-8601 with offset at millisecond precision (millisecond because ~10 ms is the rete-consistency target these stamps will eventually have to honor). When rete time lands, the same field carries it and every flor line in the rete becomes correlatable — no call-site change, no format change, nothing relearned by whoever reads logs. journald keeps adding its receive stamp throughout, which stays useful as exactly that: host time, for lining flor's lines up against the rest of the system's.

Two consequences worth writing down now rather than discovering later:

• The clock source is announced, never inferred. Each process logs its clock source at startup and at any transition (clock: host → rete (step +1.42s, ε=8ms)). Not cosmetic: rete time may not exist yet when a process starts, and B2 permits exactly one forward step at bootstrap — a reader who sees stamps jump must be able to tell a clock correction from a stalled process.
• Monotonicity improves across the migration. C0's host wall clock can step backward under NTP, so C0 stamps are not monotonic — journald's are not either. Rete time is slew-only after bootstrap, so the change makes the stream better behaved; until then nothing reading these stamps may assume monotonicity.

Levels — the convention every flor component follows, default filter INFO, controlled by RUST_LOG (EnvFilter directives, per-component overrides included):

Facade: tracing, adopted in C0 and used sparingly

C0 adopts tracing as the logging facade and subscriber, replacing env_logger as the output layer. The line format above is unchanged — it is reproduced by a custom FormatEvent — and RUST_LOG keeps working: EnvFilter takes env_logger's target=level syntax and extends it with span and field selectors ([conn{initiator=…}]=trace, which per-connection spans make useful), dropping only its trailing /regex filter on message text. The env directives layer over the component's configured filters rather than replacing them, so a component-only RUST_LOG raises that one component and leaves the rest at their defaults. Records from the log facade are captured by LogTracer, so third-party crates and every existing log:: call site flow through the same filter and formatter.

The reason for adopting it now, rather than in C1, is span context: a span attached to a connection is inherited by every event emitted inside it, including events from code that knows nothing about spans. In the vertex's SOCKS5 path this turns unattributable lines into attributed ones with no change to the call sites that emit them:

before:  DEBUG socks5_inbound: Cannot resolve target 'typo.example.com': …
after:   DEBUG socks5_inbound: conn{initiator=…/user/carol client_sock_addr=127.0.0.1:43974 target=typo.example.com} Cannot resolve target: …

Under concurrency that is the difference between a readable log and an unreadable one, and it is why the decision could not wait: C0 is already rewriting the output layer for the sink classes and the self-stamped clock, and that layer is written once either way.

Instrument sparingly — spans mark operations, never packets. C0's spans: one per proxied connection in the vertex, one per sync in the agent (with set_version as a field), one per request in the coordinator. Nothing on a per-packet or per-datagram path, where span creation would be a real cost rather than a rounding error. retectl gets none — a sequential CLI has no concurrency to attribute.

When an event must be a tracing:: event rather than a log:: one. The log bridge carries a fixed field set (message, log.target, log.module_path, log.file, log.line) — a bridged record is an opaque string plus source location, and log's own key-values are not forwarded. Therefore:

Anything a Layer must act on — not merely print — is a tracing:: event with fields. Anything only a human reads may stay log::.

In practice: flood-prone events (handshake failures, ACL denials, connect failures, resolve failures) become tracing:: events with fields, because the flood-safe Layer keys on callsite and initiator and cannot do either through the bridge. New code — the agent and coordinator — uses tracing:: throughout; there is no reason to add new log:: call sites once the facade has moved. Existing non-flood-prone log:: calls stay as they are and keep working.

Fields, not interpolation, for anything that is data. tracing::debug!(target = %target, reason = "not_rete_name", "Target unresolvable") renders like the interpolated version today and is machine-actionable later; log::debug!("Cannot resolve target '{target}'") is a string forever. The message stays a short, stable phrase with no interpolated values — that is what keeps it both greppable and matchable by callsite.

Identity in observability data

Connection-level attribution by SPIFFE identity: yes. The difference between "something is flooding us" and "user/mallory is flooding us" is most of the diagnostic value of the security counters, and the cardinality risk that makes per-identity labels dangerous elsewhere is absent by construction — the identity set is operator-authored, small, and closed (the vocabulary rule). Unauthenticated peers have no identity to attribute: they are counted in reason buckets, and their socket addresses appear only in flood-safe DEBUG/WARN lines as fields, never as labels. Request-level attribution stays out of reach by design — flor is L4/L5; that is the deferred L7 question.

The consequence, stated: logs and snapshots carry a per-identity activity trail (who connected where, when, how many bytes) that outlives the connections. In C0 that trail is node-local, inside the host's existing security boundary, retained on the system logger's schedule — acceptable for pilots. The moment an upload path is designed (C1+/B1), this becomes a data-handling question — retention, aggregation, who may read what — and must be answered there, not inherited silently. It is also the point at which the trail becomes attractive to an external consumer such as a SIEM (Collection), which sharpens rather than softens the question.

Restart and retention

Counters are process-lifetime by design; the log stream is the durable record. A restart zeroes the registry — and that is fine precisely because the readout path already moved the history out of the process: periodic snapshots survive in the host log, a clean stop emits a final snapshot, and a crash loses at most one interval's tail — while the crash itself is captured by the agent's supervision lines and restart counter, which live in the process that did not crash. The agent's own crash is the host logger's to remember, and its durable supervision facts (version high-water-marks) are already persisted state — config state, not metrics. Nothing metrics-shaped is written to disk; agent/ stays supervision-only.

Retention is therefore the host logger's, identically on every platform — flor writes to the configured syslog socket everywhere and owns no log file at all, so what is kept, for how long, and how it is rotated belong entirely to whichever daemon answers. That is the load-bearing reason for requiring a socket rather than writing files: children inherit fd 2 at fork, so a rotated file would leave every child writing to a nameless inode, and no reopen in the agent could reach them. Rotating a supervisor's own file is easy; rotating its children's is not, absent a per-workload control channel deferred past C0.

Two traps the user guide must state. Both belong to a daemon, not to an operating system, which is why there is no per-platform split here:

• journald's persistence is distro-dependent. Storage= may be volatile, in which case the whole record dies with the boot — precisely the boot worth investigating. Node setup must check it.
• Anything else keeps its own files, not the journal. Under rsyslog, syslog-ng, or busybox, retention is that daemon's rotation config and the messages land wherever it writes, so "where are the logs" is answered by reading its config rather than by journalctl.

Collection: where each signal goes

C0 collects nothing. But the shape collection will take determines whether C0's instrumentation survives it, so it is recorded here — as a forward sketch, not a commitment. There are three tiers, not two, and the middle one never leaves the node.

Why the event stream is not shipped, stated as a decision rather than an omission. First, bandwidth is a scarce resource in the environments Florete targets: the Edge Mesh premise is nodes on radio links with hard resource budgets, and continuously uploading the highest-volume, lowest-value-per-byte signal from those nodes is the wrong trade. Second, the Coordinator's Monitor→Analyze→Plan loop consumes time series, not prose — shipping logs would not feed the control loop at all, it would only make the coordinator a log database.

The diagnostic bundle is the tier that connects them. When a counter or a supervision event indicates trouble — a crash-loop declared, sync failing repeatedly, a denial rate spiking — the node uploads a bounded bundle: the flight-recorder ring for the failing span, recent WARN/ERROR lines, and a counter snapshot. Bounded by construction, sent only on a trigger. This is the "report an issue with logs attached" idea promoted from a manual user command into an automatic mechanism, and it is what the flight-recorder Layer exists to produce. There is no trace collector: what would justify one is served by episodic diagnostic upload, which is a far smaller thing.

Cross-node correlation needs none of this. Rete time (from the self-stamped clock), plus initiator/target and version numbers, already lines events up across nodes — the cheap majority of what distributed tracing would buy, at zero protocol cost. Correlation IDs for flor's own control operations (publish → sync → restart across nodes) remain a tractable later addition to the sync protocol; distributed tracing of user traffic stays blocked at the L4/L5 boundary regardless of tooling (the deferred L7 question).

External log consumers stay possible, and are not ours to build. A SIEM, or any log-aggregation stack, ingests from the host logger — journald, syslog, a file — which is exactly where the event stream already lands. We build no integration and block none. That such consumers may be wanted on some nodes and not others (a cloud node shipping a full stream; an edge node shipping a reduced or locally-processed one) is the general shape rather than a special case: stream verbosity and upload scope are per-node policy, which in Florete's model eventually means operator-authored mgmt bounds with the Coordinator choosing within them — lowering a node's verbosity under congestion, raising it during an investigation. Nothing here is designed now; the placement is recorded so it lands in the right layer when it is.

retectl

An operator CLI, not a daemon: its observability is human-readable stderr, meaningful exit codes, and RUST_LOG verbosity when needed — no counters, no snapshots. Its audit story is elsewhere by design: the git history of the rete repo (what was authored) and the coordinator's log plus flor_coordinator_requests_total{operation="publish"} (what was published, when).

User-guide obligations

What this design requires the user guide to document:

• Viewing and following logs: journalctl -u flor-agent@<scope> -t <workload> -f, plus -p warning, --grep, --since, where journald answers; the daemon's own files wherever another syslog does, macOS included. Which class a process got is in the startup line it logs.
• Exporting a time-bounded slice to attach to a bug report. On Linux the lossless form is journalctl -u flor-agent@<scope> --since … --until … -o json --no-pager, which keeps every field; -o short-iso-precise is the human form. Not -o cat — it prints MESSAGE alone, discarding the SYSLOG_IDENTIFIER that carries the workload name on the journal class.
• Handling an exported log safely: do not cat one into a terminal, and treat journalctl -a — which disables the default blob data abbreviation of fields containing unprintable characters — as a deliberate step, not a default.
• Checking retention at node setup: journald Storage=persistent on Linux; on macOS, that the file is rotated (system mode) or how to truncate it (user mode).
• Reading resource usage per OS: systemctl status / cgroup accounting on Linux, ps / Activity Monitor on macOS, including the per-child split.
• The incident sequence: flor agent status, then SIGUSR1 for a full stats snapshot, then the platform's log search for stats history.

Reserved, not built

Recorded so nothing above forecloses them; none is C0 work:

• Metrics push to the coordinator (C1+/B1) — tier 1 of Collection. The node-facing service grows a push endpoint under the coordinator-sync group (why the group is named for the role); nodes push their registries on a cadence; resource identity (rete, node, workload) attaches as labels at the transport boundary — from the spawn environment, never baked into metric families. Wire format — Prometheus remote-write vs. periodic OpenMetrics text — is decided there; the registry serves either. An external TSDB (e.g. VictoriaMetrics, which natively ingests remote-write) is then a sink behind or beside the coordinator; the coordinator itself stays a consumer, because observations feed its Monitor step.
• Flight-recorder Layer — a ring buffer of recent events held in the span's own extensions, dumped only when that span ends badly. The answer to "the data path cannot be logged at production levels", and the payload of the diagnostic bundle. Spans are what make it possible: the buffer has a natural per-connection home.
• Episodic diagnostic upload (B1+) — tier 3 of Collection: bundle plus recent WARN/ERROR lines plus a counter snapshot, triggered by a counter or supervision event, never continuous.
• Runtime log-level control — tracing-subscriber's reload::Layer swapped by signal in C0's shape, or over the per-workload control channel once it exists; lets an operator raise verbosity for one component during an incident without a restart.
• Counter aggregation into flor agent status once per-workload control channels exist — vertices report registries upward; status becomes the node's single readout.
• Readiness and health beyond the liveness window — arrives with the control channel, per the agent design.
• Histograms (handshake and connect latency first), when the push transport gives buckets a consumer to be tuned against.
• Self-sampled resource gauges (per-process RSS/CPU in the snapshot), when a collector consumes them; C0 delegates to the OS.
• Observability policy as mgmt bounds — per-node stream verbosity and upload scope authored by the operator and narrowed by the Coordinator, per Collection.
• Windows logging (C1, with Windows itself): Windows has no syslog socket, so a service there needs something else — an Event Log class, or agent-provisioned private files — selected the same explicit way every class is, so the switch stays in one place. Note what the second option brings back: files the agent owns, and therefore the rotation problem this design avoids by requiring a socket. That is a C1 decision, and it is the reason to prefer the Event Log. Two properties of the platform already constrain the design. A service has no stderr to inherit — the SCM starts it without a console, and a console-subsystem binary is handed null standard handles — so provisioning the sink is not merely the better arrangement there, it is the only way a workload logs at all. Windows has no PIPE_BUF: byte-mode pipes promise nothing about interleaving at any size, and neither does the console (where Rust's own writer additionally splits above 4096 bytes unless the console code page is UTF-8). What Windows does have is atomic append — a handle opened for FILE_APPEND_DATA places each write at end-of-file whatever its length — so a file-backed class carries a guarantee there, stronger than the 512-byte budget POSIX pipes impose, while shared carries none at all and can only be made survivable by messages that attribute themselves individually.
• flor logs [<workload>…] [-f] [--since] — one reader whatever answers the socket: journalctl behind it under journald, the daemon's files elsewhere, with per-workload color applied at read time. Colorizing in the reader is the general answer: storage keeps plain bytes, the operator still gets color.
• Native structured sinks per OS — journald's native field protocol (fields become queryable: journalctl FLOR_PRINCIPAL=…) and macOS os_log. What is reserved is emitting structured fields instead of a formatted line: a second output format, and on macOS FFI — though on Linux the maintained tracing-journald removes most of the cost. Both open their own connection rather than riding fd 2, so they do not replace the syslog class for a workload's unframed output. Both would also carry message framing natively, closing the newline hole for whichever class adopted them.
• Rendering an error stack as structure — the logger emitting a Report's lines itself instead of receiving it interpolated, so its newlines are flor's and the peer-authored strings inside it stay values subject to the newline rule. Also what lets an oversized stack be split into several self-attributed messages, and what gives each line its own <N> prefix on the journal class.
• SGR-allowlist sanitization — replace neutralize-everything with a VT-parsed allowlist (anstyle-parse, already in flor's dependency graph) admitting only foreground-color, bold, dim, italic and underline SGR, and dropping conceal, reverse, backgrounds, every other CSI, and all OSC. A peer could then recolor text and nothing else — no cursor movement, no erase, no hidden text, no window title, no clipboard — which may be safe enough to leave on in production, and would make FLOR_LOG_UNSANITIZED unnecessary. The same value-boundary wrapper carries the newline rule, so the two land together.
• Per-workload log filtering as confinement — a filtering logger process between an untrusted workload and its sink, rate-limiting and sanitizing on the workload's behalf. Needed when tenant workloads arrive (post-B2); the spawn contract already admits it without changing the child's side.
• Clock uncertainty in snapshots — once rete time exists, the stats snapshot carries the clock's ε alongside the counters, so a collector can weight cross-node correlation by the quality of the clock that stamped it (B2 · Network Clock).
• "Report an issue with logs attached" command (post-C0), packaging the export steps the user guide documents manually.

Explicitly not reserved, because it would break an invariant: deriving counters from events. Counter increments stay explicit at the call site, so metrics remain invariant under log configuration.

Non-goals restated from the scope: external collection, upload, or dashboards in C0; distributed tracing at any near milestone.