CyberCLI

← All engines · Substrate

Syslog substrate consumer

stable

v1.0.0 · last updated 2026-06-06 · source: cybercli.syslog_substrate.consumer

§1

What it does

Drains the Unix socket that Vector writes parsed NDJSON events to, routes each line to the matching per-connector IngestPipeline based on `cybercli_connector`, and handles back-pressure + reconnect when Vector restarts or the engine is busy.

§2

Why you care

The SyslogIngestConsumer is the engine that bridges Vector (the substrate transport) and the SocPipeline (the engine's reasoning loop). Without it, Vector would just be a parser writing to a void. With it, every event Vector parses lands on the right connector's IngestPipeline within milliseconds, with back-pressure so a spike doesn't OOM the engine.

The consumer is also the engine that survives Vector restarts. When Vector's systemd unit cycles (config change, crash, kill), the socket goes away briefly; the consumer's reconnect loop notices, sleeps with exponential backoff, and resumes draining the moment Vector is back up. Operators don't see ingest gaps; the audit log shows the brief reconnect as honest event metadata.

Without this engine the substrate story would be incomplete. The Vector half handles the parsing; this consumer is what makes the parsed result an actionable event in the engine's reasoning. The line between 'Vector territory' and 'engine territory' is this Unix socket.

§3

How it works

On `consumer.run()` the engine opens the configured Unix socket as a SOCK_STREAM client, sets a non-blocking read mode, and enters the read loop. Each line read from the socket is parsed as NDJSON; lines that don't parse are sidelined to the dead-letter log with a `parse_failed` audit row, not dropped silently.

Parsed events carry `cybercli_connector` (the slug the VRL parser stamped) and `event_type` (the sub-type within that connector). The consumer looks up the matching IngestPipeline in the runtime registry and calls `pipeline.process(raw_event)`. The downstream stages (dedupe, normalize, soc-pipeline) handle the rest.

Back-pressure is handled at the socket layer. If the downstream IngestPipeline is slow (the SocPipeline is busy, the SQLite is contended), the consumer's read loop blocks; Vector's sink buffers and eventually flushes to disk if the buffer overflows. That's deliberate — losing throughput is better than losing events.

Reconnect runs on `BrokenPipeError` or `ConnectionRefusedError`. The consumer's outer loop catches, logs a `substrate_socket_disconnected` row, sleeps the backoff interval (1s, 2s, 4s, 8s, capped at 30s), and retries. When Vector comes back, the consumer logs `substrate_socket_reconnected` with the gap duration and resumes draining.

§4

Configure & observe

CLI

$ cybercli syslog-ingest run

The long-running consumer entrypoint — drives the socket drain loop as a foreground process. The systemd unit (`cybercli-syslog-ingest.service`) wraps this verb.

$ cybercli syslog-ingest check

One-shot variant: read a single JSON line from stdin, push it through the consumer's routing logic, report the resulting IngestPipeline status.

Dashboard surfaces

/connectors (Substrate health)

The substrate consumer's stats (events drained, parse failures, reconnects) show on the substrate-backed connectors' health cards.

/audit (filter outcome=substrate_*)

Forensic replay of substrate connection lifecycle — every disconnect + reconnect with the gap duration.

Audit row shape

Outcomes emitted
substrate_socket_connectedsubstrate_socket_disconnectedsubstrate_socket_reconnectedsubstrate_event_parse_failed
Parameters block

Lifecycle rows carry parameters.gap_duration_seconds + parameters.events_drained_in_window. Parse-failure rows carry parameters.raw_line_sha256 (a fingerprint of the bad input — never the raw line itself) and parameters.error_class.

§5

Status

Stable — the consumer has been the substrate-to-engine bridge since 2026-05-10. The acceptance install on LXC 200 has drained 51+ events through this loop with zero stuck-socket incidents.

Known limit: one consumer per substrate. A multi-socket install (sharded by connector category) is a v1.x scale item. For the per-install event volumes we see today, one consumer is well within capacity.

Shipped 2026-06-07 (store + API half): `cybercli.syslog_substrate.DeadLetterStore` is a SQLite-backed queue with append / list_recent / replay_one / replay_pending / purge_replayed / counts_by_reason. Parse-failed frames keep their raw bytes + sha256 so a replay-after-VRL-fix can re-parse the exact bytes Vector saw. Consumer wiring (replace the JSONL-append in _dead_letter with store.append) lands in v1.x; the `cybercli syslog-ingest dlq ...` CLI surface ships in v1.x.

How this engine talks to the rest

Upstream · feeds this engine
Downstream · this engine feeds