← All engines · Pipeline
SocPipeline
stable
v1.0.0 · last updated 2026-06-06 · source: cybercli.pipeline.SocPipeline
§1
What it does
Drives every connector event through the 8 stages — ingest, prefilter, digest, case-assigned, packet, case-evidence, gate, case-decision, dispatch — and writes per-stage idempotent rows so a reprocess never double-fires an action.
§2
Why you care
The SocPipeline is the loop. From the moment Vector hands an event to the syslog substrate (or a Wazuh poll yields an alert) until the dispatcher pushes a notification, this is the engine that orchestrates the path. Every stage is named, the order is deterministic, and the audit chain records which stage each event completed.
Other AI security products either fire decisions from a black-box state machine (hard to debug when it misfires) or expose the pipeline as a YAML workflow you have to author (easy to misconfigure). CyberCLI's pipeline is a fixed graph with operator-tunable gates — the stages are the same on every install so support and forensics speak the same language.
Idempotency is built into every stage. Each event carries a `decision_id` derived from `(digest_id, action_template_id)`; reprocess the same event ten times and you get one audit row, one case, one dispatch — not ten. That property is what makes pipeline replay safe and makes the chain re-verifiable even after a partial-failure reprocess.
§3
How it works
Stage 1 (INGEST) — the per-connector IngestPipeline normalizes the raw event into a CanonicalEvent + writes the raw bytes to the evidence lake. Dedupe runs here against the connector's cursor store so a re-poll never double-emits.
Stage 2 (PREFILTER) — the stateful prefilter runs cheap deterministic rules (rate-limit, suppression matchers, low-signal markers). High-volume noise gets dropped before paying the AI cost. The detector-runner sub-stage then runs any stateful detectors that own the event (e.g. brute-force, beaconing) and may synthesize a higher-severity event of its own.
Stages 3-5 (DIGEST, CASE_ASSIGNED, PACKET) — the DigestPipeline produces the structured Digest, the case-assignment engine picks or opens a case folder, and the DecisionPacketBuilder turns the digest into the packet the bridge will gate on. Each stage writes its own audit row.
Stages 6-8 (GATE, CASE_DECISION, DISPATCH) — the AuthorityBridge gates the packet, the case-decision writer persists the bridge's verdict to the case folder, and the dispatcher routes the alert to the configured channels. After dispatch the AckLoopEngine enrolls high-CYCON events for repeat-until-ack delivery. The final OUTCOME stage records the loop's lifecycle.
§4
Configure & observe
CLI
$ cybercli syslog-ingest checkOne-shot: read a single JSON line from stdin and push it through the full pipeline. Used to wire-test the substrate without writing to the running service.
$ cybercli audit tail -n 20Each row carries the `stage` that produced it. Filter by stage to see only DIGEST, only DISPATCH, etc.
Dashboard surfaces
The analyst's primary lens. One row per ingested event with the disposition the gate assigned and the case it landed under.
Where events with the same correlation key collect into one investigation. The pipeline writes events into a case at stages 4-7.
Forensic replay of every stage — pick `stage=DISPATCH` to see only the alerts that fired; pick `stage=GATE` to see only the bridge's verdicts.
Audit row shape
soc_pipeline_acceptedsoc_pipeline_rejectedstage_dedup_suppressedstage_prefilter_suppressed Each row carries parameters.stage and parameters.stages_completed (a list of every stage the event made it through before the final outcome). When outcome=soc_pipeline_rejected, parameters.rejection_reason carries the typed error from the failing stage.
§5
Status
Stable — the 8-stage graph has been frozen since 2026-05-12 and has not needed a stage insertion or reorder. The acceptance install on LXC 200 has run 51 events through the full graph with zero stage skips.
Known limit: the pipeline runs single-threaded per consumer process. A high-event-rate install (>500 events/sec) runs multiple consumers to scale horizontally; cross-consumer dedupe relies on the per-connector cursor store + the audit log's `idempotency_key` unique constraint. A unified work-queue / fan-out model is on the v1.x backlog.
Shipped 2026-06-07: detector runner indexes detectors by both event_type (existing) AND source_connector. Detectors declare `source_connectors: tuple[str, ...] = ()` — empty means wildcard (any connector, backward-compatible), non-empty narrows the dispatch. Cross-connector event_types (e.g. auth.login.failed from both Wazuh + Keycloak) no longer pay the cost of unrelated detectors at dispatch time.
How this engine talks to the rest