Architecture

This document records the design decisions for opssignal and the trade-offs behind them. If you're implementing a new provider, sink, or binding, read this first.

Components

flowchart TB
    subgraph callers[Callers]
        PY["Python: opssignal.notify()<br/>Airflow failure_callback"]
        CLI["CLI: opssignal notify"]
        HOST["Future Java / Go hosts"]
    end

    subgraph bindings[Thin binding layers]
        PYO3["crates/python<br/>PyO3 extension"]
        CLIC["crates/cli<br/>clap binary"]
        FFI["crates/ffi<br/>C ABI, catch_unwind on every entry point"]
    end

    subgraph core["crates/core (all business logic)"]
        SRC["source + reload<br/>file / https / s3 / gs / az, cache, polling"]
        CFG["config<br/>signal.yaml + ${VAR} interpolation"]
        CLIENT["SignalClient<br/>owns a Tokio runtime"]
        VAL["signal<br/>validate, id, timestamp, dedup key"]
        ROUTE["routing<br/>resolve(), console fallback"]
        PROV["providers<br/>console, slack, webhook,<br/>pagerduty, teams"]
        SINK["sink<br/>SqliteSink"]
    end

    PY --> PYO3
    CLI --> CLIC
    HOST -.-> FFI
    PYO3 --> CLIENT
    CLIC --> CLIENT
    FFI --> CLIENT
    SRC --> CFG
    CFG --> CLIENT
    CLIENT --> VAL
    CLIENT --> ROUTE
    ROUTE --> PROV
    CLIENT --> SINK
    PROV --> EXT["Slack / Teams / PagerDuty / HTTP endpoints / stdout"]
    SINK --> DB[("signals.db")]

The Python and FFI bindings each hold one process-global SignalClient, built lazily from signal.yaml on the first call. The CLI builds one per invocation.

Lifecycle of one signal

sequenceDiagram
    participant App as Caller
    participant Client as SignalClient
    participant Router as RoutingConfig
    participant Provider
    participant Sink as SqliteSink

    App->>Client: notify_async(input)
    Client->>Client: validate, assign id + timestamp + dedup key
    Client->>Sink: record(signal), status pending
    Client-->>App: returns
    Client->>Router: plan(signal)
    Router-->>Client: delivery targets (never empty, console fallback)
    par each target, concurrently
        loop each provider of the target until one succeeds
            Client->>Sink: provider_outcome(pending)
            Client->>Provider: send(signal), up to 3 attempts with backoff
            Provider-->>Client: Ok / Err
            Client->>Sink: provider_outcome(delivered / failed)
        end
    end
    Client->>Sink: complete(signal, delivery result)

notify_sync runs the same steps but blocks the caller until the result is known or the timeout expires.

A delivery plan is a list of targets, each an ordered list of providers. A failover route's providers join the one shared failover target; each provider of a fanout route is a target of its own. Targets run concurrently and never share retries, so a provider that delivered is never sent the signal again because another provider failed. The overall status is delivered (every target succeeded), partial (some did) or failed. A config without fanout has a single target and behaves exactly as it did before delivery modes existed.

Routes that set dedup or rate_limit are checked while the plan is built, on the caller's thread, so repeats are judged in the order they were sent. A route held back adds no providers; if every matching route was held back, the plan is empty and the signal is stored as skipped (the console fallback is only for signals that match nothing). The history is a throttle table in the sink's SQLite file, written inside an IMMEDIATE transaction, so every process on a host that shares the file sees the same windows. A send that ends failed is removed again so it can't suppress the next occurrence. Any error reading the history lets the signal through. Coordinating across hosts is deliberately not part of the embedded SDK.

notify_async callers wait only for the local sink write, never for a provider. Its deliveries run on the client's runtime, so a process that exits would cut them off. SignalClient::flush(timeout) waits for them, and every binding calls it on the way out: dropping a Rust SignalClient flushes for up to 5 seconds, the Python package registers flush() with atexit, and C hosts call signal_flush(timeout_ms) before exiting (the library holds a process-global client that is never dropped). The CLI uses notify_sync, so it has nothing in flight. A signal still undelivered when the wait runs out stays in the sink as pending.

Signal model

pub struct Signal {
    pub id: Uuid,              // core-generated, never caller-supplied
    pub source: String,
    pub event_type: String,
    pub severity: Severity,    // closed enum, not a free string
    pub title: String,
    pub message: Option<String>,
    pub environment: Option<String>,
    pub metadata: serde_json::Value,
    pub timestamp: DateTime<Utc>,  // core-stamped
    pub dedup_key: Option<String>, // computed if caller omits it
    // ...
}

id and timestamp are always core-generated. Every downstream component (sink, dedup, retry tracking) needs a stable identity and ordering guarantee that doesn't depend on every language binding remembering to set these correctly.

Delivery semantics: sync vs async

Both notify_sync and notify_async share one internal pipeline (validate → record → route → deliver → complete). The only difference is whether the calling thread blocks on the delivery future.

Default for the Airflow adapter is async (fire-and-forget). A blocking call on a flaky Slack webhook inside an on_failure_callback would add latency and a new failure mode to the very DAG run being monitored. Callers who want a delivery guarantee can call notify_sync directly with an explicit timeout.

The design rule is that persistence to the sink happens before delivery is attempted, regardless of sync/async: the signal is stored as pending, then updated to delivered or failed. A failed delivery is durably recorded, and a signal whose process died mid-delivery is still visible as pending rather than lost. This is what makes delivery analytics and MTTR reporting possible later without re-architecting.

A sink error never stops delivery. If record fails it is logged and the signal is delivered anyway; complete writes the whole row, so the outcome still lands if the sink has recovered.

The storage boundary

pub trait SignalSink: Send + Sync {
    /// Before delivery: store the signal as pending.
    fn record(&self, signal: &Signal) -> Result<(), SignalError>;
    /// After delivery: store the outcome (an upsert; must work without `record`).
    fn complete(&self, signal: &Signal, result: &DeliveryResult) -> Result<(), SignalError>;
    /// During delivery: one provider's outcome, pending then delivered/failed.
    /// Optional; defaults to doing nothing.
    fn provider_outcome(&self, signal: &Signal, outcome: &ProviderOutcome) -> Result<(), SignalError>;
}

A sink is a write-only side effect, not a queryable API. This SDK does not expose querying, dashboards, or aggregation — that's explicitly out of scope for the open-source core (see "what this is not" in the README) and is the hosted control plane's job, reading from the same SQLite/Postgres file OSS users already populate.

• SQLite is the default sink: zero-config, works out of the box.
• Postgres / S3 are opt-in for centralized analytics across multiple workers — relevant once you outgrow a single-node Airflow deployment, since SQLite's single-writer model will start serializing writes under concurrent load.

Routing

Routing config matches signals against rules and resolves to a list of provider names.

Invariant: unmatched signals always fall back to the console provider. A signal that matches no route must never silently disappear — that failure mode is worse than not having the SDK at all. This is enforced in RoutingConfig::resolve, not left to configuration.

Config sources and reload

SIGNAL_CONFIG can name a local file, an https://, s3://, gs:// or az:// object, an AWS AppConfig profile (appconfig://), an Azure App Configuration key (azappconfig://), or one item of DynamoDB (dynamodb://), Firestore (firestore://) or Cosmos DB (cosmos://) (crates/core/src/source.rs). The cloud code lives in crates/core/src/cloud/, one module per cloud, and non-storage services are signed with the credentials object_store already resolves (SigV4 on AWS; Entra tokens for the service's own audience on Azure), so no cloud SDK is in the tree. The cloud stores sit behind the aws, gcp and azure cargo features of opssignal-core, so the core stays light. They run on the same reqwest/rustls stack as the providers rather than pulling in a second one.

Cloud code ships only in cloud packages. The base CLI and C library compile none of it; their aws, gcp, azure and all features build a variant that is the base plus that cloud. The Python wheel compiles none either: each cloud ships as its own wheel (crates/python-aws, -gcp, -azure), installed through the opssignal[aws], [gcp] and [azure] extras, because pip extras can add packages but cannot change what is compiled into one. At startup the base binding imports whichever add-ons are installed and registers each one's fetch with the core for every scheme of its cloud (source::register_cloud_fetcher). The core calls it on a plain thread for that scheme, with the same cache and fallback rules. An add-on is a thin wrapper around source::fetch_blocking; it has no logic of its own.

Two reliability rules shape this code:

• A config source is never a reason to lose a signal. A remote source that is unreachable or serves a broken file falls back to the last good cached copy, then to the zero-config default (console only). The cache holds the raw text before ${VAR} substitution, so it never contains secrets. ${aws-sm:...}, ${gcp-sm:...} and ${azure-kv:...} placeholders are resolved by secrets.rs after each file is fetched (values cached in memory for SIGNAL_SECRETS_TTL_SECS); a store that can't be reached is handled like an unreachable source.
• A reload is all or nothing. reload.rs polls the source on an interval (default 60 s) and, when the text or a fingerprint of its resolved secrets changes and loads cleanly, swaps routes and providers together as one Pipeline behind an RwLock<Arc<_>>. In-flight deliveries keep the Arc they started with. A failed reload keeps the current pipeline. The sink is never reopened live.

A config can be layered: a file may extends: others, and SIGNAL_CONFIG may list several (crates/core/src/chain.rs). The chain is walked depth first into a list of files, lowest priority first, with each file loaded once, cycles rejected, and a depth (8) and file (64) limit. Config::from_layers merges it: providers by name, later routes first, storage from the last file that sets it. Routes are checked only after the merge. Each remote file is cached on its own, and the reloader re-walks the whole chain on every poll and compares the texts of every file, so a change to a shared default reaches all the teams that extend it. A remote file may not extend a local path.

Polling is used instead of filesystem notifications because Kubernetes updates ConfigMap mounts by swapping a symlink, which file watchers often miss, and the same loop covers remote sources.

FFI panic safety

Every #[no_mangle] extern "C" function in crates/ffi must be wrapped in catch_unwind (see crates/ffi/src/lib.rs::guarded). A Rust panic crossing the FFI boundary is undefined behavior in the host language and can crash the calling process outright — for an embedded SDK, that's worse than the failure it was supposed to report on.

The Python bindings (crates/python) additionally translate Result::Err into real Python exceptions via PyO3 — never a silently swallowed None.

Why two separate Rust crates for bindings (ffi vs python)

crates/ffi is a generic C ABI layer intended for future Java/Go bindings via JNI/cgo. crates/python uses PyO3 macros directly, which is a different (and for Python, more idiomatic and lower-overhead) binding mechanism than hand-rolled C ABI. Both wrap the same opssignal-core logic; neither contains business logic of its own.

What's deliberately out of scope

• Event bus / message queue semantics
• Event sourcing, complex event processing
• Long-running workflow orchestration
• Java and Go bindings (designed for, not built — see roadmap in README)
• Querying/dashboards over stored signals (hosted control plane territory)

Revisit triggers

Decision	Revisit when
Embedded Tokio runtime	If embedding conflicts with host apps that have their own async runtime become common
SQLite default sink	When users run multi-worker Airflow and need unified cross-worker history
No bounded buffering/backpressure	If burst scenarios (thousands of signals in a tight loop) overwhelm provider rate limits
Hash-based dedup	If semantically-identical-but-textually-different signals need correlation (likely an AI-correlation premium feature, not OSS core)