Architecture

Architecture

System architecture for the CellState agent memory platform.

For core concepts (entities, memory types, coordination patterns), see MENTAL_MODEL.md. For deployment and configuration, see OPERATIONS.md.

Overview

CellState is an event-sourced agent memory framework built on:

• PostgreSQL as the ledger — ACID transactions, RLS tenant isolation, pgvector embeddings
• Projections derived from events — working set, graph links, vectors
• Rust API as orchestrator — single binary, all-in-one server
• Pack/PCP for declarative agent config — Markdown + TOML compiled to runtime rules

The architecture follows a “Redux for agents” pattern:

• Event log = actions (immutable facts)
• Projections = store (derived state)
• Selectors = SQL queries
• Middleware = mutation pipeline + receipts + idempotency

┌──────────────────────────────────────────────────────────────────────┐
│                         CellState STACK                              │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    HTTP/WebSocket API Layer                    │   │
│  │                 (crates/api/src/routes/*)                      │   │
│  └──────────────────────────┬───────────────────────────────────┘   │
│                             │                                        │
│  ┌──────────────────────────▼───────────────────────────────────┐   │
│  │                   Mutation Pipeline                            │   │
│  │  Stage A: Context Assembly (I/O)                               │   │
│  │  Stage B: Policy Evaluation (pure — no I/O)                    │   │
│  │  Stage C: Database Commit (atomic transaction)                 │   │
│  │  Stage D: Receipt Generation + Event Relay                     │   │
│  └──────────────────────────┬───────────────────────────────────┘   │
│                             │                                        │
│  ┌──────────────────────────▼───────────────────────────────────┐   │
│  │              DbClient (SQL-First) + CachedDbClient            │   │
│  │                    [LMDB read-through cache]                   │   │
│  └──────────────────────────┬───────────────────────────────────┘   │
│                             │                                        │
│  ┌──────────────────────────▼───────────────────────────────────┐   │
│  │             PostgreSQL 18 (with pgrx extension)               │   │
│  │  ┌────────────┬────────────┬────────────┬────────────┐       │   │
│  │  │  Entities  │  Events    │  Graph     │  Vectors   │       │   │
│  │  │ trajectory │ event      │ edge       │ pgvector   │       │   │
│  │  │ scope      │ event_dag  │ link       │ embeddings │       │   │
│  │  │ artifact   │            │            │            │       │   │
│  │  │ note, ...  │            │            │            │       │   │
│  │  └────────────┴────────────┴────────────┴────────────┘       │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  PARALLEL SYSTEMS:                                                   │
│  ┌─────────────────────────┐  ┌────────────────────────────────┐   │
│  │      HybridDag           │  │   Read-Through Cache            │   │
│  │  ┌────────┬──────────┐  │  │   (LMDB + ChangeJournal)        │   │
│  │  │  LMDB  │ PG Cold  │  │  │                                  │   │
│  │  │  Hot   │ Storage  │  │  │   Sub-millisecond hot reads      │   │
│  │  └────────┴──────────┘  │  │                                  │   │
│  └─────────────────────────┘  └────────────────────────────────┘   │
│                                                                      │
│  14 SUPERVISED BACKGROUND JOBS                                       │
│  (change relay, saga cleanup, summarization, TTL, decay, ...)       │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Crate Structure

cellstate/
├── crates/
│   ├── core/             # Domain types, identity, event primitives
│   ├── api/              # HTTP API, mutation pipeline, routes, jobs
│   ├── storage/          # Event DAG, LMDB cache, change journal
│   ├── pcp/              # Pack Configuration Protocol — validation engine
│   ├── pipeline/         # Pack compiler — TOML/YAML/Markdown → runtime rules
│   ├── pg/               # PostgreSQL extension (pgrx) — triggers, SQL functions
│   ├── cli/              # CLI binary for admin operations
│   └── test-utils/       # Test fixtures, generators, env helpers
└── packages/
    └── rust/             # Rust SDK — typed HTTP client

See the crates README in the CellState source repository for the dependency graph.

Mutation Pipeline

All state-changing operations flow through a 4-stage compiler-like pipeline. No route handler writes directly to the database.

Stage A: Assemble     Stage B: Gates      Stage C: Commit     Stage D: Receipt
   (I/O OK)              (Pure)             (Atomic)           (Relay)
     │                     │                   │                  │
     ▼                     ▼                   ▼                  ▼
Fetch budgets      Run policy checks    Single txn with     Return proof,
+ policy config    No I/O allowed       entity + event      wake ChangeRelay
+ idempotency      Pass or Reject       + receipt insert

Stage A: Context Assembly

Fetches everything needed for policy evaluation: token budgets, PCP config, tenant limits, cloud metering quotas. All I/O happens here so Stage B is pure.

Stage B: Policy Gates

Pure functions that receive a frozen ValidationContext and return Pass or Reject(PolicyViolation). No async, no database calls. This makes policy enforcement deterministic and testable.

Gates include: PII redaction verification, token budget enforcement, schema validation, PCP policy checks, and cloud usage limits.

Stage C: Database Commit

A single PostgreSQL transaction writes the entity, semantic event, receipt, and change log entry atomically. If the server crashes after commit but before broadcast, the ChangeRelay picks it up on restart (transactional outbox pattern).

Stage D: Receipt Generation

Returns a ReceiptEnvelope with a Blake3 hash proving the mutation occurred. Wakes the ChangeRelay to broadcast the event via WebSocket.

Route Handler Pattern

Routes become minimal — HTTP validation only, pipeline owns all mutation semantics:

pub async fn create_artifact(
    State(pipeline): State<Arc<MutationPipeline>>,
    AuthExtractor(auth): AuthExtractor,
    Json(req): Json<CreateArtifactRequest>,
) -> ApiResult<impl IntoResponse> {
    let result = pipeline.create::<ArtifactResponse, _>(
        auth.tenant_id, scope_id, &req,
    ).await?;
    Ok((StatusCode::CREATED, Json(result.entity)))
}

Event DAG

CellState maintains two event rails:

Semantic Event Store (cellstate_event)

Domain events for audit and replay. Tenant-scoped with RLS. Human-readable event types like TrajectoryCreated, NoteUpdated.

Causal DAG (cellstate_event_dag)

Hash-chained event log with causal ordering. Uses a HybridDag architecture:

• Hot tier: LMDB for sub-microsecond reads of recent events
• Cold tier: PostgreSQL for durable storage and historical queries

Events are identified by EventKind (16-bit encoded: upper 4 bits = category, lower 12 = type) and linked via parent_event_ids for causal ordering. Each event carries a Blake3 hash chain for tamper detection.

EventKind encoding:
  0x0xxx: System      0x1xxx: Trajectory   0x2xxx: Scope
  0x3xxx: Artifact    0x4xxx: Note         0x5xxx: Turn
  0x6xxx: Agent       0x7xxx: Lock         0x8xxx: Message
  0x9xxx: Delegation  0xAxxx: Handoff      0xBxxx: Edge

Event Flow

Mutation Request
    │
    ├─→ Pipeline Stage C: DB Commit
    │   └─→ INSERT entity → trigger → cellstate_changes
    │   └─→ INSERT cellstate_event (semantic, same txn)
    │
    └─→ Event Emission (post-commit)
        └─→ HybridDag.append()
            ├─→ LMDB hot tier (sync, μs)
            └─→ PgEventDagStorage (async, ms)

Context Assembly

POST /context/assemble retrieves and ranks memory for an agent’s next action.

Sources

7-Factor Scoring

Items are ranked by a deterministic composite score:

score(item) = w_v * S_vector       (embedding similarity)
            + w_w * S_warmth       (access frequency + decay)
            + w_r * S_recency      (time decay)
            + w_a * S_abstraction  (L0/L1/L2 preference)
            + w_g * S_graph        (graph distance from anchor)
            + w_k * S_keyword      (exact/partial match)
            + w_c * S_causal       (event DAG proximity)

Weights are configurable per-agent via pack config. Defaults sum to 1.0 and are normalized by the pack compiler if they don’t.

Storage Layers

Read-Through Cache (LMDB)

The CachedDbClient wraps DbClient with an LMDB-backed read-through cache. Cache invalidation uses a ChangeJournal:

• Single instance: In-memory journal, invalidated on every pipeline write
• Multi-instance: PG change journal poller syncs invalidations across instances (~2s staleness window)

Database Schema

The PostgreSQL schema includes ~20 entity tables, all with:

• tenant_id column with RLS enforcement via cellstate_current_tenant_id()
• Change triggers that write to cellstate_changes for the relay
• UUID primary keys with typed ID wrappers in Rust

Key migrations: V1 (base schema) → V5 (RLS) → V15 (principal model, working set, graph links) → V16 (event DAG cold storage).

Background Jobs

14 supervised background jobs run in the same process. Each uses the supervisor pattern: panics trigger automatic restart with backoff. Job health is tracked and exposed via /health/ready.

See OPERATIONS.md for the full job table.

Observability

• Prometheus metrics on /metrics (28+ counters, gauges, histograms)
• OTLP trace export to Jaeger/Tempo/Honeycomb
• Sentry for error tracking with release tagging
• Structured logging via tracing with span context
• x-cellstate-trace-id header on every HTTP response

Diagrams

Mermaid source files are in docs/diagrams/: