Marklar: A Distributed System Architecture

I designed Marklar as a distributed system that reduces entropy at every layer. Not metaphorical entropy — I mean the number of mutable truths, the number of places where state can diverge without detection, and the number of implicit authorities that could contradict each other. Every design decision here was driven by one question: what is the smallest set of things that must be true for the system to recover deterministically from any failure mode I can reason about?

This article is a detailed description of the system as it runs today. I am also using this publication as an end-to-end proof of the authenticated-effects pipeline: public GET, content-addressed hash, task grant, action receipt, completed task — all executed exactly once, with no model ever seeing a URL, header, or secret.

The Four Nodes

Marklar consists of exactly four nodes. I removed Cardputer and Scene Visualizer; they are archived and no longer participate in any plane.

VPS is the hub and control plane. It runs the authoritative Taskwarrior instance, the Git release repository, the one-shot DeepSeek dispatcher, the broker-owned model router, and the authenticated-effects socket. Every service on VPS is bounded: no service process can browse the pass vault, no agent can open an arbitrary shell, and no model invocation reaches the network except through the broker's registry-pinned grant system.

HAL is a trusted standalone workstation with a delegated operator shell and full system access. HAL may be offline for extended periods, and the system treats this as normal: VPS is authoritative, and HAL catches up on reconnect.

Vindaloop is a restricted worker. It has no operator shell, no direct vault access, and no authority to modify the task graph. It pulls sealed release archives from VPS and converges independently. If Vindaloop diverges, its state is disposable and can be rebuilt from the archive.

Superclasseur is a trusted portable environment built on PRoot and tmux, holding the second delegated operator-only full shell. It is interactive by design: I use it for exploration, debugging, and manual intervention. It is explicitly not an autonomous runner. The canonical autonomous dispatcher lives on VPS; Superclasseur observes, queries, and assists but never independently initiates authenticated effects or modifies the release pipeline.

Connectivity: Loopback MCP and Reverse SSH

Every inter-node connection is either loopback MCP or an outbound reverse SSH tunnel. There are no inbound connections to any worker. Each worker initiates an outbound tunnel to VPS, and VPS multiplexes control traffic over those established channels. The VPS firewall remains tightly closed: no ports open beyond strictly necessary public endpoints.

VPS is authoritative for all shared state. When HAL is online it maintains a reverse tunnel and receives push notifications; when offline, the system queues changes and replays them on reconnect. There is no distributed consensus, no leader election, no quorum. VPS holds the single writeable copy of every authoritative data store; workers hold derived copies.

HAL and Superclasseur are the only nodes with a full shell. VPS services run in bounded containers with no shell access. Vindaloop is similarly restricted. Shell sessions are logged, and any state mutation outside the automation path is visible as a divergence from the sealed archive.

Secret Management: The Vault and Scoped Projection

The VPS pass password store is the system's secret authority. No service process and no model ever browses the vault directly. Instead, a reviewed manifest projects only scoped credentials into bounded service boundaries. Each service receives exactly the credentials it needs and nothing more.

The manifest is version-controlled alongside infrastructure code. When I rotate a credential, I update the vault, regenerate the projection, and the change flows through the release pipeline like any code change. There is no separate secret-distribution mechanism, no environment-variable sprawl, and no .env file that might drift from its declared schema.

The blast radius of any service compromise is exactly the set of credentials explicitly projected into that service's boundary. The vault itself is only accessible through a narrow, audited path from HAL or Superclasseur.

Release Pipeline: Intent Is a Commit

origin/main is release intent. There is no separate release branch, no staging branch that drifts from production, and no manual promotion step. When I push to origin/main, I am declaring that this commit is the next release.

The VPS publisher executes a deterministic sequence: fast-forward a clean checkout; run the full gate (lint, test, type-check, integrity validators); seal immutable inputs (every dependency hash, configuration value, and projected credential is pinned into a content-addressed archive); install transactionally; restore the pause baseline; and adopt VPS — the hub runs the release it just published.

Workers pull the exact archive by content hash and converge independently. An offline worker does not block promotion. When it reconnects, it discovers the current archive hash, pulls it, and converges. A worker that cannot converge remains pinned to its last-known-good archive and raises an alert.

Rollback is a revert commit. I push a commit reversing the problematic change to origin/main, and the pipeline executes the same sequence in reverse. No separate rollback mechanism, no irreversible migration, and no state that survives a revert without being explicitly versioned.

Task Authority: One Source of Truth

VPS runs Taskwarrior 3.4.2, the sole task source of truth for the entire system. No non-hub Taskwarrior instance ever writes to it. HAL, Vindaloop, and Superclasseur may run local Taskwarrior instances for querying, but those are read replicas: they pull from VPS and never push.

This eliminates an entire class of synchronization bugs. I never reason about conflict resolution between task databases because there is exactly one writer. When I add or modify a task, the mutation happens on VPS and propagates outward. Offline work queues locally and replays on reconnect, but the authoritative state stays on VPS.

Taskwarrior also stores redacted digest-bound receipts for every authenticated effect. When a model triggers an external action — like publishing this article — the system writes a receipt containing the digest of inputs and outputs, but never raw credentials or full payloads. The task database is auditable without being a secondary secret store.

Development Planning: Inference-Only with Structured Review

Planning is inference-only. When I propose a change, I run a session with a fixed panel: the proposer, boring_engineer, marklar_network_expert, and product_lead. These are model instances with distinct system prompts, not separate code paths; the broker routes each role to the configured model.

The panel operates under strict rules. Material blockers — concrete incompatibilities, violated invariants, missing dependencies — require revision. Unsupported nitpicks — stylistic preferences, hypothetical concerns, disagreement without a specific technical objection — become dissent, not veto. The plan moves forward with dissent noted and archived.

This prevents the most common failure mode I have seen in model-assisted planning: models that cannot distinguish correctness issues from taste issues, blocking progress on preferences disguised as requirements. By making the distinction explicit, the panel converges faster and produces technically sound plans.

Model Routing: Generation/SHA CAS

The broker owns model routing and switches by generation and SHA content-addressed storage. The current native model is deepseek-v4-pro. Kimi is an admitted optional development route — the routing infrastructure supports it and I have tested it — but it is not selected for any production path. There is no Anthropic adapter and no Codex adapter.

Model upgrades are deterministic. When I switch to a newer generation, I update the routing table with the new model's hash, and the change flows through the release pipeline. Every worker picks up the new routing on its next convergence cycle. There is no floating "latest" tag, no implicit version resolution, and no risk of two workers using different models for the same logical role without that difference being explicitly declared.

Autonomous Dispatch: One Canonical Runner

The VPS-local one-shot DeepSeek dispatcher is the canonical autonomous runner. It receives a task specification — prompt, tools, completion criterion — and executes exactly one attempt. It does not loop, retry, or request clarification. If the task cannot be completed, it returns a structured failure with the reason and any partial results.

Superclasseur is interactive, not an admitted autonomous runner. This distinction constrains the surface area of autonomous action: every autonomous effect originates from exactly one place — the VPS dispatcher. Every other model interaction is either interactive (Superclasseur) or read-only (planning panels).

Authenticated Effects: The Grant System

Every authenticated effect in Marklar uses exactly one registry-pinned task grant and one per-task Unix socket. The broker owns scoped credentials and an at-most-once intent ledger. When a model needs an external action, it does not receive a URL, API key, or HTTP method. It receives four choices: a profile identifier, a resource alias, schema-checked input fields, and an idempotency key.

The profile and alias resolve server-side to a specific action with a specific input schema. The broker validates input against that schema before acting. The idempotency key — mapped to the task UUID — ensures a second invocation is a replay, not a duplicate effect.

The model never sees credentials, never constructs an HTTP request, and never chooses a URL, header, or secret. Its only authority is to say "execute action X with inputs Y for task Z," and the broker accepts or rejects based on grant existence, schema match, and idempotency validity.

Taskwarrior stores a redacted digest-bound receipt for every executed action: SHA-256 of input and output, profile, alias, and task UUID. No raw content, no credentials, no secret material. I can audit every external effect without exposing credentials to the audit log.

This article is the end-to-end proof. The sequence: a public GET returns the current content; I provide its SHA-256 (2022b50ca5b7a48e79c0fe4366e1cd376d33e5ba8a10af9d8e88a83c45ab3cc8) as the CAS guard; the task grant authorizes exactly one publish; the broker validates the schema, checks idempotency, and executes; the receipt flows into Taskwarrior as a completed task. If any step fails — CAS mismatch, schema rejection, consumed grant — the action does not execute and the task remains open.

Memory and Evidence: Bounded Retention, Deterministic Promotion

Completed results enter bounded short-term memory: a structured log of task inputs, outputs, and receipts with a fixed retention window. Retrieval is by explicit reference — task UUID, content hash, or time range — not semantic similarity search.

Deterministic gates promote durable evidence. A gate is a predicate over a completed result: did the task succeed? Did the output pass validation? Did the receipt match the expected digest? All gates passing promotes from short-term memory to durable storage. Any gate failing keeps the result in short-term memory for debugging, then expires it.

Expiry archives rather than deletes. Archived results are compressed, moved to cold storage, and indexed by digest. They are not searchable in the hot path but are recoverable. Nothing is auto-forgotten: if data is removed, it is because I explicitly chose to remove it, and that choice is a versioned event in the task log.

Closing Principle: Reduced Entropy

I designed Marklar around a single principle: reduce the number of mutable truths. The four-node topology, single authoritative VPS, scoped credential projection, release-as-commit pipeline, sole Taskwarrior writer, inference-only planning with dissent-not-veto, hash-addressed model routing, single canonical dispatcher, and grant-based authenticated effects all serve this principle.

Fewer mutable truths means fewer places where the system can be inconsistent without detection. Explicit authority means I can always answer "who is allowed to do this, and why?" Deterministic recovery means I can always reconstruct state from declared inputs: the Git repository, the vault manifest, the routing table, and the task log.

This system does not optimize for maximum throughput, flexibility, or autonomy. It optimizes for maximum recoverability. When something goes wrong — and something always goes wrong — I want to know exactly what happened, exactly what state the system was in, and exactly how to get back to a known-good state. Every constraint I have described exists because I have, at some point, been burned by its absence.

Marklar is not finished and will never be finished. But every change moves it in the same direction: fewer implicit assumptions, fewer hidden authorities, and fewer places where I must trust that something worked rather than knowing that it worked.