All field notes

Five systems, one control loop.

Nodus did not invent market making, ad auctions, graph routing, generation reconciliation, or durable checkpointing. We borrowed each from places that already solved them under load, then joined them so an AI workload can finish when capacity disappears.

The product is the join

Customers submit requirements and outcomes: budget, deadline, workload signals, continuity mode. No machine SKU, no catalog row, no recovery plan. Nodus derives provider, route, reservation, execution, and portable progress, then drives the job to verified completion.

That derivation is not one clever service. It is five boundaries with hard contracts between them:

  • Market: continuous normalized capacity book (quant inventory, not a static catalog).
  • Routing: pure optimizer over snapshots (dispatch graph, not a provider SDK call). Profiling — compiling a workload into the execution envelope the router scores — lives inside this boundary.
  • Bidding: auditable capacity commitment (campaign budget / auction, not hope).
  • Execution: desired vs observed generations (controller, not “VM created”).
  • Checkpointing: verified portable progress (durable manifests, not local disk).
01 Market

Capacity book

TTL’d offers, health facts

02 Routing

Pure plan

Snapshots in, graph out

03 Bidding

Lease truth

Ceiling protects recovery

04 Execution

Reconciler

Desired vs observed

05 Checkpoint

Portable progress

Manifest + CAS pointer

Anyone can bolt five subsystems onto a slide. The engineering is making reclaim re-enter the same path without inventing a sixth one.

Happy path, then the loop

Market runs continuously. Checkpointing runs with the workload. Reclaim does not invent a parallel stack; it re-enters route from verified progress.

Authority, on purpose

Three durable systems, intentionally separate:

Customer-visible state lives in Postgres. Temporal never owns it. Kafka never replaces it.
  • PostgreSQL: business truth: workload, route, lease, generation fence, latest-manifest CAS, ledger.
  • Temporal: orchestration progress: timers, retries, waits, recovery saga. Never customer-visible state.
  • Kafka: published facts / transport. Never assumed atomic with DB or provider side effects.

Every customer-visible transition writes Postgres state and an outbox row in one transaction. Consumers are idempotent and at-least-once. Generation fencing and latest-manifest compare-and-set live in Postgres, never in Redis locks and never in a separate lock service.

Control-plane services are Go on EKS. SDKs are Python and Go. Object storage holds checkpoints and artifacts. Redis is for ephemeral limits only.

Pilot today That stack is the destination. The shipping pilot compresses it into one cmd/api process: orchestration in-process rather than Temporal, the outbox relaying out of Postgres with Kafka fan-out optional and usually unprovisioned, checkpoint objects in a file-, Postgres-, or MinIO-backed store rather than S3, an in-process token bucket rather than Redis, Render and Neon rather than EKS. What is not compressed is the authority model. Postgres is still business truth, and every customer-visible transition still writes state and outbox in one transaction. Those are the parts you cannot retrofit.

Invariants that force the join

  • Router is pure. Reads immutable snapshots; returns a plan. Never calls providers. Reservation is a separate, auditable side effect.
  • Cost means completion cost. Routing and bidding optimize expected cost to finish (run plus expected recovery), not minimum hourly price.
  • Completed means durable. Never report completed until the final artifact manifest is verified and the CAS pointer advances (when the continuity mode requires it).
  • At most one writer. Fence before replace. Old runners lose CAS authority immediately.

What this series covers

Each following note opens one boundary: what we took from an existing architecture, what we changed for reclaimable capacity, and what is still being built.

Next after the five: fencing and the reclaim saga, then multi-stage DAGs with manifest-only handoffs.

What we are not claiming yet

Design and pilot are not the same maturity. Wholesale supply adapters exist in the tree but ship disabled, so the default supply set is first-party capacity, and parallel hedging stays off until a policy defines its worst-case committed spend. Signed reclaim ingress has left that list: supplier events arrive HMAC-verified on a timestamped window, persist before anything acts on them, and an operator drill can trigger the same path on demand. Every note carries a pilot today box marking where shipping code is narrower than the contract it implements. We publish the contracts now so each decision assumes reclaim, settlement, and replay will exist later. You cannot retrofit those boundaries. You can only plan for them.

All field notes