Field note006 Filed24 Apr 2026 TagCheckpointing Read14 min ByPranav Varshney · Viswa Kotra

Local disk is not progress.

GKE and training stacks already know the rule: if the node dies and the bytes never left it, you did not save. Nodus checkpointing writes verified manifests to provider-independent object storage, then advances a compare-and-set pointer in Postgres. Reclaim only works because progress already escaped the machine.

The durability rule

Under checkpointed, a workload resumes only from a verified manifest. Local disk, process memory, and partial uploads are never progress. Final artifacts use the same protocol: completed requires a committed final manifest.

restartable and ephemeral workloads track progress by cursor or not at all. They may still emit final manifests when the work produces durable outputs.

The runner must never claim a checkpoint is complete just because local files exist.

Stage contract

Owns: checkpoint coordinator
Before: PrepareCheckpoint validates generation ownership → short-lived scoped upload credentials
After: immutable verified manifest; CAS advances latest-manifest pointer; checkpoint.committed
Transport: gRPC + direct signed object-storage upload
Durable ack: CommitCheckpoint verifies every object SHA-256, writes manifest, then compare-and-set pointer

Checkpointing is concurrent with execution, not a post-run stage. Storage is tenant-isolated, encrypted in transit and at rest, and independently durable from the active supplier.

Pilot today Prepare and commit are POST /v1/runner/checkpoints/prepare and /commit over the same versioned HTTPS JSON the runner uses for heartbeats, and parts move through a Nodus object endpoint into a file-, Postgres-, or MinIO-backed store rather than presigned puts against S3. Every correctness property below is already in that path: fenced generations rejected, each object checksum-verified at commit, the manifest row immutable, the latest pointer advancing by compare-and-set.

Prepare, commit, CAS

A fenced generation cannot prepare or commit. Stale (generation, sequence) loses the compare-and-set.

nodus · checkpoint runner-v1

# Concurrent with execution
1. Runner reaches a safe application boundary
2. PrepareCheckpoint
     validate gen ownership + runner token
     return fencing_token + staging prefix + upload grant
3. Upload parts → provider-independent object store
4. CommitCheckpoint
     VerifySHA256 each file
     write immutable checkpoint_manifests row
     CAS advance latest_manifest (gen↑ or same gen seq↑)
     emit checkpoint.committed

A fenced generation cannot prepare or commit, and token and fence epoch must match. A stale (generation, sequence) loses the compare-and-set and surfaces as a conflict. Integrity failure fences the generation and blocks restore until an operator or automated validator establishes a valid prior manifest.

The latest-manifest pointer is a transactional record on (workload_id, stage_id), never inferred from object-list ordering. Redis is not a correctness boundary for it.

What a manifest carries

Manifests are immutable, content-addressed, checksum verified, and retained under an explicit lifecycle.

manifest json

{
  "workload_id": "wl_...",
  "stage_id": "train",
  "generation": 4,
  "sequence": 18,
  "files": [{"uri": "...", "sha256": "...", "bytes": 123}],
  "runtime_contract_version": "runner-v1",
  "restore": {"command": ["python", "train.py", "--resume", "..."]},
  "outputs": { "handoff": { "uri": "...", "sha256": "...", "bytes": 123 } },
  "final": false
}

final: true completes the stage; all stages done → workload completed
outputs enable multi-stage handoffs without shared process state
Partial checkpoints are garbage-collected only after unreachable from every committed manifest and outside forensic retention

Policy is an RPO / RTO trade

Default cadence comes from expected interruption risk, checkpoint duration and size, restore time, remaining budget, and deadline. Customers may tighten the interval. One that eats too much execution time surfaces as a route tradeoff rather than being silently accepted.

Internally we publish, per workload class, a checkpoint RPO (maximum acceptable lost progress) and a restore RTO (target time from route availability to resumed runner). The chosen cadence has to be explainable as the cost and risk trade needed to meet them, which is the same objective function feeding cost_to_complete in routing.

How this closes the loop

Market keeps the book fresh. Profiling compiles the envelope. Route scores it against the book and returns a path plus a ceiling. Bidding holds a lease. Execution runs gen N. Checkpoint makes progress portable. When reclaim arrives, the event is stored before anything acts on it, fencing kills gen N’s write authority before anything replaces it, recovery restores from the latest verified manifest, and the same route → bid → execute path starts gen N+1 against current offers. There is no second recovery stack. That is the whole design.

Independence from supplier reclaim is the whole point. Progress survives the machine that produced it.

All field notes