Guides

# Retries & idempotency

Submitting a workload commits money and capacity, so a lost response must never become a second run. Two mechanisms: a retry policy that only repeats requests whose failure says nothing about the request, and an Idempotency-Key binding every submission to one workload per tenant.

What the client retries. Client and AsyncClient retry up to max_retries times (default 2, so three attempts) with exponential backoff. The rule is narrow on purpose. Retry when the failure carries no information about validity; never retry when the second answer would match the first.

CapacityUnavailableError arrives as 503, so it falls in the retried class. Retries reuse the same key, so an attempt that does find a route returns a workload rather than a duplicate. Once attempts are exhausted the error surfaces: no route satisfies this brief right now. Widen finish_by, raise budget, or relax data_regions.

How the key is made, and what it covers. POST /v1/workloads requires an Idempotency-Key; a submission without one is rejected before any planning. Omit idempotency_key and the SDK generates a random one per call to run(), which covers that call's own retries and nothing more. Every new run() mints a new key, so a process that dies after sending and calls run() again on restart gets a second workload. Deduplicating across your own process boundaries needs a key that is stable in your domain.

The key is scoped to the tenant and the payload. Two tenants can use the same string without ever colliding. Within one tenant:

• Same key, same payload → the original workload is returned and the response carries Idempotent-Replayed: true. Nothing new is scheduled.
• Same key, different payload → 409, IdempotencyConflictError.
• New key → a new workload, even if the brief is byte-identical to an earlier one.

A replay returns the workload as it is now, not as it was at submission. Replay an hour-old key and the Workload comes back RUNNING, with a populated route and accumulated spend_usd. Code assuming run() always yields a fresh ACCEPTED workload breaks exactly when the idempotency machinery is working. Branch on wl.status or wl.is_terminal.

Why a 409 is not retryable. The key is already bound to a workload whose brief differs from the one you sent. Retrying is not just useless, it is the wrong thing to want: the key names one run, and if a second payload could claim it the key would stop identifying which work you hold. Two correct recoveries. Resend the original payload with that key to replay, or pick a new key if you genuinely want a second run. Mutating the brief and hoping is not one of them.

The practical rule. Derive the key from something your system already treats as unique, such as a job row id or a commit sha, and reuse it verbatim when resubmitting the same brief after a timeout or crash. A lost response then costs one extra request instead of one extra workload.

>>> import nodus
>>>
>>> def submit(client, key):
...     return client.run(
...         model="7B fine-tune",
...         command=["train", "--epochs", "3"],
...         peak_memory_gb=40,
...         expected_runtime_hours=9,
...         budget=600,
...         finish_by="2026-08-02T06:00:00Z",
...         continuity="checkpointed",
...         idempotency_key=key,
...     )
>>>
# stable in your domain — not random, not a timestamp
>>> key = "train:job_8412:attempt_1"
>>> with nodus.Client() as client:
...     try:
...         wl = submit(client, key)
...     except nodus.APITimeoutError:
...         # the submit may already have landed; the key resolves it
...         wl = submit(client, key)
...     except nodus.IdempotencyConflictError:
...         # this key is bound to a different brief — do not retry
...         raise
...     print(wl.id, wl.status)

Reads (get, list, events, artifacts, ledger) carry no key because repeating them changes nothing. cancel is safe to repeat as well: cancelling a workload that is already terminal leaves it in the state it reached.

# Multi-stage workloads

Real pipelines are not one process. Pass stages to run() and the list compiles to a DAG. depends_on names upstream stages, inputs reference an upstream stage's named output, and stages with no unmet dependency run concurrently. Unknown names and cycles are rejected with ValidationError at submission, before anything is planned or reserved.

Each stage declares its own requirements and continuity, and each is routed on its own terms. A shard-and-shuffle step with a 16 GB high-water mark does not belong on the same machine as a 40 GB training step, so the first can land on a vm route and the second on an accelerator route. budget and finish_by apply to the whole graph, because cost to completion is a property of finishing the pipeline rather than any one step.

>>> import nodus
>>>
>>> with nodus.Client() as client:
...     wl = client.run(
...         model="7B fine-tune",
...         image="ghcr.io/acme/trainer:2026.07",
...         budget=900,
...         finish_by="2026-08-02T06:00:00Z",
...         data_regions=["us"],
...         stages=[
...             nodus.StageSpec(
...                 name="prepare",
...                 command=["prepare", "--shards", "64"],
...                 peak_memory_gb=16,
...                 expected_runtime_hours=1.5,
...                 continuity="restartable",
...                 outputs=[dict(name="dataset", path="/out/shards")],
...             ),
...             nodus.StageSpec(
...                 name="train",
...                 command=["train", "--data", "/in/dataset"],
...                 peak_memory_gb=40,
...                 expected_runtime_hours=9,
...                 continuity="checkpointed",
...                 interrupt_tolerance="high",
...                 depends_on=["prepare"],
...                 inputs=[dict(source="prepare.dataset", path="/in/dataset")],
...                 outputs=[dict(name="weights", path="/out/ckpt")],
...             ),
...             nodus.StageSpec(
...                 name="eval",
...                 command=["eval", "--weights", "/in/weights"],
...                 peak_memory_gb=24,
...                 expected_runtime_hours=2,
...                 continuity="restartable",
...                 total_units=512,
...                 depends_on=["train"],
...                 inputs=[dict(source="train.weights", path="/in/weights")],
...             ),
...         ],
...     )
...     done = wl.wait()
...     for stage in done.stages:
...         print(stage.id, stage.status, stage.completed_units, "/", stage.total_units)
#
# stg_prepare  completed  None / None
# stg_train    completed  None / None
# stg_eval     completed  512 / 512

Stage fields. A stage takes the same vocabulary as the brief, scoped to itself, plus the three fields that place it in the graph.

Read progress from workload.stages, a list of StageRun. artifacts() returns the same material as rows carrying stage_id, sha256, and verified, so you can tell which stage produced what.

Handoffs are manifests, not filesystems. When prepare completes, its named outputs seal into a manifest with a sha256 per artifact. train materializes that manifest at /in/dataset and checks the digests; a mismatch fails the stage rather than training on the wrong bytes. A downstream stage never attaches to an upstream stage's live filesystem.

That is a consequence of how the layer works, not a stylistic choice. Stages need not run at the same time, in the same region, or on the same machine, and the machine that ran prepare may be gone before train starts. Capacity gets reclaimed, and holding a machine idle to preserve a directory is the exact cost this layer exists to remove. A checksummed artifact is the only handoff that survives it, and it makes reruns honest: the same digest means the same input.

Recovery is per stage. If the route under train is reclaimed, the workload moves to RECOVERING and train resumes from its last checkpoint on a new route. prepare stays COMPLETED and does not run again, because a verified manifest does not need recomputing to be trusted. Nine hours of training interrupted at hour seven costs the tail of one stage, not the pipeline.

restartable means the stage is safe to run again from the top, so recovery reruns it rather than resuming inside it. That is right for eval, where an hour of rework beats maintaining checkpoint state. Declaring total_units makes the rerun observable: completed_units climbs against a known denominator instead of leaving you with an opaque wait. Either way, only the affected stage moves.

# Versioning

SDK and control plane both follow SemVer and version independently. The SDK's number says nothing about which control-plane build is answering, and does not need to. The wire contract is the boundary.

Pin the SDK in your lockfile, to a major at minimum (nodus-sdk>=1.0,<2). Your control-plane pin is the path segment, /v1. A breaking wire change arrives as a new segment beside the old one, never as a change to an existing one.

Within a major, wire changes are additive. New fields appear on responses and older SDKs ignore what they do not recognise, so a deploy cannot break a running integration by adding data. Hold your own code to the same rule: a webhook handler that rejects unknown keys will break on a change designed to be safe.

Treat enumerations as open sets for the same reason. New WorkloadStatus values and new event types can be added within a major, so branch on the values you handle and keep a default branch for the rest. Prefer workload.is_terminal and workload.succeeded over enumerating terminal statuses yourself — those stay correct when the set grows.

# Billing & budgets

You are billed for execution: the capacity a workload consumed on its way to completion, recovery included. Not seats, not reservations, and not capacity we evaluated and did not use. The unit is the hour a route was held, at the rate on the nodus:… route it was placed on.

Pricing is set per account during the pilot. There is no public price list yet, and we would rather say so than publish a number we intend to change. Rates are agreed before your first billable run.

The budget is a cap, not an estimate. budget becomes budget_usd and bounds the whole graph, not one stage. It is checked against expected cost to completion at planning time, so a brief that cannot finish inside the cap is rejected up front with BudgetExceededError and a 402. A decision, not a transient failure, so the SDK does not retry it.

The cap holds mid-run too. Routing reserves recovery headroom before it reserves a machine. If spend approaches the cap, a resumable workload gets the chance to commit a checkpoint and then settles as FAILED with an error saying so, rather than overrunning your number. You pay for what was consumed, and anything committed first is still in workload.artifacts().

The account has its own ceiling. A per-workload budget bounds one job and nothing else, so a loop of individually reasonable submissions can still spend without limit. Set a monthly cap under Billing in the console and a submission whose budget would take the month past it is refused with 402 and BudgetExceededError before anything is planned or reserved. The error carries monthly_spend_cap_usd, month_to_date_usd, and estimated_cost_usd, which is enough to resubmit against real headroom rather than guess. Accounts start uncapped.

Invoices derive from the ledger. GET /v1/workloads/{id}/ledger returns the entries behind spend_usd. Invoices generate from those same rows via POST /v1/billing/invoices and deliver through Stripe, so billing and execution cannot disagree: a charge not in the ledger is not on the invoice. Billing contact is PUT /v1/billing/profile.

>>> with nodus.Client() as client:
...     wl = client.get("wl_9f3c1b2a")
...     print(f"{wl.spend_usd:.2f} of {wl.budget_usd:.2f} budgeted")
...     for entry in wl.ledger():
...         print(entry.type, entry.debit_usd, entry.credit_usd)
164.20 of 180.00 budgeted

Cancelling is safe at any time, but it does not undo work already done. cancel lets a run commit its checkpoint before settling, and the hours up to that moment are billed. What it saves is everything after.