Failure-handling best practices

The specification gives you four failure_mode values, a conditional-used pattern for merges, and — as of v0.91 — a bounded fan-out with its own tolerance. None of them is a default that fits every case; this page is about which one actually fits which case, with the reasoning that should drive the choice.

The four failure_mode values, and when each earns its keep

HALT is the default for a reason worth respecting, not routing around. A definition that never sets failure_mode is not an unfinished definition — it's one where the author decided every task's failure should stop everything, which is frequently the right call (a delivery pipeline where step 4 failing should not let step 7 ship anyway). Setting CONTINUE everywhere out of habit is how a workflow quietly ships partial work nobody meant to ship.

Worked example — a title-card generation branch and a mix-downmix branch, both optional relative to each other:

{
  "tasks": [
    { "id": "pull_asset", "executor": { "type": "service" },
      "used": ["source_lora"], "produced": ["asset_ref"] },

    { "id": "generate_ident", "depends_on": ["pull_asset"],
      "executor": { "type": "agent" }, "ai_role": "generative",
      "used": ["asset_ref"], "produced": ["ident_image"],
      "failure_mode": "CONTINUE" },

    { "id": "downmix_audio", "depends_on": ["pull_asset"],
      "executor": { "type": "saas_api" },
      "used": ["asset_ref"], "produced": ["stereo_mix"],
      "failure_mode": "CONTINUE" }
  ]
}

Neither branch depends on the other, so CONTINUE on both is not adding tolerance that would not otherwise exist — depends_on already means a task with no edge to the failed one is unaffected (see Composing independent branches). What CONTINUE actually buys here is making that independence a stated fact of the definition rather than an accident of which tasks happen to share no edge — a reader (or a validator, eventually) can tell at a glance that generate_ident failing was never supposed to take downmix_audio down with it.

Merging branches that may not both arrive

A task that needs output from two upstream branches raises a question failure_mode doesn't answer: what happens at the join if one branch failed?

The honest default: let it fail. depends_on: [generate_ident, downmix_audio] makes the merge task eligible once both have finished — a failed finish counts. If the merge genuinely cannot do its job with only one input, list both in used[] unconditionally and let it fail naturally when a reference resolves to nothing. That failure is correct: the merge really could not merge.

The tolerant version: gate the optional input with exists().

{
  "id": "combine_to_deliverable",
  "depends_on": ["generate_ident", "downmix_audio"],
  "executor": { "type": "service", "ref": "conform" },
  "inputs": {
    "video_key": "$.tasks.generate_ident.outputs.output_key",
    "audio_key": "exists($.tasks.downmix_audio.outputs.output_key) ? $.tasks.downmix_audio.outputs.output_key : null"
  },
  "used": ["ident_image"],
  "produced": ["combined_deliverable"]
}

Now a failed downmix_audio produces a picture-only deliverable instead of no deliverable at all — a deliberate choice, expressed once, at the one place that needs to make it. Resist adding a second field that says the same thing. There is no on_upstream_failure: require_any in the specification, and none is planned — exists() already expresses "proceed with whatever arrived," and a second mechanism doing the same job would just be two places a reader has to check instead of one. See Reference syntax for the full grammar.

Fan-out: the same question at a scale neither of the above answers

CONTINUE and exists() both assume a small, named set of branches — you can point at generate_ident and downmix_audio because there are two of them. Neither one has an answer for "this same subgraph, several hundred times, over a list only known at run time" — and neither should, because generating one task per item is genuinely the better tool for the tens-of-items case that CONTINUE/exists() were built around.

fan_out (v0.91) is for the case that argument stops holding. The concrete scenario it was built for: several hundred concurrent instances of the same generation subgraph, running live, on stage, while someone presents — where the question is never "list every instance's status," it's "did enough of them come back."

{
  "id": "generate_all_idents",
  "executor": { "type": "agent" },
  "subgraph": { "format": "owg", "ref": "graphs/generate_ident.owg.json" },
  "fan_out": {
    "over": "$.params.presenter_prompts",
    "as": "prompt",
    "max_concurrency": 500,
    "tolerated_failure_percentage": 3
  }
}

Why a percentage, and not just CONTINUE on 500 generated tasks: at five tasks, a human reads the run and judges whether the outcome was good enough. At 500, nobody is reading 500 rows in real time — the run itself has to be able to say "485 of 500 succeeded, that's a pass" the moment the last instance finishes, without a person deciding it afterward. That's a declared, checkable line, not a bigger version of the same tolerance — see why this needed its own field for the fuller version of this argument.

Choosing the threshold is a judgment call worth making deliberately, not defaulting. A few starting points:

• Cosmetic/exploratory output (a batch of concept images where a few misses are expected and easily re-run) — a generous tolerance, 5–10%, is reasonable.
• A live, on-stage demo — set the tolerance to what still reads as a success to an audience watching a summary tile, not to what's merely acceptable on a spreadsheet. 3% of 500 is 15 visibly-missing tiles; decide if that's fine before the show, not during it.
• Anything feeding a delivery gate — tolerate nothing (tolerated_failure_count: 0, or skip fan_out's tolerance and let the default HALT apply to the fan-out task as a whole). A missing deliverable is not a statistic.

Retry still runs first, per instance. If the task also declares retry, an instance only counts against the tolerance once its own retries are exhausted — fan_out and retry compose rather than one superseding the other.

max_concurrency bounds a real external constraint, not a target to hit. State it at the number a vendor API, a GPU pool, or a cost ceiling actually allows — an engine may run fewer instances at once for its own reasons, but must never run more than the ceiling states. See Fan-out concurrency is a ceiling, not a guarantee.

Putting it together

A single production graph typically uses all three mechanisms, at the altitude each one fits:

• failure_mode on individual named tasks, for the handful of branches an author can point at directly.
• exists()-gated inputs on any task that merges two of those branches and should tolerate one being missing.
• fan_out with a stated tolerance on the one task in the graph that is actually hundreds of instances of the same thing.

None of these needs the other two present to be worth using on its own — a graph with one fan-out task and nothing else reaching for CONTINUE is a completely ordinary graph. Reach for each one for the shape of problem it actually solves, not by default.