{"row":{"id":"ADR-019","title":"Retry + timeout ownership: stratified by layer","file":"adr-019-retry-timeout-ownership.md","status":"accepted","date":"2026-04-16","cites":4,"supersedes":[],"superseded_by":[],"layers":["L0.5","L1","L2"],"tags":["retry","timeout","reliability","layer-discipline"],"room":true},"pin":{"release_tag":"v0.107.0","commit":"5d61b7e13f1f6a10c17414f61146648f76bcf2bd"}}

[{"k":"h","depth":1,"text":"ADR-019: Retry + timeout ownership: stratified by layer","id":"adr-019-retry--timeout-ownership-stratified-by-layer"},{"k":"h","depth":2,"text":"Context","id":"context"},{"k":"p","inline":[{"k":"text","text":"The kernel already exposes building blocks for retry decisions but does not\nown a retry policy."}]},{"k":"p","inline":[{"k":"code","text":"ProviderError"},{"k":"text","text":" ("},{"k":"code","text":"crates/nika-kernel-ai/src/provider.rs:386-424"},{"k":"text","text":") carries the\ninformation needed to decide retryability:"}]},{"k":"list","ordered":false,"items":[[{"k":"code","text":"RateLimited { retry_after_ms: Option<u64> }"},{"k":"text","text":"\n("},{"k":"code","text":"provider.rs:407-410"},{"k":"text","text":") — explicit hint from the provider."}],[{"k":"code","text":"Api { status: u16, message: String }"},{"k":"text","text":" — generic API failure with an HTTP\nstatus code."}],[{"k":"code","text":"is_transient(&self) -> bool"},{"k":"text","text":" ("},{"k":"code","text":"provider.rs:428-439"},{"k":"text","text":") — returns "},{"k":"code","text":"true"},{"k":"text","text":" for\n"},{"k":"code","text":"RateLimited"},{"k":"text","text":" and "},{"k":"code","text":"Api { status: 500..=599, .. }"},{"k":"text","text":"."}]]},{"k":"p","inline":[{"k":"code","text":"ShellCommand"},{"k":"text","text":" ("},{"k":"code","text":"crates/nika-kernel-core/src/io/process.rs"},{"k":"text","text":") already has a "},{"k":"code","text":"timeout: Option<Duration>"},{"k":"text","text":" field; "},{"k":"code","text":"ShellError::Timeout { duration_ms: u64 }"},{"k":"text","text":" is the\nmatching error variant."}]},{"k":"p","inline":[{"k":"code","text":"ToolExecError"},{"k":"text","text":" ("},{"k":"code","text":"crates/nika-kernel-runtime/src/tool_executor.rs"},{"k":"text","text":") has "},{"k":"code","text":"NotAvailable"},{"k":"text","text":",\n"},{"k":"code","text":"NotFound"},{"k":"text","text":", "},{"k":"code","text":"Timeout"},{"k":"text","text":", but no "},{"k":"code","text":"is_transient()"},{"k":"text","text":" — tool failures are\npredominantly semantic (wrong args, tool missing). Retrying them masks bugs."}]},{"k":"p","inline":[{"k":"code","text":"AgentLoopConfig"},{"k":"text","text":" ("},{"k":"code","text":"crates/nika-kernel-runtime/src/agent.rs"},{"k":"text","text":") controls iterations and\nbudgets, but has no "},{"k":"code","text":"retry:"},{"k":"text","text":" block — agent loop control and provider retry\nare different concerns."}]},{"k":"p","inline":[{"k":"text","text":"Workflow YAML supports a "},{"k":"code","text":"retry:"},{"k":"text","text":" block at the task level today (legacy).\nEngine-level retry needs to respect it without re-implementing the policy at\nevery layer."}]},{"k":"p","inline":[{"k":"text","text":"NIKA-100 + NIKA-107 (MCP connection errors) already surface to users with\nguidance to retry; a sane default retry helps avoid noisy false negatives."}]},{"k":"h","depth":2,"text":"Decision","id":"decision"},{"k":"p","inline":[{"k":"strong","text":"Retry and timeout ownership is stratified by architectural layer. Each\nlayer owns the policy that matches its failure mode."}]},{"k":"h","depth":3,"text":"Retry layers","id":"retry-layers"},{"k":"list","ordered":true,"items":[[{"k":"strong","text":"L1 -- HTTP transport retry"},{"k":"text","text":" (owned by "},{"k":"code","text":"nika-http"},{"k":"text","text":"):"},{"k":"text","text":"- Connect / DNS / TLS handshake failures.\n- 3 attempts, exponential backoff (250ms / 1s / 4s), 50% jitter.\n- Configurable per-request via `RequestOptions`.\n- Respects `Retry-After` header on 429/503.\n"}],[{"k":"strong","text":"L2 -- Provider transient retry"},{"k":"text","text":" (owned by verb-* crates):"},{"k":"text","text":"- 429 (RateLimited) and 5xx (Api) failures from `ProviderError::is_transient`.\n- Default: 3 attempts, exponential backoff (1s / 4s / 16s), jitter.\n- Respects `RateLimited.retry_after_ms` when present (overrides backoff).\n- Configurable per-task via workflow YAML `retry:` block.\n"}],[{"k":"strong","text":"L2 -- ToolExecutor: no automatic retry."},{"k":"text","text":"- Tool failures are semantic (wrong params, tool missing, semantic error).\n- Retrying masks bugs.\n- User opts in explicitly via workflow `retry:` block on the task."}]]},{"k":"h","depth":3,"text":"Timeout ownership","id":"timeout-ownership"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Boundary · Owner · Default · Source"}],[{"k":"text","text":"HTTP connect · L1 (`nika-http`) · 10 s · `RequestOptions.connect_timeout`"}],[{"k":"text","text":"HTTP read · L1 (`nika-http`) · 120 s · `RequestOptions.read_timeout` (long LLM responses)"}],[{"k":"text","text":"Tool execution · L2 (`nika-process` or tool impl) · per `ShellCommand.timeout` · workflow YAML"}],[{"k":"text","text":"Task · L2 (runtime) · none · workflow YAML `timeout:` field"}],[{"k":"text","text":"Agent loop total · L2 (verb-agent) · none · `AgentLoopConfig.limits.max_duration_secs`"}],[{"k":"text","text":"Whole workflow · none · unlimited · by design"}]]},{"k":"h","depth":3,"text":"Shared types (planned, ships in S1-C)","id":"shared-types-planned-ships-in-s1-c"},{"k":"p","inline":[{"k":"code","text":"RetryConfig"},{"k":"text","text":" and "},{"k":"code","text":"ErrorCategory"},{"k":"text","text":" will live in "},{"k":"code","text":"nika-error"},{"k":"text","text":" (L0) so every\nlayer can speak the same vocabulary without depending on L0.5 kernel:"}]},{"k":"code","lang":"rust","text":"// Sketch (real impl in S1-C / ADR-033)\n#[non_exhaustive]\npub struct RetryConfig {\n    pub max_attempts: u32,\n    pub initial_backoff: Duration,\n    pub max_backoff: Duration,\n    pub jitter: f32,\n}\n\n#[non_exhaustive]\npub enum ErrorCategory {\n    Transient,\n    RateLimited,\n    Auth,\n    NotFound,\n    Permanent,\n}"},{"k":"p","inline":[{"k":"text","text":"The implementation chooses "},{"k":"em","text":"when"},{"k":"text","text":" to apply a "},{"k":"code","text":"RetryConfig"},{"k":"text","text":" based on\n"},{"k":"code","text":"ErrorCategory"},{"k":"text","text":"; the kernel does not pick a policy."}]},{"k":"h","depth":2,"text":"Consequences","id":"consequences"},{"k":"h","depth":3,"text":"Positive","id":"positive"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Each retry layer has a single responsibility — easy to reason about\nfailures."}],[{"k":"text","text":"HTTP retries cannot mask provider-level rate limits (which need\n"},{"k":"code","text":"retry-after"},{"k":"text","text":" respect)."}],[{"k":"text","text":"Tool failures are not silently retried — bugs surface immediately."}],[{"k":"text","text":"One vocabulary ("},{"k":"code","text":"RetryConfig"},{"k":"text","text":" + "},{"k":"code","text":"ErrorCategory"},{"k":"text","text":") shared across crates,\nno per-crate retry types."}]]},{"k":"h","depth":3,"text":"Negative","id":"negative"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Three retry layers means the user must understand which one is firing\nwhen something goes wrong. Mitigated by tagging retry events with\nthe layer name ("},{"k":"code","text":"http"},{"k":"text","text":", "},{"k":"code","text":"provider"},{"k":"text","text":", "},{"k":"code","text":"task"},{"k":"text","text":") in the EventLog."}],[{"k":"text","text":"Cumulative latency under cascading failures (HTTP retried 3× then\nprovider retried 3× = up to 9 round-trips). Accepted because each\nlayer's max-attempts is small and bounded."}],[{"k":"text","text":"Workflow-level retry config has to flow through to the right layer —\nneeds careful documentation in the YAML reference."}]]},{"k":"h","depth":3,"text":"Neutral","id":"neutral"},{"k":"list","ordered":false,"items":[[{"k":"code","text":"ProviderError::is_transient"},{"k":"text","text":" stays as the kernel's only retry\nclassifier. No new traits needed at L0.5."}],[{"k":"code","text":"tower::retry::Retry"},{"k":"text","text":" is not adopted (see Alt D); we may revisit at\nv0.100 when tower's middleware gets seriously evaluated."}]]},{"k":"h","depth":2,"text":"Evidence / Affected code","id":"evidence--affected-code"},{"k":"list","ordered":false,"items":[[{"k":"code","text":"crates/nika-kernel-ai/src/provider.rs:407-410"},{"k":"text","text":" — "},{"k":"code","text":"RateLimited { retry_after_ms }"},{"k":"text","text":"."}],[{"k":"code","text":"crates/nika-kernel-ai/src/provider.rs:428-439"},{"k":"text","text":" — "},{"k":"code","text":"is_transient()"},{"k":"text","text":" impl."}],[{"k":"code","text":"crates/nika-kernel-core/src/io/process.rs"},{"k":"text","text":" — "},{"k":"code","text":"ShellCommand.timeout"},{"k":"text","text":",\n"},{"k":"code","text":"ShellError::Timeout"},{"k":"text","text":"."}],[{"k":"code","text":"crates/nika-kernel-runtime/src/tool_executor.rs"},{"k":"text","text":" — "},{"k":"code","text":"ToolExecError"},{"k":"text","text":" (no\nTimeout-as-retry signal — intentional)."}],[{"k":"code","text":"crates/nika-kernel-runtime/src/agent.rs"},{"k":"text","text":" — "},{"k":"code","text":"AgentLoopConfig"},{"k":"text","text":" (no retry block —\nretry is per-task, not per-loop)."}],[{"k":"text","text":"Workflow YAML schema (legacy) — "},{"k":"code","text":"retry: { max_attempts, delay_ms, backoff }"},{"k":"text","text":"."}],[{"k":"text","text":"ADR-033 (S1-C) — adds "},{"k":"code","text":"RetryConfig"},{"k":"text","text":" + "},{"k":"code","text":"ErrorCategory"},{"k":"text","text":" to "},{"k":"code","text":"nika-error"},{"k":"text","text":" (L0)."}]]},{"k":"h","depth":2,"text":"Alternatives considered","id":"alternatives-considered"},{"k":"h","depth":3,"text":"Alt A -- Retry everything (including tool calls)","id":"alt-a----retry-everything-including-tool-calls"},{"k":"p","inline":[{"k":"text","text":"Engine retries on any error.\nRejected: masks semantic errors. A tool returning \"file not found\" should\nnot be retried — the path is wrong, retrying makes the symptom worse\n(user waits, then sees the same error)."}]},{"k":"h","depth":3,"text":"Alt B -- Single global retry policy","id":"alt-b----single-global-retry-policy"},{"k":"p","inline":[{"k":"text","text":"One "},{"k":"code","text":"RetryConfig"},{"k":"text","text":" for the whole engine.\nRejected: HTTP retries operate on millisecond-scale and benefit from\nsmall backoffs; provider 429 retries operate on second-scale (the\n"},{"k":"code","text":"retry-after"},{"k":"text","text":" is often 5-30s). One config cannot serve both."}]},{"k":"h","depth":3,"text":"Alt C -- No retry in the engine, push to user","id":"alt-c----no-retry-in-the-engine-push-to-user"},{"k":"p","inline":[{"k":"text","text":"Every workflow author writes their own retry block.\nRejected: 429 handling is so common that boilerplate would dominate\nworkflow files. Sane defaults at the engine level reduce friction\nwithout taking away user override."}]},{"k":"h","depth":3,"text":"Alt D -- `tower::retry::Retry` middleware","id":"alt-d----towerretryretry-middleware"},{"k":"p","inline":[{"k":"text","text":"Use the tower ecosystem.\nRejected for v0.81: "},{"k":"code","text":"tower"},{"k":"text","text":" is not in the dep tree before v0.100. The\nMEGA_ULTRATHINK rejections list (Batch B) defers tower adoption until\nthe runtime crate decides whether to standardize on it. We can revisit."}]},{"k":"h","depth":3,"text":"Alt E -- Retry as a kernel trait (`Retryable`)","id":"alt-e----retry-as-a-kernel-trait-retryable"},{"k":"p","inline":[{"k":"text","text":"Make every error type implement "},{"k":"code","text":"is_retryable()"},{"k":"text","text":".\nRejected: "},{"k":"code","text":"is_transient()"},{"k":"text","text":" already exists on "},{"k":"code","text":"ProviderError"},{"k":"text","text":" — promoting\nit to a trait would force every error type (including "},{"k":"code","text":"BlobError"},{"k":"text","text":",\n"},{"k":"code","text":"MemoryError"},{"k":"text","text":") to implement it even when retry doesn't apply."}]},{"k":"h","depth":2,"text":"Related","id":"related"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"ADR-006 — layer stratification. ADR-019 is the retry-policy projection\nof ADR-006."}],[{"k":"text","text":"ADR-007 — forward-compat invariants. "},{"k":"code","text":"ProviderError::RateLimited"},{"k":"text","text":"\ncarries "},{"k":"code","text":"retry_after_ms"},{"k":"text","text":" as a future-proof hint; ADR-019 names the\nconsumer."}],[{"k":"text","text":"ADR-018 — runtime + sync primitives. "},{"k":"code","text":"tokio::time::sleep"},{"k":"text","text":" and\n"},{"k":"code","text":"tokio::time::timeout"},{"k":"text","text":" are the implementation primitives."}],[{"k":"text","text":"ADR-033 (S1-C) — adds "},{"k":"code","text":"RetryConfig"},{"k":"text","text":" + "},{"k":"code","text":"ErrorCategory"},{"k":"text","text":" to "},{"k":"code","text":"nika-error"},{"k":"text","text":"."}]]},{"k":"h","depth":2,"text":"Notes","id":"notes"},{"k":"p","inline":[{"k":"text","text":"Follow-ups:"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"ADR-033 + S1-C will land "},{"k":"code","text":"RetryConfig"},{"k":"text","text":" and "},{"k":"code","text":"ErrorCategory"},{"k":"text","text":" in\n"},{"k":"code","text":"nika-error"},{"k":"text","text":"."}],[{"k":"text","text":"L1 retry implementation lands when "},{"k":"code","text":"nika-http"},{"k":"text","text":" is admitted."}],[{"k":"text","text":"L2 retry implementation lands when verb-* crates are admitted."}],[{"k":"text","text":"Document the task-level "},{"k":"code","text":"retry:"},{"k":"text","text":" block schema in the workflow YAML\nreference (Phase E docs)."}]]}]

ADR-019

Retry + timeout ownership: stratified by layer

accepted · 2026-04-16 · L0.5 L1 L2 · cites 4

ADR-019: Retry + timeout ownership: stratified by layer #

Context #

The kernel already exposes building blocks for retry decisions but does not own a retry policy.

ProviderError (crates/nika-kernel-ai/src/provider.rs:386-424) carries the information needed to decide retryability:

RateLimited { retry_after_ms: Option<u64> } (provider.rs:407-410) — explicit hint from the provider.
Api { status: u16, message: String } — generic API failure with an HTTP status code.
is_transient(&self) -> bool (provider.rs:428-439) — returns true for RateLimited and Api { status: 500..=599, .. }.

ShellCommand (crates/nika-kernel-core/src/io/process.rs) already has a timeout: Option<Duration> field; ShellError::Timeout { duration_ms: u64 } is the matching error variant.

ToolExecError (crates/nika-kernel-runtime/src/tool_executor.rs) has NotAvailable, NotFound, Timeout, but no is_transient() — tool failures are predominantly semantic (wrong args, tool missing). Retrying them masks bugs.

AgentLoopConfig (crates/nika-kernel-runtime/src/agent.rs) controls iterations and budgets, but has no retry: block — agent loop control and provider retry are different concerns.

Workflow YAML supports a retry: block at the task level today (legacy). Engine-level retry needs to respect it without re-implementing the policy at every layer.

NIKA-100 + NIKA-107 (MCP connection errors) already surface to users with guidance to retry; a sane default retry helps avoid noisy false negatives.

Decision #

Retry and timeout ownership is stratified by architectural layer. Each layer owns the policy that matches its failure mode.

Retry layers #

L1 -- HTTP transport retry (owned by nika-http):- Connect / DNS / TLS handshake failures. - 3 attempts, exponential backoff (250ms / 1s / 4s), 50% jitter. - Configurable per-request via `RequestOptions`. - Respects `Retry-After` header on 429/503.
L2 -- Provider transient retry (owned by verb-* crates):- 429 (RateLimited) and 5xx (Api) failures from `ProviderError::is_transient`. - Default: 3 attempts, exponential backoff (1s / 4s / 16s), jitter. - Respects `RateLimited.retry_after_ms` when present (overrides backoff). - Configurable per-task via workflow YAML `retry:` block.
L2 -- ToolExecutor: no automatic retry.- Tool failures are semantic (wrong params, tool missing, semantic error). - Retrying masks bugs. - User opts in explicitly via workflow `retry:` block on the task.

Timeout ownership #

Boundary · Owner · Default · Source
HTTP connect · L1 (`nika-http`) · 10 s · `RequestOptions.connect_timeout`
HTTP read · L1 (`nika-http`) · 120 s · `RequestOptions.read_timeout` (long LLM responses)
Tool execution · L2 (`nika-process` or tool impl) · per `ShellCommand.timeout` · workflow YAML
Task · L2 (runtime) · none · workflow YAML `timeout:` field
Agent loop total · L2 (verb-agent) · none · `AgentLoopConfig.limits.max_duration_secs`
Whole workflow · none · unlimited · by design

Shared types (planned, ships in S1-C) #

RetryConfig and ErrorCategory will live in nika-error (L0) so every layer can speak the same vocabulary without depending on L0.5 kernel:

rust

1// Sketch (real impl in S1-C / ADR-033)2#[non_exhaustive]3pub struct RetryConfig {4    pub max_attempts: u32,5    pub initial_backoff: Duration,6    pub max_backoff: Duration,7    pub jitter: f32,8}910#[non_exhaustive]11pub enum ErrorCategory {12    Transient,13    RateLimited,14    Auth,15    NotFound,16    Permanent,17}

The implementation chooses when to apply a RetryConfig based on ErrorCategory; the kernel does not pick a policy.

Consequences #

Positive #

Each retry layer has a single responsibility — easy to reason about failures.
HTTP retries cannot mask provider-level rate limits (which need retry-after respect).
Tool failures are not silently retried — bugs surface immediately.
One vocabulary (RetryConfig + ErrorCategory) shared across crates, no per-crate retry types.

Negative #

Three retry layers means the user must understand which one is firing when something goes wrong. Mitigated by tagging retry events with the layer name (http, provider, task) in the EventLog.
Cumulative latency under cascading failures (HTTP retried 3× then provider retried 3× = up to 9 round-trips). Accepted because each layer's max-attempts is small and bounded.
Workflow-level retry config has to flow through to the right layer — needs careful documentation in the YAML reference.

Neutral #

ProviderError::is_transient stays as the kernel's only retry classifier. No new traits needed at L0.5.
tower::retry::Retry is not adopted (see Alt D); we may revisit at v0.100 when tower's middleware gets seriously evaluated.

Evidence / Affected code #

crates/nika-kernel-ai/src/provider.rs:407-410 — RateLimited { retry_after_ms }.
crates/nika-kernel-ai/src/provider.rs:428-439 — is_transient() impl.
crates/nika-kernel-core/src/io/process.rs — ShellCommand.timeout, ShellError::Timeout.
crates/nika-kernel-runtime/src/tool_executor.rs — ToolExecError (no Timeout-as-retry signal — intentional).
crates/nika-kernel-runtime/src/agent.rs — AgentLoopConfig (no retry block — retry is per-task, not per-loop).
Workflow YAML schema (legacy) — retry: { max_attempts, delay_ms, backoff }.
ADR-033 (S1-C) — adds RetryConfig + ErrorCategory to nika-error (L0).

Alternatives considered #

Alt A -- Retry everything (including tool calls) #

Engine retries on any error. Rejected: masks semantic errors. A tool returning "file not found" should not be retried — the path is wrong, retrying makes the symptom worse (user waits, then sees the same error).

Alt B -- Single global retry policy #

One RetryConfig for the whole engine. Rejected: HTTP retries operate on millisecond-scale and benefit from small backoffs; provider 429 retries operate on second-scale (the retry-after is often 5-30s). One config cannot serve both.

Alt C -- No retry in the engine, push to user #

Every workflow author writes their own retry block. Rejected: 429 handling is so common that boilerplate would dominate workflow files. Sane defaults at the engine level reduce friction without taking away user override.

Alt D -- `tower::retry::Retry` middleware #

Use the tower ecosystem. Rejected for v0.81: tower is not in the dep tree before v0.100. The MEGA_ULTRATHINK rejections list (Batch B) defers tower adoption until the runtime crate decides whether to standardize on it. We can revisit.

Alt E -- Retry as a kernel trait (`Retryable`) #

Make every error type implement is_retryable(). Rejected: is_transient() already exists on ProviderError — promoting it to a trait would force every error type (including BlobError, MemoryError) to implement it even when retry doesn't apply.

ADR-006 — layer stratification. ADR-019 is the retry-policy projection of ADR-006.
ADR-007 — forward-compat invariants. ProviderError::RateLimited carries retry_after_ms as a future-proof hint; ADR-019 names the consumer.
ADR-018 — runtime + sync primitives. tokio::time::sleep and tokio::time::timeout are the implementation primitives.
ADR-033 (S1-C) — adds RetryConfig + ErrorCategory to nika-error.

Notes #

Follow-ups:

ADR-033 + S1-C will land RetryConfig and ErrorCategory in nika-error.
L1 retry implementation lands when nika-http is admitted.
L2 retry implementation lands when verb-* crates are admitted.
Document the task-level retry: block schema in the workflow YAML reference (Phase E docs).

read at v0.107.0 · the decision record ships with the engine

Every decision, with its status This one in the repo