V3 provider errors, timeouts, and retries implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking.

Goal: Complete R008, R009 and R167 in issue #231.

Architecture: Provider-owned errors normalize transport failures. A context-scoped deadline, cancellation event and four-attempt budget are shared by adapters, runtime workers and fallback. Successful response decoding stays outside retry.

Tech Stack: Python 3.12/3.13, standard-library contextvars/threading, existing provider SDKs, unittest.

Spec: The user-approved issue #231 plan and GitHub source requirements.

Global constraints

• One issue and one PR; branch Evan/v3-231-provider-errors starts at current main. No merge.
• No new dependency or paid calls. Preserve successful response contracts, model mapping and usage attribution.
• Default deadlines: 90 seconds foreground, 180 child; finite override clamped 1–600 seconds.
• Four transport attempts total, including fallback; disable SDK retries and pass remaining timeout.
• Retry transport, transport timeout, rate limit and server failures only. Authentication may advance to another provider without repeating credentials. Other failures and overall expiry/cancellation are terminal.
• Try fallback candidates in order before repeating transient failures. No replay after a streamed chunk or received response.
• Legacy text boundary retains error rendering and one context-compaction recovery. Tool retry and full tool/process cancellation are out of scope.

Review focus

• SDK retries must not multiply the shared attempt limit.
• Cancelled/timed-out workers must not start another request or publish late output.
• Structured context errors must still trigger exactly one existing compaction attempt.
• Errors and chained fallback summaries must not display API keys or request bodies.
• Partial streams and successful responses must never be replayed; successful usage remains charged once.

Task 1: Normalize provider failures

Files: providers/errors.py, providers/init.py, tests/test_provider_errors.py; adapter transport boundaries. Interfaces: ProviderErrorKind; ProviderError(kind, provider, model, status_code, provider_code, retry_after, attempts); normalize_provider_error(exc, provider, model).

• [x] Add regressions for status/code classification, safe diagnostics, typed errors and terminal contract errors; watch them fail.
• [x] Implement provider-owned normalization using optional SDK exception types and structured status/code fields, with narrow context-overflow matching for legacy exceptions.
• [x] Preserve original causes; expose only bounded safe diagnostic fields. Run focused tests.

Task 2: Bound retries and fallback

Files: providers/retry.py, providers/fallback.py, adapter transports; tests/test_provider_errors.py, tests/test_providers.py. Interfaces: provider_request_scope(timeout=90, cancelled=None); remaining_timeout(); shared request state; existing _retry_with_backoff gains provider/model identity.

• [x] Add failing tests for message-only false positives, four-attempt shared limit, candidate order, authentication exclusion and partial-stream replay.
• [x] Share monotonic deadline/cancellation/attempt state through contextvars. Preserve standalone adapter calls by opening a default request scope when none exists.
• [x] Disable SDK retries and set finite transport timeouts for every adapter. Use a one-attempt adapter mode inside fallback, retaining the same global counter.
• [x] Back off from one second with bounded jitter; respect bounded Retry-After and interrupt waits. Stop terminal failures and never retry received-response decoding.
• [x] Normalize iteration failures; close streams in finally. Run focused tests on both supported Python versions.

Task 3: Integrate runtime lifecycle and compatibility

Files: providers/calls.py, agent/context.py; runtime and delegation tests. Interfaces: Existing runtime methods/signatures preserved; structured boundary raises ProviderError; explicit text boundary renders legacy errors.

• [x] Add failing tests for cancellation/deadline during requests and backoff, stalled streams, late output, nonfinite timeout settings and compaction compatibility.
• [x] Enter shared scope before copying context into workers. Poll cancellation/deadline, set local cancellation on every foreground exit, reuse child coordinator events without cancelling completed child runs.
• [x] Keep usage ledger and context bookkeeping, ProviderUnavailableError and ContextOverflowError compatibility.
• [x] Use local HTTP servers/real installed SDKs to prove timeout settings, disabled hidden retries and request counts; verify no tool execution.

Task 4: Verify and deliver

• [x] Update provider documentation and index; include worker-termination limitations and retry/fallback compatibility changes.
• [x] Run focused 3.12/3.13 tests; full make test/check/lint/docs-check and offline agent/subagent acceptance. Report blocked checks distinctly.
• [x] Obtain fresh independent whole-branch review and reproduce/fix material findings with regressions.
• [ ] GPG-sign task-scoped commit(s), verify signatures locally, push issue branch, create and attach one PR.
• [ ] Verify GitHub signatures and wait for CI on the final commit. Leave PR unmerged and stop.

Verification record

• RED→GREEN: message-only retry, typed status, fallback ordering/budget, partial-stream replay, HTTPX subtype classification, runtime/nested fallback entry, received metadata-only stream failure and malformed error metadata.
• Real SDK local HTTP tests cover all nine adapter families and streaming initialization; no paid calls. Google Gen AI 1.0.0 compatibility was additionally exercised through an extracted public wheel.
• Independent whole-branch review found three material defects; each was reproduced, fixed and independently rechecked. No remaining material implementation findings were reported.
• Final command outcomes and signed commit/CI evidence are recorded in the pull request; live-provider interoperability is not claimed by these synthetic checks.

Local final verification: 597 full Python 3.12 tests; 96 focused Python 3.13 tests; make check/lint/docs-check and offline agent/subagent acceptance PASS. No paid calls.

Final follow-up regressions cover explicit retry ownership on dynamic callables, all-candidate Retry-After, and cancellation synchronized during actual backoff. Independent review and all local checks pass.

Automated review follow-up

GitHub findings were reproduced before fixes. Runtime and standalone adapters now reuse provider-owned bounded workers, including continuous-trickle responses. Stream request context remains in its producer worker; completed borrowed scopes stay usable while aborted consumption cancels. Azure identity and truncated Requests responses classify correctly, mid-stream attempt counts persist, unread error JSON does not replace the original failure, exhausted budgets remain typed, and impossible Retry-After waits fail immediately with the classified error. Legacy context sentinels also become typed at the structured boundary. Fresh whole-branch independent review reports no material findings.