The Console Hang Fix: Every Question Gets a Answer or a Timeout, Never an Infinite Wait
| Who | ANZ Cloud Operations Teams · CIOs · Cloud Platform Owners |
| What | The Command Center console now guarantees bounded termination — every question either completes successfully or fails with a clear error message within 10 minutes |
| When | August 2026 production deployment |
| Where | Console gateway (AI cost/network assistant UI) |
| Why | Users faced indefinite hangs with no error message and no recovery path — eroding trust in the tool. Bounded termination makes failures visible and actionable. |
| How | A 10-minute idle deadline in the adapter layer + unconditional drain-task cancellation + explicit error staging at the gateway ensures one terminal event (success or named error) per question, every time. |
Why: An infinite spinner with no error message is worse than a timeout error — the user cannot tell if the system is working or dead.
What if missing: Users wait indefinitely, lose trust, refresh the page, and create duplicate support tickets. Dashboard uptime claims become meaningless if the tool hangs silently.
Business value: Bounded termination means ops engineers get immediate feedback on failed requests, enabling faster diagnosis and resolution of underlying issues.
Critical thinking: The root cause of the stalls remains under investigation — this fix makes the symptom visible so diagnosis is possible, rather than waiting for a complete root-cause fix before shipping any improvement.
The Problem
Users submitted questions to the console about cloud costs or network topology. Unpredictably, the page would display a spinner indefinitely with zero error message and no way to know if the console was still working or frozen. Users waited minutes, gave up, and refreshed. No diagnostics existed.
Impact: Frustration, lost trust in the tool, wasted support time.
Measurement on 2026-08-02 showed three consecutive console runs across two different products:
docker logs console-gateway-1 2>&1 | grep -c run_started → 3 (three runs started)
docker logs console-gateway-1 2>&1 | grep -cE 'run_finished|run_error' → 0 (zero terminated)
Three runs started. Zero terminated. The browser sat on a spinner indefinitely.
The Fix
Every console question now runs under a 10-minute idle deadline. When the question completes, the user sees:
- Success path: the answer (as before)
- Failure path: a clear error message like
"Request timed out — please try again"or"Unable to connect to AWS — check your profile"
The guarantee: The console cannot hand the user an infinite spinner anymore.
When a timeout occurs, the system now logs exactly where it got stuck — which tool ran, which step, what the error was. Before this fix, those logs were silent or invisible.
What Stays Open
Root cause: The exact reason why the hang happens is still being investigated. We know the stalls occur after the system checks tool permissions but before the tool finishes running. We don't know why — there's no obvious performance bottleneck or infinite loop.
Does this block anything? No. The 10-minute deadline + clear error message solve the user-facing problem. Finding the root cause is valuable for robustness and optimization, but it does not prevent the console from shipping this fix.
How It Works (For the Ops Team)
The termination guarantee rests on three structural properties:
- Idle-time bound at the adapter queue — the only point observing forward progress from both SDK and approval gate
- Worker acquisition gets its own separate bound — connect latencies vary (4.0s to 26.7s measured); a single global bound would fail on slow connections
- Cleanup is itself bounded — drain-task is cancelled unconditionally (a no-op on healthy paths) and awaited under a separate budget, preventing re-hangs
The termination path works like this:
For the full design rationale and test evidence, see CC-ADR-062: Bounded Termination Guarantee for Console Agent Runs (in progress).
Three Decisions Needed
| Decision | What | Why |
|---|---|---|
| Approve the hang fix | Merge the 10-minute deadline + error-message fix to production | The hang is the only Severity-1 usability defect; it is solved and ready to ship. |
| Prioritize the $0 cost issue | Add the "false-zero spend" bug to the backlog and flag it for the next sprint | Users should never see confident wrong answers. This is a separate, lower-severity issue. |
| Add to audit trail | Formally log both the hang fix (implementation complete, ready to deploy) and the $0 cost bug (separate fix, next sprint) | Tracking matters for roadmap visibility and compliance audit trail. |
Ask your operations team these two questions THIS WEEK:
- Reliability visibility: Can you commit to a 10-minute maximum wait time on every console question going forward?
- Error clarity: Does every failed request now return a named error (e.g., "AWS profile error", "timeout", "permission denied") instead of an infinite spinner?
If yes to both, the console is ready for production deployment.
See CC-ADR-062 for the full architectural decision (in progress).