Observability & Troubleshooting

Complex Agent tasks involve the frontend, the control plane, the model gateway, the Orchestrator, MCP, the Sandbox, Redis, the database, and external services. Looking only at the final error message is usually not enough to locate the problem.

Key Identifiers

When diagnosing, keep and correlate the following identifiers:

• Session ID: a single user session;
• Swarm ID: a single multi-Agent execution;
• Task ID: a specific task or sub-task;
• Agent ID: the Agent executing that task;
• Model Request ID: the request identifier returned by the model provider;
• Trace ID: the cross-service call-chain identifier;
• Workspace / Sandbox ID: the actual execution environment.

The UI, logs, and ticketing system should use the same set of identifiers where possible, to avoid having to guess correspondences from time and text alone.

Layered Investigation

1. Frontend & Connection Layer

Check the connection address, authentication information, whether the WebSocket or streaming connection dropped, and whether the UI state is consistent with the backend.

2. Orchestrator & Queue

Check whether the task was enqueued, whether a worker picked it up, whether its state keeps updating, and whether retry / reopen triggers as expected.

3. Model Calls

Save the masked request parameters, response fields, HTTP status, error code, token accounting, and latency. Focus on distinguishing: authentication failure, throttling, context overflow, timeout, empty content, invalid tool call, and JSON parsing failure.

4. MCP & Tools

Check protocol negotiation, authentication headers, tool schema, request timeout, response size, and transport method. An MCP Server being connectable doesn't mean every tool can be called correctly.

5. Workspace / Sandbox

Check whether the repository is mounted, whether the current directory is correct, whether dependencies exist, whether there is enough disk space, whether the network policy allows access, and whether the workspace was destroyed prematurely.

6. Result Aggregation

Check whether the final report is generated from the actual workspace, test logs, and Agent results, to avoid an aggregator that only summarizes natural language while dropping real failures.

Common Failure Categories

Recommended Error-Code Format

HC-MODEL-1001  Model authentication failed
HC-MODEL-1203  Structured-output parsing failed
HC-SWARM-2104  Agent execution timed out
HC-MCP-3102    MCP Server unreachable
HC-VM-4101     Sandbox creation failed
HC-NET-5103    Subnet or NSG misconfiguration

An error code should include the responsible domain, a stable number, a user-readable explanation, a suggested action, and internal diagnostic fields.

Diagnostic Bundle

When reporting a problem, export a masked diagnostic bundle containing:

• The Heicode version and runtime environment;
• The Session, Swarm, Task, and Agent identifiers;
• The task time range;
• A timeline of state changes;
• A summary of model and tool errors;
• Key logs;
• The list of changed files;
• The test commands and exit codes;
• The basic status of the network and Sandbox.

A diagnostic bundle should not include API keys, cookies, Authorization headers, SSH private keys, full environment variables, or user privacy data by default.

State Inconsistency

When the UI, Redis, and the database show different states, determine which system is the source of truth for the final state, and record the timestamp of the most recent state update. Don't "fix" the UI state by manually editing Redis alone — doing so may break task recovery and the audit trail.