Agent Swarm Capability Evaluation Standard

Note: the evaluation framework below is an aspirational framework for Heicode's Swarm capability, used to guide product and engineering direction — the current system has not yet implemented automated scoring. The ACO/pheromone mathematical model in this document, capability-gain formulas such as G_E = Q_swarm − Q_base, the weighted reward function, and the L0-L5 maturity levels with their numeric thresholds have not yet been implemented in code as an automated scoring system — they are a planned target standard, and do not mean the current system will output these scores or levels. What genuinely exists today is: the agent_swarm event callback system (see Section 12), and the lead-plus-worker (up to 8, 3 by default) Swarm execution architecture.

1. Purpose of the standard

This standard is used to evaluate the Agent Swarm system's overall capability across the software development lifecycle — specifically, whether multiple Agents can form a stable, controllable, measurable software delivery capability through role division, task-graph collaboration, status feedback, verification loops, failure recovery, security governance, and enterprise observability.

This standard isn't concerned with "how strong a single model is," nor with "whether a task happened to get completed eventually" — it's concerned with:

1. Whether the Swarm truly delivers a comprehensive effect beyond a single Agent.
2. Whether the Swarm can complete complex software engineering tasks at a controllable cost.
3. Whether the Swarm has stable collaboration, verification, and recovery capability.
4. Whether the Swarm meets the permission, audit, and security governance requirements of enterprise scenarios.
5. Whether the Swarm process is observable, replayable, and measurable.

This standard can be used for:

• Heicode's internal technical evaluation.
• Comparing the three generations of Agent modes: Chain, Sub, and Swarm.
• Comparing the effects of different model combinations.
• Enterprise customer acceptance.
• Product whitepapers and academic paper metric systems.
• Building Heicode's own benchmark going forward.

2. Background of the standard

2.1 Assessment of the current state

Academia and industry already have related research or systems such as AgentBench, AgentBoard, GAIA, MultiAgentBench, MAESTRO, AgentsNet, COLLAB, ChatDev, MetaGPT, AgileCoder, and HyperAgent.

These works respectively cover:

• Single-Agent tool-use capability.
• Multi-turn interaction capability.
• Multi-Agent collaboration and competition.
• Multi-Agent trace and observability.
• Multi-Agent software development workflows.
• Metrics such as task completion rate, communication efficiency, execution trace, and resource consumption.

However, there is currently no widely accepted industry standard specifically for LLM Agent Swarm software development systems.

2.2 Heicode's opportunity for standardization

Heicode's distinguishing feature is placing Agents within the full software lifecycle, rather than only having Agents complete isolated tasks. Heicode simultaneously has:

• The client-side core experience.
• The Manager console.
• The Agent execution platform.
• The CodeGW model and usage foundation.
• The secret vault.
• The Resource Grant permission system.
• Approval, audit, logging, and an enterprise command center.

Because of this, Heicode's Agent Swarm evaluation standard can't just look at task outcomes — it must also evaluate:

• Software engineering quality.
• Multi-Agent collaboration efficiency.
• Cost and token consumption.
• Security and permission boundaries.
• Audit and replay capability.
• Enterprise engineering-management visibility.

3. Scope of application

This standard applies to the following types of Agent Swarm tasks:

This standard is not directly used to evaluate:

• Single-turn chat quality.
• Single-function code generation capability.
• Single-model reasoning capability.
• Tool-free, stateless, task-graph-free simple conversational systems.

4. Core definitions

4.1 Agent

An Agent is an AI development role within the Heicode system. Each Agent has a clearly defined role, task goal, available tools, resource permissions, model configuration, budget constraints, and audit records.

Typical roles include, from the six-role catalog: Product Agent, Architect Agent, Frontend Agent, Backend Agent, Reviewer Agent, Ops Agent; the Waterfall template additionally has a separate testing role (qa-tester).

4.2 Agent Swarm

An Agent Swarm is a collaborative system made up of multiple Agents. Through task graphs, role division, dependency scheduling, status reporting, logging, verification, approval, and resource permission control, it jointly completes a complex software engineering task.

An Agent Swarm is not simple concurrency of multiple Agents — it's a system that encompasses orchestration, state, communication, verification, and governance.

4.3 Capability gain

Agent capability gain refers to:

Under the same task, the same resource boundary, or a controllable cost condition, multiple Agents — through role division, task-graph collaboration, status feedback, verification loops, and failure recovery — produce task completion capability, quality stability, delivery speed, or governance controllability that exceeds the baseline of a single Agent or a single strong model.

Emergent capability is a special form of capability gain, emphasizing system-level behavior exhibited by the Swarm as a whole that no individual Agent possesses. In other words, emergence is a key component of the Agent Swarm capability standard, but not the entirety of the standard.

4.4 Swarm capability gain

Swarm capability gain is the Swarm system's overall capability improvement relative to a baseline system.

The baseline can be:

• A single Agent.
• A Chain of Agents.
• Sub mode.
• A single strong-model Agent.
• A manually defined traditional process.

Formal expression:

G_E = Q_swarm − Q_base

Where:

• G_E: capability gain, including emergent gain.
• Q_{\mathrm{swarm}}: the Swarm system's overall performance on the same task set.
• Q_{\mathrm{base}}: the baseline system's overall performance on the same task set.

After accounting for cost:

G_{E,c} = (Q_swarm / C_swarm) − (Q_base / C_base)

Where:

• G_{E,c}: cost-normalized capability gain.
• C_{\mathrm{swarm}}: Swarm system cost, which can be a weighted combination of tokens, time, infrastructure, and human intervention.
• C_{\mathrm{base}}: baseline system cost.

5. The three-generation Agent evolution model

Agent collaboration patterns can be divided into three generations.

The 1st-generation Chain serves only as the weakest conceptual baseline in evaluation — Heicode's product itself does not offer Chain mode. This standard primarily evaluates the 3rd-generation Swarm mode, but also requires forming comparable baselines against the 1st and 2nd generations.

6. Theoretical model

6.1 Theoretical origin: Markov processes and ACO

Swarm behavior is often modeled as a Markov process or an approximate Markov process. The basic idea: when the system is in a given state, an Agent selects its next action based on the current state, the set of candidate actions, historical feedback, and immediate heuristics, thereby forming a state transition.

Ant Colony Optimization (ACO) is a classic swarm-intelligence model. In ACO, an ant chooses the next node j from node i, and the path-selection probability is jointly determined by the pheromone concentration on the path and heuristic information.

It should be emphasized: the Agent formula in this standard is not a newly invented mathematical formula — it takes the ACO path-selection probability as its prototype, mapping "physical path selection" onto "action selection under a software engineering state."

6.2 The original ACO path-selection probability formula

Taking Ant Colony Optimization (ACO) as an example, the path-selection probability formula is:

                 [τ_ij]^α · [η_ij]^β
P_ij^k  =  ─────────────────────────────────
            Σ ( l ∈ N_i^k )  [τ_il]^α · [η_il]^β

Where:

• τ_ij: the pheromone concentration on edge (i,j).
• η_ij: heuristic information (e.g., the inverse of distance).
• α, β: weighting parameters.
• N_i^k: the set of candidate neighbors available to Agent k at node i.

This formula expresses a probabilistic transition process: an Agent doesn't deterministically choose a given path — within the candidate set, it decides the probability of the next choice jointly from historical experience and immediate heuristics.

6.3 Mapping ACO symbols onto Agents

In an Agent Swarm, an Agent no longer moves along edges of a physical graph — it advances tasks between software development states. So this standard maps the core elements of ACO as follows:

6.4 The Agent action-selection probability formula

Based on the mapping above, migrating ACO's path-selection probability formula to the Agent software engineering scenario yields the Agent action-selection probability:

                        [τ(s,a_j,r_k)]^α · [η(s,a_j,r_k)]^β
P_(s,a_j)^(r_k)  =  ────────────────────────────────────────────────────
                     Σ ( a_l ∈ A(s,r_k) )  [τ(s,a_l,r_k)]^α · [η(s,a_l,r_k)]^β

Where:

• P_{s,a_j}^{r_k}: the probability that role r_k selects action a_j in software development state s.
• τ(s,a_j,r_k): the historical effectiveness of role r_k executing action a_j in state s.
• η(s,a_j,r_k): the immediate payoff of action a_j in the current context.
• α, β: weighting parameters for historical experience versus current heuristics.
• A(s,r_k): the set of legal actions for role r_k in state s.
• a_l: a candidate action within the legal action set.

This formula is not an independently original formula — it's a semantic evolution of the ACO formula applied to the Agent setting. It retains ACO's core structure: the numerator represents the attractiveness of a given candidate action, and the denominator represents the sum of attractiveness across all candidate actions, used to normalize into a probability distribution.

6.5 Pheromone definition

An Agent's pheromone is not simply a success count — it's a multi-dimensional historical signal.

τ(s,a,r) = f( success, quality, cost, time, risk, rollback, acceptance )

Can be composed of the following factors:

6.6 Heuristic-information definition

Current heuristic information represents whether a given action is worth executing in the current context.

η(s,a,r) = g( match, urgency, dependency, resource, risk, budget, confidence )

Can be composed of the following factors:

6.7 Reward function

An Agent Swarm's reward function should avoid rewarding "completion" alone. It's recommended to define it as:

R =  w1·S_task + w2·Q_quality + w3·V_speed + w4·E_cost
   + w5·R_robust + w6·G_governance
   − w7·P_risk − w8·P_rework

Where:

• TaskSuccess: whether the task was completed.
• Quality: quality as measured by tests, review, bugs, and user acceptance.
• Speed: speed of completion.
• CostEfficiency: output per unit cost.
• Robustness: failure-recovery capability.
• Governance: whether approval, permissions, and audit are compliant.
• RiskPenalty: risk penalty.
• ReworkPenalty: rework penalty.

7. Evaluation dimensions

This standard uses seven top-level dimensions:

Total score:

S_swarm =  0.25·S_completion + 0.20·S_gain + 0.15·S_collaboration
         + 0.10·S_communication + 0.10·S_cost
         + 0.10·S_robustness + 0.10·S_governance

8. Top-level metrics and sub-metrics

8.1 Task completion capability

8.2 Capability gain

8.3 Collaboration efficiency

8.4 Communication quality

8.5 Cost efficiency

8.6 Robustness

8.7 Security governance and auditability

9. Scoring method

9.1 Individual scoring

Each sub-metric is scored from 0 to 5:

9.2 Top-level dimension scoring

A top-level dimension's score is the average of its sub-metrics, normalized to 100 points.

DimensionScore = Average(SubMetricScores) / 5 · 100

9.3 Total score

FinalScore =
0.25 · Completion
+ 0.20 · Gain
+ 0.15 · Collaboration
+ 0.10 · Communication
+ 0.10 · CostEfficiency
+ 0.10 · Robustness
+ 0.10 · Governance

10. Swarm capability levels

This standard defines six Agent Swarm capability levels. Emergent capability is an important factor in level determination, but the level itself evaluates the Swarm's overall capability.

10.1 Level determination conditions

10.2 Mandatory downgrade rules

Even with a high total score, the following events must trigger a downgrade:

11. Experimental design

11.1 Control groups

Each task should set up at least the following controls:

11.2 Task set

It's recommended to build an internal Heicode task set:

11.3 Experimental constraints

Every evaluation run must record:

• Task description.
• Code repository version.
• Model version.
• Agent count.
• Role configuration.
• Tool permissions.
• Resource authorization.
• Token budget.
• Time budget.
• Whether human intervention is allowed.
• Whether high-risk approval is allowed.

11.4 Result recording

Every evaluation run must output:

• Success / failure.
• Code diff.
• Test results.
• Build results.
• Deployment results.
• Token consumption.
• Total elapsed time.
• Agent contribution.
• Failure reason.
• Approval records.
• Audit events.
• User acceptance result.

12. Data collection specification

12.1 Required events

The following event names match the current agent_swarm callback system:

12.2 Required fields

Every event must include at least:

{
  "event_id": "evt_xxx",
  "event_type": "task.completed",
  "timestamp": 0,
  "user_id": "user_xxx",
  "project_id": "proj_xxx",
  "deployment_id": "dep_xxx",
  "task_id": "task_xxx",
  "agent_id": "agent_xxx",
  "agent_role": "backend",
  "resource_refs": [],
  "approval_id": "appr_xxx",
  "usage": {
    "input_tokens": 0,
    "output_tokens": 0,
    "cost": 0
  },
  "result": {
    "status": "success",
    "artifact_refs": []
  }
}

12.3 Trace requirements

Every Swarm run must form a complete trace:

User input
-> Task-graph generation
-> Role assignment
-> Resource authorization
-> Agent execution
-> Tool calls
-> Test verification
-> Approval
-> Deliverables
-> Audit result

Tasks without a trace must not be included in high-level evaluation.

13. Enterprise management metrics

For team development and enterprise scenarios, the Agent Swarm must also provide R&D management metrics.

ETA can initially use an explainable formula:

T_ETA = ( W_remaining / V_effective ) + T_blocker + T_rework + T_approval

14. Security governance requirements

14.1 Least privilege

Each Agent can only access resources needed for the current task.

Must satisfy:

• Has a Resource Binding.
• Has a Resource Grant.
• Has allowed actions.
• Has constraints.
• Has a TTL or revocation mechanism.

14.2 High-risk approval

The following operations must be approved:

• Production deployment.
• Database write or migration.
• Creating, deleting, or scaling cloud resources.
• Access to production secrets.
• Large token or budget consumption.

Approval must record:

• Approver.
• Approval time.
• Risk level.
• Resource scope.
• TTL.
• Corresponding task.
• Corresponding deployment.

14.3 Secret security

Prohibited:

• Plaintext secrets entering Git.
• Plaintext secrets entering Markdown.
• Plaintext secrets entering ordinary logs.
• Plaintext secrets entering frontend responses.
• Long-lived secrets entering a sub-agent's long-lived state.

Allowed:

• secret_ref entering the manifest.
• Short-lived credential leases entering the runtime.
• Deriving short-lived credentials by TTL once approval passes.

15. Standard output format

Every evaluation should output a standard report.

Agent Swarm Evaluation Report

1. Basic information
   - Task ID
   - Project ID
   - Code version
   - Model version
   - Agent configuration

2. Task result
   - Whether completed
   - Test results
   - Build results
   - Deployment results

3. Scores
   - Task completion capability
   - Capability gain
   - Collaboration efficiency
   - Communication quality
   - Cost efficiency
   - Robustness
   - Security governance and auditability
   - Total score

4. Swarm capability level
   - L0-L5
   - Whether a mandatory downgrade was triggered

5. Cost
   - Tokens
   - Time
   - Infrastructure resources
   - Number of human interventions

6. Audit
   - Resource access
   - Approval records
   - Credential lease
   - Artifact provenance

7. Conclusion
   - Whether it outperforms the baseline
   - Main benefits
   - Main risks
   - Next optimization suggestions

16. Standard application process

16.1 Evaluation process

Select the task set
-> Set the baseline
-> Configure the Agent Swarm
-> Execute the task
-> Collect events and trace
-> Run tests and verification
-> Compute metrics
-> Determine the Swarm capability level
-> Generate the evaluation report

Appendix: Simplified formula summary

Action-selection probability

                        [τ(s,a_j,r_k)]^α · [η(s,a_j,r_k)]^β
P_(s,a_j)^(r_k)  =  ────────────────────────────────────────────────────
                     Σ ( a_l ∈ A(s,r_k) )  [τ(s,a_l,r_k)]^α · [η(s,a_l,r_k)]^β

Capability gain

G_E = Q_swarm − Q_base

Cost-normalized capability gain

G_{E,c} = (Q_swarm / C_swarm) − (Q_base / C_base)

Total score

S_swarm =  0.25·S_completion + 0.20·S_gain + 0.15·S_collaboration
         + 0.10·S_communication + 0.10·S_cost
         + 0.10·S_robustness + 0.10·S_governance

ETA

T_ETA = ( W_remaining / V_effective ) + T_blocker + T_rework + T_approval