Automatisiertes Benchmarking als Qualitaetsmotor fuer LINE AI Business Card King

Posted on Oct 23 • Originally published at evanlin.com on Oct 23

In the first four articles of the "Agent Platform in Action" series, we have step-by-step advanced the LINE AI Business Card Master ( line-card-assistant ) from a standalone prototype to a cloud-native, dual-layer decoupled Google Cloud Agent Platform architecture:

With the architecture and features in place and ready to officially deploy the Agent to the cloud production environment (Phase 5), every AI engineering team faces a core soul-searching question:

"When we modify a line of Prompt, adjust a Graph Node's routing condition, or upgrade the underlying model from gemini-2.5-flash to gemini-3.8-flash, how do we ensure that business card OCR fields aren't missed, duplicate card promotion detection doesn't fail, HITL modification loops don't get stuck, and hybrid search doesn't degrade?"

In today's fifth episode of the series, we are going to implement Phase 4 ( Issue #7, corresponding to PR #13 ): using agents-cli eval and ADK 2.0 Go trajectory testing to build a quantifiable, comparable, and CI/CD-automatable AI Agent Quality Flywheel for the LINE AI Business Card Master!

In traditional software engineering, we are used to writing deterministic unit tests like if got!= want. However, in an AI Agent system driven by ADK 2.0 Graph Workflow + Multimodal LLM, what we need to verify is far more than just the return value of a single function; it is the entire "Execution Trajectory and State Evolution (State Delta)":

To systematically solve this problem, Google Cloud agents-cli proposes a five-step Quality Flywheel workflow:

flowchart LR A [ "1. Generate<br/>(agents-cli eval generate)<br/>Start ADK server and collect real trajectories" ] -- > B [ "2. Grade<br/>(agents-cli eval grade)<br/>Execute Trajectory and quality metric scoring" ] B -- > C [ "3.

Analyze<br/>Review scores for each metric and<br/>Explanation for failed cases" ] C -- > D [ "4. Fix<br/>Fix Graph Node, Prompt<br/>or event tracking structure" ] D -- > E [ "5. Compare<br/>(agents-cli eval compare)<br/>Baseline vs. Candidate quantitative comparison" ] E -- >| Continuous Iteration | A Enter fullscreen mode Exit fullscreen mode II.

Revealing How agents-cli eval generate Tests a Go-based ADK 2.0 Agent? Many Go developers new to agents-cli wonder: "The agents-cli is a CLI tool written in Python; how can it evaluate my adk/v2/workflow Agent written in Go?"

The answer lies within the main.go and appinfo/appinfo.go we introduced in Phase 1! When you execute:

DRY_RUN = true agents-cli eval generate --dataset eval /datasets/basic-dataset.json Enter fullscreen mode Exit fullscreen mode agents-cli automatically completes the following four steps in the background:

go run. web --port <random_port> api -path_prefix / a2a appinfo Enter fullscreen mode Exit fullscreen mode Introspect Agent structure via GET /apps/app/app-info reflection: appinfo/appinfo.go uses Go's reflect and unsafe to traverse the Edges and Nodes of the entire workflowagent, assembling the embedded CompanyEnrichAgent ( llmagent ), Instructions, and tool declarations into a standard AppInfo JSON returned to agents-cli. Create Sessions one by one and call POST /run_sse to collect event streams: For each test case ( eval_case_id ) in eval/datasets/basic-dataset.json, agents-cli first calls POST /apps/app/users/eval_user/sessions to create an independent Session, then sends the test case's prompt to POST /run_sse, and records every ADK Event (including author, content, actions.state_delta, actions.requested_tool_confirmations, etc.) streamed back from the server. Output standardized trace file artifacts/traces/traces_<timestamp>.json: After testing is complete, it automatically shuts down the Go server and saves the full trajectory as a JSON, passing it to the next stage, agents-cli eval grade, for scoring.

III. Practical Design: A Golden Evaluation Set Covering 4 Core Scenarios ( eval/datasets/basic-dataset.json ) To fully cover the four core acceptance scenarios defined in Issue #7, we designed 6 Golden Eval Cases in eval/datasets/basic-dataset.json, marking the expected node trajectory ( expected_nodes ) and state assertions in each case's custom_Properties:

After writing eval/eval_config.yaml and eval/metrics.py, we confidently ran the first Baseline evaluation:

DRY_RUN = true agents-cli eval run --config eval /eval_config.yaml Enter fullscreen mode Exit fullscreen mode The first Baseline scorecard ( results_20261022_223104.json ) printed in the terminal shocked us:

╭─ Evaluation Summary ───────────────────────────╮ │ Metric Score Valid │ │ ────────────────────────────────────────────── │ │ workflow_trajectory_accuracy 0.0000 6/6 │ │ ocr_and_enrichment_quality 1.0000 6/6 │ │ duplicate_and_hitl_precision 0.7333 6/6 │ │ search_and_email_grounding 1.0000 6/6 │ ╰────────────────────────────────────────────────╯ Enter fullscreen mode Exit fullscreen mode At the same time, a very critical warning message appeared in the agents-cli eval grade execution log:

Dropped 26 event(s) with no content; the eval service rejects them

Why were 26 events dropped? We immediately checked the implementation of ADK Go workflow.FunctionNode and the event preprocessing logic of agents-cli:

Once the root cause was found, the solution was clear and elegant! We designed a lightweight helper function setNodeTrace in internal/workflow/nodes_ingest.go:

// setNodeTrace attaches a concise model text part to intermediate workflow node // events so that `agents-cli eval` (which drops events with empty Content) // retains the full node execution trajectory and state_delta history. func setNodeTrace ( ev * session. Event, nodeName, summary string ) { if ev == nil || ev. Content!= nil { return } ev. Content = genai. NewContentFromText ( fmt. Sprintf ( "[%s] %s", nodeName, summary ), genai. RoleModel ) } Enter fullscreen mode Exit fullscreen mode Then, we call setNodeTrace before each intermediate node finishes, for example:

At the same time, to prevent the [NodeName]... trajectory text of intermediate nodes from overwriting the ReplyText that Engine.executeTurn ultimately returns to the LINE caller, we simultaneously strengthened the state convergence logic in internal/workflow/graph.go —allowing TurnResult to read authoritative StateKeyReplyText, StateKeyFlexPreviewJSON, StateKeyFlexCarouselJSON, and StateKeyDraftEmail directly from the final converged Session.State!

After adding setNodeTrace and supporting single-turn batch HITL modification verification, we re-executed DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml:

Loaded 6 cases from eval/datasets/basic-dataset.json Starting local Go ADK server... Running 6 cases against http://127.0.0.1:56649 (app=app, concurrency=4)... Wrote 6 traces to artifacts/traces/traces_20261022_223515.json Loaded 6 trace(s) from artifacts/traces/traces_20261022_223515.json Evaluating metrics: ['workflow_trajectory_accuracy', 'ocr_and_enrichment_quality', 'duplicate_and_hitl_precision', 'search_and_email_grounding'] ╭─ Evaluation Summary ───────────────────────────╮ │ Metric Score Valid │ │ ────────────────────────────────────────────── │ │ workflow_trajectory_accuracy 1.0000 6/6 │ │ ocr_and_enrichment_quality 1.0000 6/6 │ │ duplicate_and_hitl_precision 1.0000 6/6 │ │ search_and_email_grounding 1.0000 6/6 │ ╰────────────────────────────────────────────────╯ Enter fullscreen mode Exit fullscreen mode This time, Dropped 0 event(s), and all intermediate node trajectories and state_delta were 100% preserved!

We immediately used the 5th step of the Quality Flywheel, agents-cli eval compare, to quantitatively compare the Baseline before the fix with the Candidate after the fix:

agents-cli eval compare \ --baseline artifacts/grade_results/results_20261022_223104.json \ --candidate artifacts/grade_results/results_20261022_223515.json ╭─ Summary Comparison ─────────────────────────────────────────────────────────╮ │ Metric Baseline Candidate Delta │ │ ──────────────────────────────────────────────────────────────────────────── │ │ duplicate_and_hitl_precision 0.7333 1.0000 +0.2667 │ │ ocr_and_enrichment_quality 1.0000 1.0000 0.0000 │ │ search_and_email_grounding 1.0000 1.0000 0.0000 │ │ workflow_trajectory_accuracy 0.0000 1.0000 +1.0000 │ ╰──────────────────────────────────────────────────────────────────────────────╯ Enter fullscreen mode Exit fullscreen mode Advancing from 0.0000 / 0.7333 to all four metrics reaching 1.0000 ( +1.0000 / +0.2667 ) is exactly the most charming value of the agents-cli eval quality flywheel— making every architectural adjustment traceable and provable!

In the eval/ directory, we adopted a "Python agents-cli eval + Go go test dual-track verification architecture":

In eval/eval_config.yaml, we registered 4 deterministic trajectory and quality graders (plus 1 card_assistant_llm_rubric available for online Vertex AI Judge models):

metrics_to_run: - workflow_trajectory_accuracy - ocr_and_enrichment_quality - duplicate_and_hitl_precision - search_and_email_grounding custom_metrics: workflow_trajectory_accuracy: type: custom module: eval.metrics function: evaluate_workflow_trajectory ocr_and_enrichment_quality: type: custom module: eval.metrics function: evaluate_ocr_and_enrichment duplicate_and_hitl_precision: type: custom module: eval.metrics function: evaluate_duplicate_and_hitl search_and_email_grounding: type: custom module: eval.metrics function: evaluate_search_and_email Enter fullscreen mode Exit fullscreen mode In eval/metrics.py, we defined helper functions _extract_executed_nodes(instance) and _merged_state_delta(instance) to parse the actual node sequence traversed by the Graph Workflow and the cumulative Session State directly from agent_data.turns[].events:

def _extract_executed_nodes ( instance: dict [ str, Any ]) -> list [ str ]: """ Extract executed workflow node names in order from `[NodeName]...` traces. """ nodes: list [ str ] = [] for ev in _iter_events ( instance ): content = ev. get ( " content " ) or {} for part in content. get ( " parts " ) or []: if isinstance ( part, dict ) and isinstance ( part. get ( " text " ), str ): for match in NODE_TRACE_PATTERN. findall ( part [ " text " ]): if not nodes or nodes [ - 1 ]!= match: nodes. append ( match ) return nodes Enter fullscreen mode Exit fullscreen mode This way, evaluate_workflow_trajectory can accurately check:

In addition to agents-cli eval called via HTTP SSE, we also wrote Go native TestEvalDatasetAndAppInfoIntrospection and TestPhase4WorkflowTrajectorySuite in eval/eval_test.go, which can be executed in milliseconds via go test./... or GitHub Actions CI:

Now, with just one line make eval, you can run both agents-cli eval run and the Go native test suite:

$ make eval DRY_RUN=true agents-cli eval run --config eval/eval_config.yaml... go test -v./eval/... === RUN TestEvalDatasetAndAppInfoIntrospection --- PASS: TestEvalDatasetAndAppInfoIntrospection (0.00s) === RUN TestPhase4WorkflowTrajectorySuite --- PASS: TestPhase4WorkflowTrajectorySuite/Case1_IngestFanOutFanInOCR (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case2_DuplicatePromotionDetection (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case3_MultiTurnHITLEditAndConfirmLoop (0.00s) --- PASS: TestPhase4WorkflowTrajectorySuite/Case4_SearchHybridAndBroadFallbackAndEmailDraft (0.00s) PASS ok github.com/kkdai/line-card-assistant/eval 0.436s Enter fullscreen mode Exit fullscreen mode VII. Summary and Next Preview (Phase 5 Final Chapter: Official Deployment, Terraform IaC, and Cloud Trace Observability) As of now, our LINE AI Business Card Master Roadmap (Issue #9) has completed the first four major stages:

In the next article (which is also the final chapter of this series, Phase 5 — Issue #8 ), we will officially push the LINE AI Business Card Master, which passed the evaluation with a perfect 1.0000 score, into the production environment via agents-cli deploy. We will also fully integrate Terraform IaC (Infrastructure as Code), GitHub Actions CI/CD automated evaluation and deployment pipelines, and OpenTelemetry + Google Cloud Trace full-link observability. Stay tuned!