[Agent Platform in Action (6)] Towards Production-Grade AI Agent Ops: Building Full-Stack Observable LINE Agents with agents-cli

Posted on Oct 25 • Originally published at evanlin.com on Oct 25

Welcome to the sixth (and final) article in the " Agent Platform in Action " series! Looking back at the past five articles, we have taken LINE AI Card Assistant ( kkdai/line-card-assistant ) through the four major stages of evolution from a local prototype to an enterprise-grade AI Agent architecture:

In this final chapter today, we will complete the last piece of the puzzle for Roadmap #9 — Phase 5 ( Issue #8 ): Official Agent Runtime Deployment, Terraform IaC, GitHub Actions CI/CD, and Full-Stack Cloud Trace Observability!

In many AI projects, even if the code and evaluation tests are written, production environments often become operational nightmares due to the following four issues:

This is exactly what the Phase 5 Full-Stack Architecture Diagram at the beginning of the article depicts through four core pillars:

In Phase 1, we introduced the Prototype-First philosophy of agents-cli: keep the project lightweight first, and then execute agents-cli scaffold enhance to layer on production infrastructure when ready for the cloud.

We execute the following directly in the project root directory:

uvx google-agents-cli@1.10.0 scaffold enhance. \ --cicd-runner github_actions \ --region asia-east1 \ --yes Enter fullscreen mode Exit fullscreen mode agents-cli automatically detected our agents-cli-manifest.yaml ( template: adk_go, deployment_target: agent_runtime ) and, without breaking any existing app/, internal/, eval/, or sessions/ code, added three major modules to the project:

Next, we perform customized enhancements for the specific requirements of LINE AI Card Assistant (LINE credentials, Cloud SQL PostgreSQL permissions, Taiwan asia-east1 region)!

Let's open deployment/terraform/ and see how Terraform declaratively builds the entire Google Cloud Agent Platform:

In deployment/terraform/single-project/service.tf, we use the google_vertex_ai_reasoning_engine resource from the google-beta provider to package the Go source code and declare it as a managed Agent Runtime instance. We also set lifecycle { ignore_changes = [spec.source_code_spec] } so that daily code updates are handled by the CI/CD's agents-cli deploy for fast rolling deployment, while infrastructure settings (IAM, Buckets, environment variables) are unified under Terraform control:

resource "google_vertex_ai_reasoning_engine" "app" { provider = google-beta display_name = var. project_name project = var. project_id region = var. region labels = { "agent-framework" = var. agent_framework } spec { identity_type = "AGENT_IDENTITY" agent_framework = var. agent_framework source_code_spec { inline_source { source_archive = data. archive_file. source. output_path } developer_connect_source {} } container_spec { image_uri = "us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-15:latest" } } lifecycle { ignore_changes = [ spec [ 0 ]. source_code_spec, spec [ 0 ]. container_spec, ] } } Enter fullscreen mode Exit fullscreen mode 2. Least Privilege IAM and Secret Manager Integration ( iam.tf & variables.tf ) To prevent LINE_CHANNEL_SECRET, LINE_CHANNEL_ACCESS_TOKEN, and DATABASE_URL from appearing in plain text in configuration files, we declare Google Cloud Secret Manager secrets in deployment/terraform/single-project/iam.tf and variables.tf. We grant roles/secretmanager.secretAccessor and roles/cloudsql.client to both the Vertex AI Service Agent ( gcp-sa-aiplatform-re ) and the Agent Identity Principal ( principal://${google_vertex_ai_reasoning_engine.app.spec.effective_identity} ):

# Secret Manager secrets for LINE Messaging API credentials and Cloud SQL DSN locals { line_card_secrets = [ "line-channel-secret", "line-channel-access-token", "line-card-database-url", ] } resource "google_secret_manager_secret" "app_secrets" { for_each = toset ( local. line_card_secrets ) project = var. project_id secret_id = each. value replication { auto {} } depends_on = [ resource. google_project_service. services ] } Enter fullscreen mode Exit fullscreen mode This way, during deployment, you only need to pass --secrets="LINE_CHANNEL_SECRET=line-channel-secret,LINE_CHANNEL_ACCESS_TOKEN=line-channel-access-token" to agents-cli deploy, and the Agent Runtime will securely mount the secrets from Secret Manager when starting the container!

In deployment/terraform/shared/telemetry.tf, Terraform automatically creates a Cloud Logging Bucket with a 365-day retention period, enables enable_analytics = true, and directly maps telemetry logs to a BigQuery Dataset ( telemetry_logs_dataset ) via google_logging_linked_dataset:

resource "google_logging_project_bucket_config" "telemetry_logs_bucket" { project = each. value location = "global" bucket_id = " ${ var. project_name } -telemetry" retention_days = 365 enable_analytics = true } resource "google_logging_linked_dataset" "telemetry_logs_dataset" { bucket = google_logging_project_bucket_config. telemetry_logs_bucket [ each. key ]. bucket_id link_id = replace ( " ${ var. project_name } _telemetry", "-", "_" ) project = each. value location = "global" } Enter fullscreen mode Exit fullscreen mode This means all structured telemetry data output by OpenTelemetry can be analyzed directly in BigQuery using standard SQL without writing additional ETL scripts!

In the observability.go generated in Phase 1, the system was already configured with an OTLP HTTP Exporter pointing to telemetry.googleapis.com and Google ADC automatic authentication. However, by default, we only see top-level traces for the entire HTTP request or LLM call, but not the execution time and routing decisions of each FunctionNode within the ADK 2.0 Graph Workflow.

To present the processing of each business card as a complete waterfall span tree on Google Cloud Trace, we added internal/workflow/telemetry.go:

package workflow import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" "go.opentelemetry.io/otel/trace" ) const workflowTracerName = "github.com/kkdai/line-card-assistant/internal/workflow" const ( AttrWorkflowNode = attribute. Key ( "adk.workflow.node" ) AttrWorkflowRoute = attribute. Key ( "adk.workflow.route" ) AttrInvocationID = attribute.

Key ( "adk.invocation.id" ) AttrSessionID = attribute. Key ( "adk.session.id" ) AttrUserID = attribute. Key ( "adk.user.id" ) AttrCardID = attribute.

Key ( "adk.card.id" ) AttrCardCompany = attribute. Key ( "adk.card.company" ) AttrCardDuplicate = attribute. Key ( "adk.card.is_duplicate" ) AttrHITLInterrupt = attribute.

Key ( "adk.hitl.interrupt_id" ) AttrHITLAction = attribute. Key ( "adk.hitl.action" ) AttrSearchMatches = attribute. Key ( "adk.search.matches" ) AttrSearchKeywords = attribute.

Key ( "adk.search.keywords" ) ) func startNodeSpan ( ctx context. Context, nodeName string, attrs... attribute. KeyValue ) ( context.

Context, trace. Span ) { baseAttrs:= make ([] attribute. KeyValue, 0, len ( attrs ) + 1 ) baseAttrs = append ( baseAttrs, AttrWorkflowNode.

String ( nodeName )) baseAttrs = append ( baseAttrs, attrs... ) return otel. Tracer ( workflowTracerName ). Start ( ctx, "workflow.node." + nodeName, trace.

WithAttributes ( baseAttrs... ), ) } Enter fullscreen mode Exit fullscreen mode Next, we inject startNodeSpan and endNodeSpan into all nodes in internal/workflow/nodes_ingest.go and internal/workflow/nodes_search.go. For example, in HybridSearchNode:

func newHybridSearchNode ( cardStore store. CardStore ) * workflow. FunctionNode { return workflow.

NewFunctionNode ( NodeHybridSearch, func ( ctx agent. Context, _ any ) ( ev * session. Event, err error ) { spanCtx, span:= startNodeSpan ( ctx, NodeHybridSearch, AttrInvocationID.

String ( ctx. InvocationID ())) defer func () { routeAttr:= "" if ev!= nil && len ( ev. Routes ) > 0 { routeAttr = ev.

Routes [ 0 ] } endNodeSpan ( span, err, AttrWorkflowRoute. String ( routeAttr )) }() //... execute cardStore.HybridSearch(spanCtx, filter)... span. SetAttributes ( AttrSearchMatches.

Int ( len ( cards ))) //... }, workflow. NodeConfig {}, ) } Enter fullscreen mode Exit fullscreen mode When a user uploads a business card or performs a hybrid search, Cloud Trace automatically draws a hierarchical Span tree like the one below, including the time spent in the three child nodes of the parallel Fan-Out and the business attributes of each node:

flowchart TD Root [ "invoke_workflow (line_card_assistant)<br/>trace_id: 4bf92f3577b34da6a3ce929d0e0e4736" ] -- > R [ "workflow.node.RouterNode<br/>adk.workflow.route = 'ingest' (1.2ms)" ] R -- > V [ "workflow.node.VisionOCRNode<br/>adk.card.company = 'Taiwan AI Labs' (640ms)" ] V -- > C1 [ "workflow.node.CleanAndDedupNode<br/>adk.card.is_duplicate = false (8.5ms)" ] V -- > C2 [ "workflow.node.CompanyEnrichNode<br/>GoogleSearch Grounding (780ms)" ] V -- > C3 [ "workflow.node.ContextMatchNode<br/>inferMetContext (310ms)" ] C1 & C2 & C3 -- > B [ "workflow.node.BuildPreviewNode<br/>Flex Bubble JSON (2.1ms)" ] B -- > H [ "workflow.node.HITLConfirmNode<br/>adk.hitl.interrupt_id = 'hitl_confirm_1'<br/>adk.hitl.action = 'pause' (0.8ms)" ] Enter fullscreen mode Exit fullscreen mode We also wrote automated unit tests in internal/workflow/telemetry_test.go using the OpenTelemetry SDK's tracetest.NewInMemoryExporter(), ensuring that "card recognition + HITL modification loop + confirmation save" and "hybrid search with 0 results automatically degrading to fuzzy search" 100% generate the expected Span nodes and attribute tags!

With Terraform IaC and OpenTelemetry tracing in place, we officially embedded the agents-cli eval quality flywheel created in Phase 4 into the GitHub Actions CI/CD pipeline, forming a two-stage automated delivery:

flowchart LR subgraph PR [ "1. Pull Request Quality Gate (pr_checks.yaml)" ] P1 [ "go test -race -v./...<br/>(Unit + OTel Span + Trajectory Tests)" ] -- > P2 [ "go test -tags=e2e<br/>(A2A + /app-info E2E Tests)" ] P2 -- > P3 [ "agents-cli eval run<br/>(4 major metrics must = 1.0000)" ] end subgraph CD [ "2. Staging & Production CD (staging.yaml -> deploy-to-prod.yaml)" ] S1 [ "WIF Keyless Auth<br/>(google-github-actions/auth)" ] -- > S2 [ "agents-cli deploy<br/>Deploy to Staging Agent Runtime" ] S2 -- > S3 [ "Vegeta Load Test<br/>30s pressure test + report upload to GCS" ] S3 -- > S4 [ "GitHub Environment<br/>Production Manual Approval Gate" ] S4 -- > S5 [ "agents-cli deploy<br/>Deploy to Production Agent Runtime" ] end PR -- >| Merge to main | CD Enter fullscreen mode Exit fullscreen mode 1.

PR Quality Gatekeeper (.github/workflows/pr_checks.yaml ) In.github/workflows/pr_checks.yaml, whenever a PR modifies app/**, internal/**, cmd/**, sessions/**, or eval/**, GitHub Actions will execute in order:

After the PR is merged into the main branch,.github/workflows/staging.yaml will:

Before submitting the code as a PR, we fully executed three major verifications locally:

$ uvx google-agents-cli@1.10.0 deploy \ --project line-vertex \ --region asia-east1 \ --dry-run \ --no-confirm-project Enter fullscreen mode Exit fullscreen mode Preflight output results:

$ go test -v -tags = e2e./... Enter fullscreen mode Exit fullscreen mode Includes the newly added TestWorkflowOpenTelemetrySpans_IngestAndHITL, TestWorkflowOpenTelemetrySpans_SearchFallback, and all tests under e2e/integration such as TestA2AAgentCard, TestA2AMessageSend, TestA2AErrorHandling, and TestAppInfo — all PASS!

$ DRY_RUN = true make eval Enter fullscreen mode Exit fullscreen mode The 6 golden cases across the 4 custom metrics ( workflow_trajectory_accuracy, ocr_and_enrichment_quality, duplicate_and_hitl_precision, search_and_email_grounding ) still maintain 1.0000 (100% Pass)!

With the completion of Phase 5 ( Issue #8 ), the five stages planned in Roadmap Issue #9 have been 100% achieved and closed! Let's conclude with a summary table reviewing this journey from 0 to 1 with Google ADK 2.0 Go + Google Cloud Agent Platform:

The complete open-source code, Terraform modules, and evaluation datasets have been synchronized to the GitHub Repo: kkdai/line-card-assistant. If you are also building enterprise-grade AI Agents with Go and Google ADK 2.0, I hope this series serves as your best practical blueprint for moving from local development all the way to the Google Cloud Agent Platform production environment!