Use decision models with Amazon Bedrock AgentCore Evaluations

Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.

Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model.

Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong.

This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev, a hosted decision model from TypeSafe. The second is Strands Decider 2B, an open-source decision model you run yourself. By the end of this post, you will know how to convert the AgentCore evaluator input into a decision model request, and the model’s probabilities back into an AgentCore evaluation result. First, though, when should you reach for a custom LLM-as-a-judge evaluator, and when does a decision model fit better? The next section breaks down what a decision model is and the trade-offs involved.

A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically. This makes decision models well-suited for evaluation tasks where the judgment can be stated as a choice among a fixed set of options: “Is this response grounded?” or “Did the agent complete the user’s goal?”

Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it.

Jev, the hosted decision model from TypeSafe, and Strands Decider, an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types of questions. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution and a confidence. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score, the distribution over levels, and a confidence. Many evaluation rubrics fit one of these types. “Is every fact in this response supported by the tool results?” is a yes/no question, and “How helpful was this response, on a four-level rubric?” is a score.

The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks where the explanation is as useful as the score. Decision models are not a replacement for LLM evaluators. They’re an addition that the code-based evaluator framework makes possible. Use them for narrow, well-defined checks alongside the built-in evaluators’ broader assessment.

To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.

The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fixed data in this code sample to make it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When you additionally have ground truth, you can confirm whether each evaluator catches the right failures. This code sample helps you set up the framework for testing decision models compared to LLM-as-a-judge for your own use cases.

Figure 1: End-to-end architecture of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider

To deploy the sample, you need the following:

You can find the full code sample in the AgentCore samples GitHub repository. Two Python scripts handle everything:

Running evaluate.py without flags executes five steps:

At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag, and they run alongside the base evaluators with no changes to the existing flow. Using the flags --with-jev --jev-secret-arn <ARN> deploys jev-evaluator and registers three evaluators: JevGroundedness, JevHelpfulness, and JevGoalCompletion. Using the flag --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. Add --decider-ec2 to auto-provision the Amazon EC2 instance, security groups, IAM role, and Lambda VPC configuration without manual console steps.

Figure 2: The evaluation pipeline deployed by evaluate.py, from deterministic checks through built-in and decision-model evaluators

Each decision-model backend contributes three evaluators:

Using the flags --with-jev and --with-decider creates a custom code-based evaluator, where you have complete control of the code you want to run as your evaluator. We create AWS Lambda functions where the decision model is invoked inside an AWS Lambda function and returns a label or a numeric value. This Lambda function is registered as the code-based evaluator in AgentCore Evaluations.

The Lambda that is registered as the evaluator receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content can arrive in and then formats it appropriately to make the call to the decision model. For trace-level evaluation, only the targeted turn and brief prior history are sent, whereas the full conversation is included for session-level.

Figure 3: How the code-based evaluator Lambda adapts OpenTelemetry spans for the decision model and maps the result back to AgentCore

The adapter converts the model’s probability distribution into a result that AgentCore evaluation expects. For a noul question, the probability of true becomes the value and the most probable answer becomes the label. For a score question, the value is the probability-weighted expected level scaled to [0, 1]. For a choice question, it is the weighted average of predefined option values.

When you rewrite a rubric as a decision model question, you’ve created a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than the original intended, and you won’t see that drift until you check it against real sessions. Before you gate a release on a decision-model evaluator, validate it the same way you’d validate any new measurement, against a labeled sample of your own traffic. The goal of this section is to walk you through that validation end to end, so you finish with an evaluator you can threshold with confidence.

Step 1. Build a labeled sample. Pull a set of stored sessions from your agent’s traffic, with enough coverage of the failures and edge cases you care about. A sample that covers only the happy path tells you nothing about where the evaluator breaks.

Step 2. Get human reference labels. Have three independent labelers score each turn for the property your question measures and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split. Although unanimous turns are the clear-cut cases, the split-vote turns are the valuable ones, because they reveal ambiguity in the rubric itself before the model ever sees it.

Step 3. Run both evaluators on the same inputs. Run the decision-model evaluator and the matching built-in evaluator over those same sessions in a single batch evaluation. Scoring identical inputs is what makes the comparison apples to apples.

Step 4. Read the results through three lenses. With the majority-vote labels and the easy and hard split in hand, you can now answer the three questions that tell you whether the evaluator is ready.

Step 5. Re-run the check when anything changes. Treat this as a repeatable check rather than a one-time gate. Any time you reword a question or move to a new model version, run it again. Each result’s explanation records the model version that produced it, so you know which version a given score came from.

The built-in evaluators remain the foundation of broad evaluation strategy in AgentCore Evaluations. They assess the response against a rubric and its supporting context, and explain the judgment so developers can investigate what went wrong. For the HR Assistant, a faithfulness evaluation can identify a claim that contradicts a tool result, whereas a helpfulness evaluation can point out that the agent answered a balance question but left a requested action incomplete.

Faithfulness, helpfulness, and correctness can require connecting evidence, identifying missing information, or working through a solution. The preprint JEV-as-a-Judge: Accept When Confident, Escalate When Unsure describes a related distinction as “reading versus deriving.” A narrow decision-model check can flag a turn for attention, whereas the built-in evaluator provides the broader assessment and reasoning needed to review it. Decision models shine brightest when you treat them as a gate rather than a replacement. The cascade pattern works like this. First, run a fast, calibrated decision model on every turn of every session to get a cheap pass/fail signal. Then invoke a full LLM evaluator only on sessions where the signal falls below your threshold.

Figure 4: The cascade pattern, where a calibrated decision model screens every turn and only low-confidence turns escalate to a built-in evaluator

To avoid ongoing charges, run the cleanup.py that deletes the agent hosted on AgentCore runtime, the evaluators and evaluation config, the AWS Lambda functions that are registered as the evaluators, the EC2 instance that hosts the Strands Decider model, AWS Secrets Manager key and the associated log group on CloudWatch.

In this post, you connected two decision models, Jev and Strands Decider, to AgentCore Evaluations using code-based evaluators, and the accompanying code sample gives you a framework for testing those models and seeing where they add value as evaluators. The built-in evaluators remain your baseline. Decision models add narrow, validated checks that come with a measure of certainty you can threshold and act on. The practical path forward is to add decision-model checks alongside the built-in evaluators you already run, measure how often they agree with your reference labels on your own traffic, and expand coverage only once you trust them.

To get started, review the Amazon Bedrock AgentCore Evaluations documentation, integrate custom evaluators using code-based evaluators and take a deeper look into Strands Decider model. Get started with code samples by deploying the sample evaluators from the Amazon Bedrock AgentCore sample repository.

Sungyeon is an Applied Scientist at AWS, where he conducts research and develops machine learning solutions for agentic applications. His research interests include language model training and evaluation, generative AI, and agentic systems. He has worked across various AI domains, including computer vision, multimodal learning, and retrieval systems, in both academia and industry.

Bharathi is a Generative AI Data Scientist at AWS. She is passionate about Responsible AI to increase the reliability of AI agents in real world scenarios. Bharathi guides internal teams and AWS customers on their responsible AI journey. She has presented her work at various machine learning conferences.

Jonathan is a Principal Engineer at AWS. His work focuses on infrastructure and tooling to support AI agent environments and evaluation. Jonathan works with both research and engineering teams to shape the direction of next-generation AI products and services for AWS customers.

Haibo is a Principal Applied Scientist and Science Manager working on agentic AI at Amazon. He holds a Ph.D. from the University of Utah. His work focuses on large language models (LLMs) and AI agents, where he leads research in areas such as LLM post-training, benchmarking, agent evaluation, and tool optimization. He has served as an area chair for conferences such as AAAI and ACL, and previously as Program Chair for KDD 2025 Workshop on Prompt Optimization.