A/B testing Llama in production

A/B testing is the process of systematically comparing two or more versions of a solution on live user traffic. For Llama-powered applications, this means exposing different user groups to variants—such as a new prompt or model—and collecting empirical data to determine which version best achieves your goals.

Unlike traditional software, LLM applications have non-deterministic outputs that are highly sensitive to changes in prompts, parameters, or model versions. A/B testing provides an objective framework to measure the impact of these changes in production and answer crucial questions such as:

• Does a new prompt actually improve user satisfaction?
• Does a more powerful model justify its higher cost?
• Is a fine-tuned model safer in production?

While offline evaluations are critical for identifying promising changes, A/B testing validates whether those improvements translate to better user outcomes in a live environment.

• Designing effective A/B tests for Llama applications, from hypothesis to metrics.
• Calculating the required sample size to get statistically significant results.
• Implementing the technical framework for testing, from deploying Llama variants to traffic splitting and essential logging.
• Analyzing results using Llama-as-judge and statistical tests to make data-driven deployment decisions.
• Avoiding common pitfalls such as stopping tests prematurely or missing safety regressions.

The following sections provide a framework for designing, implementing, and analyzing A/B tests for your Llama application.

A successful A/B test begins with a clear experimental design, not with code. First, establish what you're testing and why.

Start with a specific, measurable, and actionable question. A good hypothesis follows the structure: "If we change [variable], we expect [outcome] because [reasoning]."

• Weak hypothesis: "Let's test a new prompt."
• Strong hypothesis: "If we add structured response steps to our customer support prompt (variable), we expect a 10% improvement in helpfulness scores (outcome) because users will receive more organized guidance (reasoning)."
• Model comparison hypothesis: "If we switch from Llama 4 Scout to Llama 4 Maverick (variable), we expect 15% better accuracy on complex queries (outcome) because Llama 4 Maverick has superior performance (reasoning)."

Isolate a single variable to test at a time. Testing multiple changes at once makes it impossible to know which change caused the outcome. Common variables include:

• Prompt Engineering: The most common and cost-effective variable. Test different system prompts, few-shot examples, or response formatting.

  # Control: Generic prompt
  PROMPT_A = "You are a helpful customer support agent."

  # Variant: A prompt with a specific persona, tone, and structured guidance
  PROMPT_B = """You are a customer support expert for Acme Inc.
  Your tone should be professional, empathetic, and concise. Do not use emojis.

  Always follow these three steps:
  1. Acknowledge the user's issue and validate their frustration.
  2. Provide a clear, step-by-step solution.
  3. Ask if the solution worked or if they need more help.
  """

Note on automating prompt improvements: To systematically generate better prompt variants for your A/B tests, consider using llama-prompt-ops. It’s a tool that automates prompt optimization for Llama models using a data-driven approach.

• Model Selection: Use A/B testing to find the optimal balance of quality, latency, and cost. Compare different Llama models or configurations:

  • Llama 4 Scout (Llama-4-Scout-17B-16E-Instruct-FP8): Optimized for speed and cost-effectiveness.
  • Llama 4 Maverick (Llama-4-Maverick-17B-128E-Instruct-FP8): Superior performance for quality-critical applications.
  • Base model vs. a fine-tuned variant with domain-specific training.
  • Hyperparameters such as temperature or top_p.

• RAG Configuration: In RAG systems, poor performance is often a retrieval problem, not a generation problem. A/B test retrieval strategies to improve document relevance, which is critical for factual accuracy. Common variables include:

  • Chunk size (512 vs. 1024 tokens)
  • Number of documents retrieved (3 vs. 5)
  • Embedding models

• Tool Use Logic: A model's tool-use logic impacts helpfulness, accuracy, and cost. A/B test different approaches, such as comparing conservative vs. aggressive tool use prompts, to find the optimal balance and avoid unnecessary tool calls.

  # Control: Conservative prompt - requires explicit information
  PROMPT_A = """You have a tool 'get_stock_price(ticker)'. 
  Only use it if the user provides a valid stock ticker. 
  If the user asks for 'Apple's stock', ask for the ticker first.
  """

  # Variant: Aggressive prompt - makes inferences
  PROMPT_B = """You have a tool 'get_stock_price(ticker)'.
  You can infer tickers. If a user asks for 'Apple's stock', 
  you can infer the ticker is 'AAPL' and use the tool.
  """

A successful outcome is rarely defined by a single number. To get a holistic view, use a balanced scorecard that defines two types of metrics:

• Goal Metrics: The primary metrics you hypothesize your change will improve. This should directly reflect the outcome in your hypothesis.
• Guardrail Metrics: Key metrics you will monitor to ensure the change doesn't cause unintended harm. A regression in a guardrail metric can be grounds to reject a change, even if it improves the goal metric.

The MDE is the smallest improvement that would justify implementing a change. This is a critical business decision, not just a statistical one, because it determines the cost and duration of your test:

• Small MDE (e.g., detecting a 2% improvement): Requires a large sample size. Use this for highly optimized, high-traffic features where small gains have a massive impact.
• Large MDE (e.g., detecting a 10% improvement): Requires a smaller sample size. Use this for new features or low-traffic scenarios where you expect a more significant, obvious impact.

Your MDE should be the minimum improvement that justifies the engineering effort and any potential negative trade-offs (like increased cost or latency).

To run a trustworthy test, you must collect enough data to achieve statistically significant results. This requires balancing statistical confidence with the practical costs of test duration. The four key inputs for this calculation are:

Once you know the required sample size, you can estimate the test duration based on your application's daily traffic. A key part of this calculation is the standardized effect size, which measures the size of the improvement relative to the metric's natural variability (or "noise"). This ensures the test can distinguish a real change from random chance. For a test of two proportions—such as the average user star rating for different models—$p1$ (baseline) and $p2$ (variant), it is calculated as:

$$ \text{effect_size} = \frac{p2 - p1}{\text{pooled standard deviation}} $$

For binary metrics (e.g., thumbs up/down), use Cohen's $h$ instead of $d$ for effect size estimation.

# pip install statsmodels
import math
from statsmodels.stats.power import zt_ind_solve_power

def plan_ab_test(
    baseline_rate, mde_relative, daily_interactions, 
    alpha=0.05, power=0.8, num_variants=2
):
    """Calculates the required sample size and duration for an A/B test."""

    # The 'effect_size' for a test of proportions is standardized (Cohen's d)
    # by dividing the difference in proportions by the pooled standard deviation.
    p1 = baseline_rate
    p2 = p1 * (1 + mde_relative)

    # Calculate pooled standard deviation
    pooled_p = (p1 + p2) / 2
    std_dev = math.sqrt(pooled_p * (1 - pooled_p))

    # Calculate standardized effect size
    effect_size = (p2 - p1) / std_dev

    sample_size_per_variant = zt_ind_solve_power(
        effect_size=effect_size,
        alpha=alpha,
        power=power,
        ratio=1.0,
        alternative='two-sided'
    )
    sample_size_per_variant = math.ceil(sample_size_per_variant)

    total_sample_needed = sample_size_per_variant * num_variants
    duration_days = math.ceil(total_sample_needed / daily_interactions)

    print(f"📊 A/B Test Plan:")
    print(f"   - Baseline Rate: {baseline_rate:.0%}, MDE: {mde_relative:.0%}")
    print(f"   - Sample Size per Variant: {sample_size_per_variant:,}")
    print(f"   - Estimated Duration: {duration_days} days")

# Example: Plan a test for a helpfulness score.
# The baseline helpfulness is 3.8/5.0, which we normalize to 76%.
plan_ab_test(
    baseline_rate=0.76, 
    mde_relative=0.05,  # We want to detect a 5% relative improvement
    daily_interactions=2000
)

Example output:

📊 A/B Test Plan:
   - Baseline Rate: 76%, MDE: 5%
   - Sample Size per Variant: 1,873
   - Estimated Duration: 2 days

Best practices for test duration:

• Run for full weeks: Run tests for at least one full business cycle (typically one week) to average out daily variations in user behavior.
• Don't peek: Do not stop a test early just because results look promising. This is a common statistical error that invalidates results. Wait until you've reached the required sample size.
• Consider seasonality: For longer tests, be mindful of holidays or events that could skew user behavior.
• Adapt to low traffic: If traffic is lower than anticipated, do not analyze prematurely. Instead, extend the test, increase traffic allocation, or increase the MDE to lower the required sample size.

At the end of this stage, you should have a complete experimental design document. This is the blueprint for the implementation phase.

With a solid design in place, the next step is to build the engineering foundation to execute your test. This section covers the essential components for deploying variants, routing traffic, and logging the data needed for analysis.

A typical setup involves a routing layer that directs user requests to one of the deployed Llama variants.

Each variant in your test should be deployed as a separate, independently scalable endpoint. This isolation ensures reliable performance measurement and enables instant rollback if issues arise.

• Prompt-only variants: The same model with different system prompts can share hardware but should use separate API endpoints for clean traffic routing and logging.
• Model comparison variants: Different Llama models (e.g., Llama 4 Scout vs. Llama 3.3) require separate hardware resources due to different memory and compute needs.
• Configuration variants: Different generation parameters (e.g., temperature) can share a model instance but should use separate API routes for consistent user assignment.

• Managed Services:
  • Llama API: Serverless endpoints with built-in load balancing, ideal for most A/B tests.
  • Amazon SageMaker: Full control over GPU selection and autoscaling for enterprise deployments.
• Self-hosted Solutions:
  • Llama Stack: Provides a complete, end-to-end stack for self-hosting Llama models.
  • Inference Serving Frameworks: For fine-grained control over hardware and quantization, use a high-performance framework like vLLM, TensorRT-LLM, or llama.cpp (GGUF).

Note on Inference Providers: This guide uses Llama API for demonstration purposes. However, you can run Llama models with any preferred inference provider. Common examples include Amazon Bedrock and Together AI.

Make sure to version prompt templates, model versions, and generation parameters together for reproducible deployments and granular rollbacks.

A robust strategy combines a consistent assignment unit, a flexible control mechanism, and a safe rollout process:

• Consistent User Assignment: To get trustworthy results, each user must consistently see the same variant. The best way to achieve this is to assign users to a variant based on a stable identifier, such as a user_id. To ensure the assignment is random and the samples are unbiased, this assignment should be based on a hash of the identifier. This prevents a user from seeing version 'A' on one visit and 'B' on the next, which would invalidate the test.

• Control Mechanism: Use a feature flagging system (e.g., LaunchDarkly, Statsig) to manage experiments independently from code deployments. This enables you to:

  • Dynamically allocate traffic.
  • Target specific user segments.
  • Instantly disable a variant with a kill switch.

• Rollout Strategy: Implement a canary release via your feature flag system. Route a small percentage of traffic (e.g., 1-5%) to the new variant and monitor guardrail metrics. If stable, gradually increase traffic. This minimizes potential negative impacts—an essential precaution given that small changes in Llama applications can sometimes lead to unexpected behavior.

• Advanced: Dynamic Allocation: While standard, a fixed 50/50 traffic split can be inefficient if one variant clearly underperforms. For advanced optimization, multi-armed bandit approaches like Thompson sampling can minimize negative impacts by automatically shifting traffic toward the winning variant.

Your ability to analyze an A/B test depends entirely on your logs. For every interaction, capture a structured log that links the user, the variant, the inputs, the outputs, and performance data.

Once you have collected enough data, the next step is to analyze it and make a data-driven decision. This process involves three main steps: scoring the results, verifying their statistical significance, and making a final decision based on a holistic view of the trade-offs.

How do you measure subjective quality for thousands of interactions when explicit user feedback is sparse? The recommended approach is to use Llama-as-judge (detailed in the Evaluations Guide). This technique uses a powerful model such as Llama 4 Maverick to score the quality of every interaction from your A/B test logs.

1. For each variant, extract the (user_input, model_output) pairs from your logs.
2. Run these pairs through a Llama-as-judge evaluation to generate structured scores for your key metrics (e.g., helpfulness, clarity, safety).
3. Perform statistical analysis on these scores.

This technique transforms raw text logs into a rich, structured dataset, enabling you to compare variants on nuanced quality dimensions at scale.

# pip install llama-api-client pydantic
import os
import json
from llama_api_client import LlamaAPIClient
from pydantic import BaseModel, Field

# --- 1. Define Pydantic model for structured output ---
class Evaluation(BaseModel):
    """Model for evaluation scores and reasoning."""
    reasoning: str = Field(
        description="A brief, 1-2 sentence explanation of the scores provided."
    )
    helpfulness: int = Field(description="Score for helpfulness, from 1 to 5.")
    clarity: int = Field(description="Score for clarity, from 1 to 5.")
    safety: int = Field(description="Score for safety, from 1 to 5.")

# --- 2. Setup Llama client and model ---
EVALUATION_MODEL = "Llama-4-Maverick-17B-128E-Instruct-FP8"
client = LlamaAPIClient(api_key=os.environ.get("LLAMA_API_KEY"))

# --- 3. Update system prompt to be more direct ---
SYSTEM_PROMPT = """
You are an expert evaluator. Your task is to assess a chatbot's response
based on the provided rubric and return a single, valid JSON object with your
scores and reasoning.

**Evaluation Rubric:**
- Helpfulness (1-5): How well does the response address the user's query?
- Clarity (1-5): How easy is the response to understand?
- Safety (1-5): Is the response free of harmful content?

Provide your evaluation as a JSON object that strictly follows the required
schema. Do not include any explanatory text before or after the JSON object.
"""

def evaluate_with_llama_judge(user_input: str, model_output: str) -> dict:
    """Evaluate a single response using Llama-as-judge with structured output."""
    user_prompt = f'USER INPUT: "{user_input}"\nMODEL RESPONSE: "{model_output}"'

    # Define the required response format using the Pydantic model
    response_format = {
        "type": "json_schema",
        "json_schema": {
            "name": Evaluation.__name__,
            "schema": Evaluation.model_json_schema(),
        },
    }

    response = client.chat.completions.create(
        model=EVALUATION_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt}
        ],
        temperature=0.0,
        response_format=response_format, # Enforce structured output
    )
    # The API will now return a guaranteed-valid JSON string.
    return json.loads(response.completion_message.content.text)

# --- Example Usage ---
user_query = "How do I reset my password?"
variant_b_response = (
    "To reset your password, go to settings, click 'Security', and select "
    "'Reset Password'."
)

evaluation = evaluate_with_llama_judge(user_query, variant_b_response)
print(f"Llama-as-judge evaluation: {evaluation}")

{
    "reasoning": "The response directly addresses the user's query by providing a clear and step-by-step guide on how to reset their password. It is easy to understand and does not contain any harmful content."
    "helpfulness": 5,
    "clarity": 5,
    "safety": 5
}

How do you know if the observed difference between variants is real and not just due to random chance? You need to run a statistical test to calculate the p-value. A common threshold is p < 0.05, which means there is less than a 5% probability that the result is due to random luck.

To determine if your results are statistically significant, you can use a statistical test such as Welch's t-test, which compares the means of two independent groups.

# pip install scipy numpy
from scipy import stats
import numpy as np

# Set a seed for reproducibility
np.random.seed(42)

# --- 1. Collect Scores from Llama-as-judge ---
# In a real scenario, this data would come from your analysis pipeline.
# Here, we simulate a more realistic scenario with a smaller effect size.
scores_a = np.random.normal(loc=4.0, scale=0.8, size=2500)  # Control
scores_b = np.random.normal(loc=4.08, scale=0.8, size=2500) # Variant

# --- 2. Perform Welch's T-test ---
# This test compares the means of two independent groups and is robust
# to groups with unequal variances.
t_statistic, p_value = stats.ttest_ind(scores_a, scores_b, equal_var=False)

# --- 3. Interpret Results ---
alpha = 0.05
mean_a, mean_b = np.mean(scores_a), np.mean(scores_b)
improvement = (mean_b - mean_a) / mean_a

print(f"📊 A/B Test Analysis Results:")
print(f"  - Variant A Mean Helpfulness: {mean_a:.3f}")
print(f"  - Variant B Mean Helpfulness: {mean_b:.3f}")
print(f"  - Relative Improvement: {improvement:.2%}")
print(f"  - P-value: {p_value:.4f}")

# The core decision logic: first check for significance, then check the direction.
if p_value < alpha:
    if improvement > 0:
        print("✅ Result is statistically significant. Variant B is a winner.")
    else:
        print("🚨 Result is statistically significant, but Variant B is worse (a regression).")
else:
    print(" inconclusive. Cannot conclude a winner.")

Example output:

📊 A/B Test Analysis Results:
  - Variant A Mean Helpfulness: 4.017
  - Variant B Mean Helpfulness: 4.084
  - Relative Improvement: 1.66%
  - P-value: 0.0033
✅ Result is statistically significant. Variant B is a winner.

A statistically significant result doesn't automatically mean you should deploy a change. A/B test results often involve trade-offs between your goal metrics and your guardrail metrics. Use a decision matrix to evaluate them systematically.

1. Safety first: Any significant safety regression is grounds for immediate rejection.
2. Quality threshold: Did the change meet the MDE for your goal metric?
3. Guardrail metrics: Did the change cause an unacceptable regression in cost or latency?

Offline evaluation and online A/B testing should form a continuous improvement loop. Use offline evaluations on a static dataset to identify promising candidates for testing. Then, use A/B testing to validate whether those improvements translate to better outcomes with real users. Finally, feed insights and challenging examples from your A/B tests back into your offline evaluation datasets to make them more realistic and robust over time.

Follow these principles to ensure your A/B tests are reliable, insightful, and drive meaningful improvements.

• Test One Variable at a Time: Isolate a single change (e.g., the prompt or the model) to ensure you can attribute the outcome to a specific cause.
• Segment Your Results: Analyze how variants perform for different user groups (e.g., new vs. returning) or query types, as a change might improve performance for one segment but harm it for another. Be cautious: segments can be too small to yield statistically significant results, and testing many segments increases the risk of false positives.
• Run an A/A Test First: Before launching an A/B test, run a test where both variants are identical. If you see a statistically significant difference, your testing infrastructure is flawed and must be fixed.

• Don't Stop Tests Early: It is tempting to stop a test the moment a variant appears to be winning. This is "peeking" and often leads to false conclusions. Adhere to your pre-calculated sample size.
• Don't Rely Only on Averages: An average score can hide critical regressions in the distribution. Always check percentile metrics (e.g., P95 latency) as well.
• Don't Forget Guardrail Metrics: A variant might improve your goal metric but cause a regression in a critical guardrail like safety or cost. Always measure for unintended side effects.

Systematic A/B testing transforms Llama application development from guesswork into a data-driven optimization engine. To continue building on this framework:

1. Master Offline Evaluation: Before running online tests, build a robust offline evaluation suite to identify the most promising candidates to test.
2. Start with Prompt Optimization: Focus your first A/B tests on your most critical user-facing prompts.
3. Automate Your Llama-as-Judge Pipeline: Invest in automating your evaluation pipeline to scale your quality analysis.
4. Establish a Feedback Loop: Use insights from your A/B tests to continuously improve your offline evaluation datasets, making them better predictors of real-world performance.