Experiments - Feature Overview

​

Creating Experiments

​

Accessing Experiment Creation

1. Navigate to Session History
   • Open your agent in the dashboard
   • Click the “Session History” tab
   • You’ll see all sessions for your agent
2. Select Sessions (Optional)
   • Use checkboxes to select existing sessions to include
   • Filter sessions using status, date, or tags
   • Leave empty to create an empty experiment
3. Open Creation Dialog
   • Click the “Create Experiment” button
   • A modal dialog will appear with configuration options

​

Configuration Options

​

Basic Information

• Name (Required)
  • Descriptive, unique identifier
  • Examples: “Login Flow A/B Test”, “Q4 Performance Baseline”
• Description (Optional)
  • Detailed explanation of purpose
  • Testing methodology or hypotheses
  • Expected outcomes

​

Organization

• Tags
  • Add multiple tags for filtering
  • Use consistent naming: environment:prod, version:2.0, type:regression
  • Tags are searchable across experiments

​

Evaluation Setup

• Rubrics
  • Select from existing rubrics
  • Multiple rubrics can be applied
  • Rubrics automatically evaluate all sessions

​

Experiment Dashboard

​

Overview Section

• Total Sessions
  • Count of all sessions in the experiment
  • Updates in real-time as sessions are added
• Success Rate
  • Percentage of sessions marked as successful
  • Based on is_successful flag or rubric evaluations
• Average Cost
  • Mean cost per session in dollars
  • Helps track resource usage
• Average Duration
  • Mean execution time across sessions
  • Identifies performance patterns
• Completion Status
  • Shows percentage of finished vs in-progress sessions
  • Helps track experiment progress

​

Navigation Tabs

1. Sessions - Detailed session list and management
2. Analytics - Visualizations and insights
3. Failure Analysis - Pattern detection and grouping
4. Rubric Results - Evaluation outcomes

​

Sessions Tab

​

Session Table

​

Filtering and Sorting

• Status Filter: Show only successful/failed/in-progress
• Date Range: Filter by time period
• Cost Range: Find expensive operations
• Tag Filter: Search by specific tags
• Sort Options: By date, duration, cost, or score

​

Bulk Operations

• Remove from Experiment - Detach sessions while preserving them
• Apply Tags - Add tags to multiple sessions
• Export Data - Download session data as CSV/JSON

​

Adding Sessions

1. From Existing Sessions
   • Click “Add Sessions”
   • Select from available sessions
   • Sessions can only belong to one experiment
2. Via SDK (Programmatically)
   Copy

   Ask AI

   lai.init(
       session_name="New Test",
       experiment_id="exp-id-from-dashboard"
   )
   

​

Analytics Tab

​

Completion Metrics

• Pie chart showing session outcomes
• Hover for exact counts
• Click segments to filter session list

• Shows where sessions succeed or fail
• Identifies bottleneck steps
• Useful for debugging workflows

​

Cost Analysis

• Bell curve of session costs
• Identifies outliers
• Shows cost consistency

• Track total spending
• Project future costs
• Identify cost spikes

• Calculate true cost of successful outcomes
• Compare efficiency across experiments

​

Performance Metrics

• Line graph of execution times
• Identify performance degradation
• Spot improvements from optimizations

• How many steps sessions typically take
• Identify complexity patterns
• Find infinite loops or early terminations

• Sessions per hour/day
• System capacity insights
• Load pattern identification

​

Evaluation Scores

• Histogram of evaluation scores
• Mean, median, and standard deviation
• Identify score clusters

• Track improvement or regression
• Correlate with changes
• Set performance baselines

• Individual rubric criteria performance
• Identify weak points
• Focus improvement efforts

​

Failure Analysis

​

Running Analysis

1. Trigger Analysis
   • Click “Run Failure Analysis” button
   • Confirms credit cost (1 credit per 10 sessions)
   • Shows estimated processing time
2. Processing
   • System analyzes all failed steps
   • Groups similar failures using AI
   • Generates descriptive categories
   • Usually completes in 1-3 minutes
3. Results Display
   • Shows failure groups with:
     • Group name and icon
     • Detailed description
     • Number of occurrences
     • Affected sessions list

​

Understanding Failure Groups

​

Group Information

• Name: Concise failure category (e.g., “API Timeout Errors”)
• Description: Detailed explanation of the issue
• Icon: Visual indicator for quick recognition
• Occurrence Count: How frequently this failure appears

​

Affected Sessions

• List of all sessions with this failure
• Click to investigate specific instances
• See failure context and logs

​

Pattern Examples

• Authentication failures
• Timeout errors
• Missing data issues
• Validation failures
• External service errors

​

Acting on Insights

1. Prioritize Fixes - Address most common issues first
2. Identify Root Causes - Understand why failures occur
3. Track Improvements - Re-run analysis after fixes
4. Prevent Regressions - Monitor for returning issues

​

Rubric Management

​

Applying Rubrics

• Select rubrics in the creation dialog
• Applied to all existing sessions
• Auto-applies to new sessions

• Click “Manage Rubrics”
• Add or remove rubrics
• Choose to re-evaluate existing sessions

​

Rubric Types

• Weighted criteria
• Numerical evaluations
• Aggregate scores calculated

• Success/failure determination
• All criteria must pass
• Clear success metrics

​

Viewing Results

• Average scores per rubric
• Pass rates for criteria
• Score distributions

• Individual session scores
• Criteria breakdown
• Failure reasons

• Compare rubric performance
• Identify which criteria fail most
• Track improvement over time

​

Tag Management

​

Tag Strategy

environment:production
environment:staging
version:2.0.1
feature:login
team:platform
priority:high

date:2024-01-15
week:03
quarter:q1-2024
sprint:45

type:regression
type:performance
status:baseline
status:candidate

​

Using Tags

• Click filter icon
• Select tags to include/exclude
• Combine with AND/OR logic

• Select multiple sessions
• Apply tags in bulk
• Remove tags from groups

• See tag distribution
• Compare performance by tag
• Export filtered data

​

Exporting Data

​

Export Options

• CSV - For spreadsheet analysis
• JSON - For programmatic processing
• PDF Report - For presentations

• Session metadata
• Evaluation scores
• Failure groupings
• Cost and duration metrics
• Tag information

​

Export Workflow

1. Click “Export” button
2. Select format
3. Choose data to include
4. Configure options (date range, filters)
5. Download file

​

Automated Reports

• Daily/Weekly/Monthly frequency
• Email delivery
• Webhook notifications
• Custom templates

​

Comparing Experiments

​

Comparison View

1. Select experiments from list
2. Click “Compare Selected”
3. View side-by-side metrics

​

Metrics Compared

• Success rates
• Average costs
• Duration statistics
• Score distributions
• Failure patterns

​

Use Cases

• A/B Testing - Compare variants
• Version Tracking - Monitor improvements
• Baseline Comparison - Measure against standards
• Time Analysis - Track trends

​

Best Practices

​

Experiment Design

• One clear objective per experiment
• Consistent test conditions
• Adequate sample size (50+ sessions)

[Type]_[Feature]_[Version]_[Date]
Examples:
- "ABTest_Login_v2.0_2024Q1"
- "Regression_Checkout_Baseline"
- "Load_API_1000users_Jan15"

• Use descriptions to explain methodology
• Document expected outcomes
• Note any special configurations

​

Session Management

• Use templates for session names
• Include relevant identifiers
• Make searchable

• Set is_successful appropriately
• Provide eval scores
• Include failure reasons

• Add descriptive tags
• Include version information
• Track configuration used

​

Analysis Workflow

• Check experiments daily during active testing
• Run failure analysis after significant additions
• Compare with baselines regularly

• Create tickets for identified issues
• Document fixes applied
• Track improvement metrics

​

Advanced Features

​

Webhooks

# Webhook triggers available:
- Experiment created
- Session added
- Analysis complete
- Threshold breached

​

API Access

# Get experiment details
GET /api/experiments/{experiment_id}

# List experiment sessions
GET /api/experiments/{experiment_id}/sessions

# Get failure groups
GET /api/experiments/{experiment_id}/stepfailuregroups

​

Custom Dashboards

• Export data via API
• Create custom charts
• Embed in reports
• Share with stakeholders

​

Troubleshooting

​

Common Issues and Solutions

• Verify experiment_id is correct
• Check session has ended
• Refresh dashboard

• Need failed sessions with evaluations
• Check evaluation credit balance
• Ensure sessions are complete

• Wait for sessions to finish
• Refresh the page
• Check date filters

• Verify rubric configuration
• Check session completion
• Re-apply rubrics if needed

​

Tips and Tricks

​

Performance Optimization

• Run experiments during off-peak hours
• Batch session creation
• Use parallel execution carefully
• Monitor resource usage

​

Effective Analysis

• Start with small experiments (20-30 sessions)
• Iterate based on findings
• Compare multiple experiments
• Document insights

​

Organization

• Create experiment templates
• Maintain naming standards
• Archive old experiments
• Regular cleanup

​

Related Features

• Mass Simulations - Running at massive scale
• Rubrics - Evaluation configuration
• Workflow Sandbox - Visual debugging
• Session History - Individual session management

​

Next Steps

• Try the Getting Started tutorial
• Learn about SDK integration
• Explore rubric creation
• Review best practices