Skip to main content

Overview

The Human Review system allows you to queue test runs for manual evaluation by human reviewers. This provides a way to validate model outputs beyond automated checks.

Accessing a Run’s Review

Once a run has been queued for human review (via the web app), you can access the review through the SDK:

Working with Reviews

Once a run is queued, you can access the review through the run.review cached property:

Key Properties

  • id - Same as run.review_id
  • status - Current review status (Pending, Archived, or Completed)
  • pass_rate_human_eval - Pass rate across all human reviews
  • agreement_rate_human_eval - Agreement rate between human reviewers
  • agreement_rate_auto_eval - Agreement rate between human and auto evaluations
  • flagged_rate - Rate of flagged test results
  • created_at - When the review queue was created
  • completed_time - When the review was completed (if applicable)
  • test_results - List of completed test results (cached property, requires await)

Working with Test Results

Access individual test result reviews to get detailed feedback:
You can also access the first review for a test result using the convenience property:

Test Result Properties

  • reviewed_by - List of reviewer email addresses
  • reviews - List of all reviews for this test
  • review - Convenience property returning the first review
  • test - The original test being reviewed
  • check_results - Auto-evaluated check results

Test Review Properties

  • feedback - Optional reviewer feedback
  • completed_by - Reviewer who completed this review
  • completed_at / started_at - Timestamps
  • reviewed_check_results - Human validation of auto-evaluations
  • custom_review_values - Custom template review data (when using custom review templates)