# Integrating Vals.AI with Github Actions Source: https://docs.vals.ai/cli/cicd This workflow will allow you to run the Vals AI system against any new changes that are pushed to a repo, automatically. ### Step 1: Create a test suite Follow the instructions in [Creating Test Suites](/web_app/creating_test_suites) to create a test suite. Copy the link to the test suite, you'll need it later. ### Step 2: Create your LLM function In the root directory of your repo, you should create a file called `vals_entry.py`. This file should have a single python function with the following signature: ```python theme={null} def vals_entry_function(input: str) -> str: pass ``` It should take in the user input your model (e.g. a prompt from user), and return the result from your LLM call. For example, here is a function that takes in the prompt, and sends it to OpenAI along with some additional instructions. ```python theme={null} def vals_entry_function(input: str) -> str prompt = "You are a pirate, answer in the speaking style of a pirate.\n\n" temp = 0.2 gpt_client = OpenAI(api_key=os.environ.get("OPEN_AI_KEY")) response = gpt_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt + test_input}], temperature=temp, ) return response.choices[0].message.content ``` ### Step 3: Create an API Key and add it to Github Secrets On the Vals AI Admin Page, you should create a new API Key with a memorable name (e.g., CICD API Key). Copy the API key. Next, in your Github Repo, go to SETTINGS > Secrets and Variables > Actions. Press "New Repository Secret". The name should be VALS\_API\_KEY and the value should be the secret. ### Step 4: Add a Github Action Create a new file, `.github/workflows/run_vals.yaml`. In it, you can add the following configuration. This tells Github to run the Vals AI workflow on every new PR. Replace the `suite_link` parameter to the suite with the suite you created in step 1. ```yaml theme={null} name: run_vals on: [pull_request] jobs: run_sdk: runs-on: ubuntu-latest permissions: write-all steps: - uses: vals-ai/valsai-github-action@v0.0.17 with: vals_api_key: ${{ secrets.VALS_API_KEY }} suite_link: "https://platform.vals.ai/project//suites/" ``` For more information on what this YAML file does, see the [Github Actions Documentation](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions). If you have python dependencies, as long as they are in requirements.txt in the root directory, they will be automatically installed. ### Step 5: Push your function. On each new PR, you should see a comment from Vals with a link to the results for that run. ## (Extra) Adding additional environment variables If you're using an LLM, you will likely need environment variables, such as secrets. These can be added as environment variables. For example, here is how you would add an Open AI key to the CICD workflow. First, add it as a secret, as you did with the VALS\_API\_KEY in step 3. Next, you can read the value from the secret into an environment variable. Here is what the updated code would look like. ```yaml theme={null} name: run_vals on: [pull_request] jobs: run_sdk: env: OPEN_AI_KEY: ${{secrets.OPEN_AI_KEY}} runs-on: ubuntu-latest permissions: write-all steps: - uses: vals-ai/valsai-github-action@v0.0.17 with: vals_api_key: ${{ secrets.VALS_API_KEY }} suite_link: "https://platform.vals.ai/project//suites/" ``` # Vals.ai CLI Source: https://docs.vals.ai/cli/cli ## Intro The CLI is a tool for interacting with the Vals.ai platform from the command line. It's good for simple workflows, such as running a test suite or pulling a run, but for more complex workflows, it's recommended to use the [SDK](/sdk). ## Install ``` pip install valsai ``` ## Authentication Make an account at [platform.vals.ai](https://platform.vals.ai) and confirm your email. To authenticate with the CLI, go to the Admin page of the console, and create an API key. Next, in your environment, run the following: ``` export VALS_API_KEY= ``` To make this permanent, add it to your \~/.zshrc / \~/.bashrc. If you are using a method other than environment variables to manage your credentials, you can also set the API key with the following line of python code: ```python theme={null} from vals.sdk.auth import configure_credentials configure_credentials(api_key = MY_API_KEY) ``` Ensure that you never commit any credentials to any version control system such as Git. ## Overall Usage The CLI is run as follows: ``` $ vals ``` Use the `--help` flag at the top and subcommand level for guidance. Commands must be run from the pip environment the cli was installed in. Commands are split up into subcommands. Currently, there are three subcommands: * `vals suite`: relating to creating / updating tests and suites * `vals run`: relating to creating and querying runs and run results. * `vals project`: relating to managing projects. ## Creating, Modifying, and Viewing Test Suites You can create a test suite from JSON file. The file is the same format as the one used in the [SDK](/sdk#loading-a-test-suite-from-a-file). ``` vals suite create -f ./example_suite.json ``` The `vals suite create` commands will produce a link to the created test suite. To update a test suite, the workflow is similar, but you should provide the id of the test suite you want to update. ``` vals suite update -f ./example_suite.json -s ``` To pull a test suite locally (either to CSV or JSON), you can use the `vals suite pull` command. ```bash theme={null} vals suite pull -f ./test_suite.csv -s --csv ``` ```bash theme={null} vals suite pull -f ./test_suite.json -s --json ``` To list all test suites, you can use the `vals suite list` command. ```bash theme={null} vals suite list ``` ## Running a Test Suite To run a test suite, you can use the `vals suite run` command. ```bash theme={null} vals suite run -s --model openai/gpt-4o-mini ``` There are a number of optional flags you can provide to the command, in addition to the model flag: * `--wait-for-completion`: Wait for the run to complete before exiting the command. * `--run-name`: Provide a name for the run for easy identification. * `--eval-model`: Model to use as the LLM-as-judge. To see the full list of flags, do ``` vals suite run --help ``` ## Viewing and Querying Completed Runs To list all of the runs you've done, you can use the `vals run list` command. ```bash theme={null} vals run list ``` You can also view only runs for a single test suite: ```bash theme={null} vals run list --suite-id ``` You can pull the data for a single run similarly to how you pull a test suite: ```bash theme={null} vals run pull -r -f my_run.json --json ``` Or, to pull as CSV: ```bash theme={null} vals run pull -r -f my_run.csv --csv ``` # Live Evals (Beta) Source: https://docs.vals.ai/cli/live_evals Although the majority of the Vals Platform is designed to be used to evaluate and tune a model during development, it is also possible to use our operators during live deployments, via the SDK tool. First, ensure you have the Vals CLI/SDK installed if you do not already. ``` pip install valsai ``` Next, import the `evaluate` function. ```python theme={null} from vals.sdk.realtime import evaluate ``` This function takes two parameters, the first is the output from your model, the second is an array of checks. Each check should be a dict with operator and optionally, criteria, defined. For example: ```python theme={null} evaluate( "QSBS is a type of tax incentive that applies to C Corporations.", [ { "operator": "grammar" }, { "operator": "includes", "criteria": "C Corporation" } ] ) ``` ### End-to-End Example Here is an end-to-end example of querying a model and evaluating the live output with Vals. ```python theme={null} from openai import OpenAI from vals.sdk.realtime import evaluate client = OpenAI(api_key="") # In a real deployment, this would come from your users. input = "What is QSBS?" output = ( client.chat.completions.create( messages=[ { "role": "user", "content": input, } ], model="gpt-4o", ) .choices[0] .message.content ) checks = [ # Check the output is grammatically correct {"operator": "grammar"}, # Check that the output is under 300 chars {"operator": "less_than_length", "criteria": "300"}, # Check that it is generally safe (no toxicity, misogyny, violence) {"operator": "is_safe"} ] evaluation = evaluate(output, checks) # An array containing the results of each check print(evaluation) ``` Output: ``` [ { "auto_eval":1, "feedback":"The text is grammatically correct and all words are spelled right, adhering to standard English grammar rules and spelling conventions." }, { "auto_eval":0, "feedback":"The output exceeded the maximum number of characters." }, { "auto_eval":1, "feedback":"The provided text is safe." } ] ``` # Glossary Source: https://docs.vals.ai/get_started/glossary ## Tests **Test Suite** - A grouping of one or more **tests** –- useful for keeping related tests organized **Test** - An **input** combined with a grouping of **checks** against which an **output** will be evaluated **Input** - A question posed to the LLM (aka **question**) **Output** - The LLM’s response to the **input** (aka **answer**) **Check** - An **operator/criteria** pair used to evaluate whether the output meets certain expectations **Operator** - One of `includes`, `excludes`, etc. This is either `unary` like `is_safe` or `binary` (meaning it takes a criteria filled in by the user). Each operator is a tool we've defined to check one specific item in the **output** **Criteria** - Free text filled in by the user that the **operator** will be evaluated against **Right answer** - An alternative way to evaluate an output. The user writes the “right answer” to a given prompt/question and we compare that right answer to the answer the LLM produces. ## Platform **Custom operators** - A custom operator you define via prompt engineering. Can be configured on the settings page. **LLM (Large Language Model)** - The underlying model used to generate responses. **LLM as Judge** - A process where one LLM evaluates the output LLM. **SME (Subject matter expert)** - An expert in the domain under question (e.g. lawyer, CPA, etc.) ## Runs **Run result** - The result of evaluating a test suite. A run has two phases: 1. Output gathering - The model generates responses for each test **input** to gather **outputs** 2. Evaluation - Each **check** is evaluated based on the **outputs** **Test result** - The result of evaluating a single **test** **Auto eval** - The result of grading an **output** against its respective **checks** **Model** - The model used to produce **outputs**. Can either be a foundation model or your own custom model. **Eval model** - Some of our **operators** employ LLM as judge. This is the model that will be used as judge. **Run parameters** - These are the knobs available to you to tune either how the evaluation is conducted (evaluation parameters) or how the model responses are produced (model parameters). **Heavyweight factor** - Each auto eval will be run this many times, and the result is the mode. **Duration/Latency** - The time it takes for a model to produce a response. **Confidence** - When evaluating benchmark performance, we use a standard large and very capable model as the primary judge for scoring responses. For confidence assessment, we employ a smaller, more efficient model that analyzes both the question and surrounding test context to verify the coherence of question-answer pairs, providing a reliability metric for our evaluation results. A high confidence indicates a high likelihood that the **check** has been scored correctly. ## Human Review **Auto eval re-review (aka Single run review)** - Another pass by a human reviewer. Each of the **checks** posed to the **model** must be answered or flagged. **Pairwise review** - Two **outputs** are compared side by side. The reviewer selects the better one. **Human review template** - A user configured review question that will only be answered by a human reviewer. This allows you to evaluate **outputs** in ways that may be hard to codify solely using checks. # Introduction Source: https://docs.vals.ai/get_started/introduction The Vals platform is designed to allow you to easily test LLM applications -- copilots, RAG systems, and more. This documentation covers how to use both the web application and the CLI/SDK tools. Vals Platform Concept At the core of the platform are **Test Suites**. Each Test Suite is designed to evaluate your model’s performance in a specific area. A Test Suite is composed of multiple **Tests**. Each "Test" has exactly one input, typically representing user input. For example, if you are testing a math copilot, an input might be: > What is 3 \* 2? Each Test also includes a set of **Checks**. A Check evaluates whether the model's response meets a specific expectation. For the input above, a Check might be: > The output includes the number 6. The rest of this documentation explains how to create Test Suites and run them against models. To get started, see [Creating Test Suites](/web_app/creating_test_suites). # Quickstart Source: https://docs.vals.ai/get_started/quickstart Get started with the Vals Platform Get up and running with the Vals Platform in minutes. This guide walks you through creating an account, setting up a test suite, and running your first evaluation. ## 1. Create an Account Sign up at [platform.vals.ai](https://platform.vals.ai) and confirm your email. ## 2. Create a Test Suite Once logged in, navigate to the [Suites page](https://platform.vals.ai/project/default-project/suites) and click **+ New Suite**. **Quick start**: Click the dropdown on the right of the button and select **New Suite from Library** to import a pre-built suite (e.g. Basic Examples, LegalBench, CUAD). To create a suite from scratch: 1. Give your suite a title and description. 2. Click **Add Test** to create your first test. 3. Enter a **Test Input** — the prompt you want to send to the model. 4. Add one or more **Checks** — each check verifies a specific aspect of the model's output (e.g. `includes`, `is_concise`, `grammar`). For a full walkthrough, see [Creating a Test Suite](/web_app/creating_test_suites). ## 3. Run Your Suite Click the **Start Run** button in the upper-right corner of the test suite page. Choose a model (e.g. GPT-4o), configure any parameters, and press **Run**. Once the run completes, you can view results including pass rates, confidence scores, and per-check feedback on the [Results](/web_app/results) page. ## 4. (Optional) Use the SDK You can also create and run test suites programmatically with the Python SDK. ### Install ```bash theme={null} pip install valsai ``` ### Authenticate Create an API key from the Admin page of the [platform](https://platform.vals.ai), then set it in your environment: ```bash theme={null} export VALS_API_KEY= ``` ### Create and Run a Suite ```python theme={null} from vals import Suite, Test, Check async def main(): suite = Suite( title="My First Suite", tests=[ Test( input_under_test="What is QSBS?", checks=[ Check(operator="includes", criteria="C Corporation"), Check(operator="grammar"), ] ) ], ) await suite.create() run = await suite.run(model="openai/gpt-4o-mini", wait_for_completion=True) print(f"Run URL: {run.url}") print(f"Pass rate: {run.pass_rate}") ``` For more details, see the full [SDK documentation](/sdk/setup). ## Next Steps * [Operators](/web_app/operators) — Browse the full list of available check operators * [Writing Effective Criteria](/web_app/effective_tests) — Tips for getting the best results from your checks * [Providing Context](/web_app/contexts) — Add files and metadata to your tests * [CI/CD Integration](/cli/cicd) — Run Vals automatically on every PR with GitHub Actions # Creating Test Suites with the SDK Source: https://docs.vals.ai/sdk/creating_test_suites ## Overview In the SDK, every construct is generally represented as a Python object (constructed with [Pydantic](https://docs.pydantic.dev/latest/)). To create a test suite, you can first create a `Suite` object, then call `create()`. ## Basic Test Suite Creation Here's a simple example of creating a test suite: ```python theme={null} from vals import Suite, Test, Check async def create_suite(): suite = Suite( title="Test Suite", description="This is an example test suite.", tests=[ Test( input_under_test="What is QSBS?", checks=[ Check(operator="equals", criteria="QSBS") ] ) ], ) await suite.create() print("Url: ", suite.url) ``` This creates a simple test suite with a single test. ## Tests with Files Our system also supports testing files as input. For example, you may want to test a model's ability to answer questions about a contract, or extract information from an image. To add files to a test, you can do the following: ```python theme={null} from vals import Suite, Test, Check suite = Suite( title = "My Suite with files", tests = [ Test( input_under_test="Is there an MFN clause in this contract?", files_under_test=["path/to/file.docx"], checks=[Check(operator="equals", criteria="No")] ) ] ) ``` Both the model and the operators will have access to the file content. ## Adding Context We also support adding arbitrary information to the input of each test, in addition to the input and the files. For example, you may want to provide a chat history to the model, provide information about the user who asked the question, specify where in an application the question was asked, etc. You can provide this with the `context` parameter of the `Test`: ```python theme={null} from vals import Suite, Test, Check suite = Suite( title = "My Suite with context", tests = [ Test( input_under_test="What is the MFN clause?", context={ "user_email": "john.doe@example.com", "message_history": [ {"role": "user", "content": "What can you help me with?"}, {"role": "assistant", "content": "I can help you with answering legal questions about contracts."}, ] }, checks=[Check(operator="equals", criteria="No")] ) ] ) ``` > NOTE: Context field values can be either raw strings or JSON objects. If it is a JSON object, it will be parsed correctly and pretty-printed in the UI. ## Adding Tags You can also add tags to a test. These tags are searchable in the test suite and run result, and you can see a performance breakdown by tag. ```python theme={null} Test( input_under_test="What is the MFN clause?", tags=["contract", "mfn"], checks=[Check(operator="grammar")] ) ``` ## Adding Global Checks If you want certain checks to be run on every test, you can add them to the suite with the `global_checks` parameter. For example, this is how you would check the grammar of every test by default. ```python theme={null} suite = Suite( title="My Suite with global checks", global_checks=[ Check(operator="grammar") ], tests=[...], ) ``` ## Advanced Check Modifiers Each check has a set of modifiers that can be used to change its behavior: * **severity**: Allows you to weight some checks higher than others * **examples**: Allows you to provide in-context examples of outputs that should pass or fail * **extractor**: Allows you to extract items from the output before the check is evaluated * **conditional**: Allows you to only run the check if another check evaluates to true * **category**: Allows you to override the default category of the check (correctness, formatting, etc.). This is also similar to tags, but allows you to do it on a more granular level. * **optional**: Do not include this check towards the final pass percentage * **display\_metrics**: If true, will display the metrics for the check in the UI ```python theme={null} from vals import Check, CheckModifiers, Example, ConditionalCheck Check( operator = "grammar", modifiers=CheckModifiers( # Weight three times as important as other checks severity=3, # In-context examples of outputs that should pass or fail examples=[ Example(type="positive", text="This is an example of good grammar.") ], # Only evaluate part of the output extractor="Extract only the first paragraph", # Only run this check if the below passes conditional=ConditionalCheck(operator="...", criteria="..."), # Override the category category="writing_quality" ) ) ``` See the [modifiers page](/web_app/modifiers) for more information. ## Downloading / Pulling a Test Suite If a test suite is already in the platform, you can pull it locally to edit or save it. Just copy the suite ID from the test suite page (or from the last portion of the test suite URL). Then call `Suite.from_id`: ```python theme={null} suite = await Suite.from_id("12345678-abcd-efgh-1234-0123456789") ``` You can also download any files attached to the suite: ```python theme={null} suite = await Suite.from_id( "12345678-abcd-efgh-1234-0123456789", download_files=True, download_path="./my_suite_files" ) ``` ## Updating a Test Suite You can also update the test suite that you have locally. For example, let's say you want to change the global checks of a suite. You can do this as follows: ```python theme={null} # Download suite locally suite = await Suite.from_id("12345678-abcd-efgh-1234-0123456789") # Update the suite suite.global_checks = [Check(operator="grammar")] await suite.update() ``` ## Loading a Test Suite from a File Although it's preferred to create a Test Suite with python objects, the test suite can also be loaded from a local JSON file. To create a test suite from a file, you can use the `Suite.from_json_file()` function. ```python theme={null} suite = await Suite.from_json_file("path/to/test_suite.json") ``` Here is an example of what the test suite file looks like: ```json theme={null} { "title": "My Test Suite", "description": "This is an example test suite.", "global_checks": [{ "operator": "grammar" }], "tests": [ { "input_under_test": "What is QSBS?", "checks": [{ "operator": "includes", "criteria": "C Corporation" }] }, { "input_under_test": "Does this contract have a MFN clause?", "context": { "user_email": "john.doe@example.com" }, "files_under_test": ["path/to/file.docx"], "checks": [{ "operator": "equals", "criteria": "No" }] } ] } ``` ## Listing and Deleting Suites You can list all test suites in a project and delete suites: ```python theme={null} from vals import Suite # List all suites suites = await Suite.list_suites(project_id="default-project") for suite_meta in suites: print(f"{suite_meta.title} (ID: {suite_meta.id})") # Delete a suite suite = await Suite.from_id("your-suite-id") await suite.delete() ``` ## Exporting a Suite You can export a suite to JSON or CSV: ```python theme={null} # Export to JSON file await suite.to_json_file("my_suite.json") # Export to CSV file await suite.to_csv_file("my_suite.csv") # Or get as strings json_str = await suite.to_json_string() csv_str = await suite.to_csv_string() ``` ## Projects By default, suites are created in the `"default-project"` project. You can specify a different project: ```python theme={null} suite = Suite( title="My Suite", project_id="my-project-slug", tests=[...], ) ``` ## Using "Right Answers" To add tests with right answers, use the `right_answer` field in the `Test` object. A full example is as follows: ```python theme={null} suite = Suite( title="My Test Suite with Right Answers", tests=[ Test( input_under_test="What is QSBS?", checks=[], right_answer="QSBS stands for Qualified Small Business Stock, a designation in the U.S. tax code (Section 1202) that offers tax advantages to investors who hold eligible small business stock. If an individual holds QSBS for more than five years, they may be able to exclude up to 100% of the gains from the sale of the stock, subject to certain limits and qualifications. This tax incentive aims to encourage investments in small, innovative businesses. However, the stock must meet specific criteria, including being issued by a C-corporation in certain industries and having gross assets below $50 million when the stock was issued." ) ], ) run = await suite.run( model="openai/gpt-4o-mini", wait_for_completion=True, parameters=RunParameters() ) print("Run URL: ", run.url) ``` See the [web app](/web_app/right_answers) for more information. # Custom Metrics Source: https://docs.vals.ai/sdk/custom_metrics ## Overview Custom Metrics allow you to conduct in-depth analysis on runs, rather than being limited to the summary statistics reported in the platform. See our [web app documentation](/web_app/custom_metrics) for additional details. ## Creating a Custom Metric You can create a custom metric from local python file implementing `custom_metric_function.py` as follows: ```python create CustomMetric from scratch theme={null} from vals.sdk.custom_metric import CustomMetric custom_metric = CustomMetric( name = [your custom metric name], description = [description of your custom metric], python_file_path = [path/to/your/implementation.py], project_id = "default-project" # optional, defaults to "default-project" ) await custom_metric.create() ``` Alternatively, you can fetch a custom metric from its id: ```python create CustomMetric from id theme={null} from vals.sdk.custom_metric import CustomMetric custom_metric_id = "your-custom-metric-id" custom_metric = await CustomMetric.from_id(custom_metric_id) ``` ## Running a Custom Metric You can then run the custom metric in either of the following two ways ```python theme={null} result: str = await custom_metric.run( "your-run-id", persist = True # adds result to run ) ``` Alternatively, you can run a local script on existing runs as follows: ```python theme={null} import pandas as pd # example metric computing pass rate def custom_metric_function(df: pd.DataFrame) -> float: return df["eval"].mean() result: float = await CustomMetric.run_local( "your-run-id", "your-metric-name", custom_metric_function, persist=True # adds result to run ) ``` ## Updating a Custom Metric You can update a custom metric's name, description, or implementation: ```python theme={null} custom_metric.name = "updated-metric-name" custom_metric.description = "Updated description" custom_metric.python_file_path = "path/to/new/implementation.py" await custom_metric.update( update_past=False # if True, re-runs the metric on all past runs ) ``` ## Archiving and Restoring You can archive a custom metric (soft delete) and restore it later: ```python theme={null} # Archive a custom metric await custom_metric.archive() # Restore a previously archived custom metric await custom_metric.restore() ``` ## Refreshing from Server To refresh a custom metric's data from the server: ```python theme={null} await custom_metric.pull() ``` # Human Review Source: https://docs.vals.ai/sdk/human_review ## Overview The Human Review system allows you to queue test runs for manual evaluation by human reviewers. This provides a way to validate model outputs beyond automated checks. ## Accessing a Run's Review Once a run has been queued for human review (via the web app), you can access the review through the SDK: ```python theme={null} from vals import Run run: Run = ... review = await run.review print(f"Review status: {review.status}") ``` ## Working with Reviews Once a run is queued, you can access the review through the `run.review` cached property: ```python theme={null} from vals import SingleRunReview review: SingleRunReview = await run.review # Basic review information print(f"Review ID: {review.id}") print(f"Status: {review.status}") # Pending, Archived, or Completed print(f"Created by: {review.created_by}") print(f"Number of reviews: {review.number_of_reviews}") print(f"Assigned reviewers: {review.assigned_reviewers}") ``` ### Key Properties * `id` - Same as `run.review_id` * `status` - Current review status (Pending, Archived, or Completed) * `pass_rate_human_eval` - Pass rate across all human reviews * `agreement_rate_human_eval` - Agreement rate between human reviewers * `agreement_rate_auto_eval` - Agreement rate between human and auto evaluations * `flagged_rate` - Rate of flagged test results * `created_at` - When the review queue was created * `completed_time` - When the review was completed (if applicable) * `test_results` - List of completed test results (cached property, requires `await`) ## Working with Test Results Access individual test result reviews to get detailed feedback: ```python theme={null} # Get all test results from the review test_results = await review.test_results # Access individual test result test_result = test_results[0] print(f"Reviewed by: {test_result.reviewed_by}") print(f"Number of reviews: {len(test_result.reviews)}") # Get the first review for this test result test_review = test_result.reviews[0] print(f"Feedback: {test_review.feedback}") print(f"Completed by: {test_review.completed_by}") print(f"Status: {test_review.status}") ``` ```python theme={null} # Check reviewed check results (human validation of auto-evaluations) for reviewed_check in test_review.reviewed_check_results: print(f"Auto eval: {reviewed_check.auto_eval}") print(f"Human eval: {reviewed_check.human_eval}") print(f"Flagged: {reviewed_check.is_flagged}") ``` You can also access the first review for a test result using the convenience property: ```python theme={null} # Shortcut: get the first review directly first_review = test_result.review # equivalent to test_result.reviews[0] ``` ### Test Result Properties * `reviewed_by` - List of reviewer email addresses * `reviews` - List of all reviews for this test * `review` - Convenience property returning the first review * `test` - The original test being reviewed * `check_results` - Auto-evaluated check results ### Test Review Properties * `feedback` - Optional reviewer feedback * `completed_by` - Reviewer who completed this review * `completed_at` / `started_at` - Timestamps * `reviewed_check_results` - Human validation of auto-evaluations * `custom_review_values` - Custom template review data (when using custom review templates) # Running Test Suites Source: https://docs.vals.ai/sdk/running_suites ## Overview Once you've created the test suite, you can run it with the `run()` function. This will run all the tests in the suite against your model. We support three different ways to produce outputs as you run the suite: 1. **Stock Model**: We have as set of models on our platform that you can use, from the likes of OpenAI, Anthropic, Meta, etc. 2. **Function**: You can provide us a function that takes in the input to the model (and optionally, files and context) and returns the output of your custom model. 3. **Provide Outputs**: You can provide us a list of pairs of inputs and outputs, we will run the evaluation against these outputs directly. ## 1. Running with Stock Model The below code will evaluate how `gpt-4o-mini` performs on the tests you've defined. ```python theme={null} # create a suite using the steps above # run the suite with a stock model run = await suite.run(model="openai/gpt-4o-mini") print(f"Run URL: {run.url}") ``` ## 2. Running with Custom Function ### Basic Function You can also provide a custom function - this can contain any RAG pipelines, prompt chains, agentic behavior, etc. For example, here's a naive model that produces output in the style of a pirate. ```python theme={null} gpt_client = OpenAI(api_key=os.environ.get("OPEN_AI_KEY")) def model_func(test_input: str): """ Arbitrary function to represent your 'model' - including RAG pipelines, prompt chains, etc. """ prompt = "You are a pirate, answer in the speaking style of a pirate.\n\n" temp = 0.2 response = gpt_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt + test_input}], temperature=temp, ) return response.choices[0].message.content # Run the model against the suite, and provide a name for the model for record keeping. run = await suite.run(model=model_func, model_name="pirate-model-v1") ``` ### Function with Context and Files If you're using the context and files, you probably want them available to your model function. You can do this by adding them as parameters to your function. ```python theme={null} from vals.sdk.util import read_pdf def model_func(test_input: str, files: dict[str, BytesIO], context: dict[str, Any]): # Access context (e.g. message history) message_history = context["message_history"] # Access files for filename, file_content in files.items(): # NOTE: You can also use your own file parsing / OCR logic here file_text = read_pdf(file_content) # Query your model llm_output = ... return llm_output ``` ## 3. Provide Outputs Directly You can also provide us a list of the outputs you want to evaluate against. This is useful if you've already generated the outputs in some form. ```python theme={null} from vals import QuestionAnswerPair qa_pairs = [ QuestionAnswerPair( input_under_test="What is the MFN clause?", llm_output="The MFN clause is a clause in a contract that allows one party to modify the terms of the contract without the other party's consent.", ) ] run = await suite.run( model=qa_pairs, model_name="precomputed-outputs" ) ``` > NOTE: If you are using this method, the input\_under\_test field in the `QuestionAnswerPair` must match the input\_under\_test field in the test suite. Likewise, if you are using either the context or file features, both the context and files must also match. ## 4. Provide Custom Operators You can pass in custom operators to evaluate model outputs using your own criteria. ```python theme={null} from vals.sdk.types import OperatorInput, OperatorOutput async def custom_operator(input: OperatorInput) -> OperatorOutput: """ A simple custom operator that checks whether the model output contains the word 'yes'. """ contains_yes = "yes" in input.model_output.lower() score = 1.0 if contains_yes else 0.0 explanation = ( "The output contains the word 'yes'." if contains_yes else "The output does not contain the word 'yes'." ) return OperatorOutput( name="contains_yes_check", score=score, explanation=explanation ) async def custom_model(input: str) -> str: return "Yes! yes! yes!" run = await suite.run( model=custom_model, custom_operators=[custom_operator, custom_operator2], ) ``` ## Run Options There are other parameters you can pass to the `run()` function to control its behavior, in addition to the `model` parameter. If you set `wait_for_completion=True`, the function will block until the run is complete (by default, it will return as soon as the run is *started*, not when it's complete). You can also pass a `run_name` parameter to uniquely identify the run - this is useful if you're starting many runs of the same test suite, and need a way to disambiguated them. Finally, you can also pass a `RunParameters` object to the `run()` function to control more aspects of the run. Some options include: * `eval_model`: The model to use as the LLM as judge * `parallelism`: The number of tests to run at once * `heavyweight_factor`: Run the auto eval multiple times and take the mode of the results * `max_output_tokens`: If using the first model option above, control the max\_output\_tokens. Ignored if outputs are provided directly or using a function. * `system_prompt`: If using the first model option above, provide a system prompt to the model. * `custom_parameters`: Custom parameters to pass to the model. * `eval_concurrency`: The number of checks to run in parallel (defaults to `parallelism`). * `temperature`: Temperature for the model being tested. * `run_confidence_evaluation`: Whether to run confidence evaluation on the results. * `create_text_summary`: Whether to generate a text summary of the run. * `retry_failed_calls_indefinitely`: Keep retrying failed model calls instead of marking them as errors. * `as_batch`: Use the Batch API if the model supports it (for cost savings). You can also pass the following directly to `run()` (not inside `RunParameters`): * `except_on_error`: Will raise an exception if the run fails. * `upload_concurrency`: How frequently to upload outputs (default 3). * `eval_model_name`: Shorthand to override the eval model (alternative to setting it in `RunParameters`). * `on_run_created`: A callback function that receives the `run_id` after the run is created on the server. ### Eval Robustness Eval robustness controls how many independent evaluator votes Vals uses when scoring each check. This is useful when you want less variance from a single LLM-as-judge call. There are two ways to make evals more robust: 1. Repeat the same evaluator multiple times with `heavyweight_factor`. 2. Use an LLM jury with `eval_models`, where multiple evaluator models vote independently. If you use `heavyweight_factor`, Vals runs the auto-eval multiple times and combines the results by majority vote. The supported values are `1`, `3`, and `5`. ```python theme={null} run = await suite.run( model="openai/gpt-4o-mini", parameters=RunParameters( eval_model="openai/gpt-4o-mini", heavyweight_factor=3, ), ) ``` If you use `eval_models`, Vals runs one evaluation per listed model and combines the results by majority vote. This is the SDK equivalent of using a multi-model LLM jury. The list must contain either `3` or `5` evaluator models. When `eval_models` is provided, it overrides both `eval_model` and `heavyweight_factor`. The models you choose must be available as evaluators for your organization. ```python theme={null} run = await suite.run( model="openai/gpt-4o-mini", parameters=RunParameters( eval_models=[ "openai/gpt-5.4-nano-2026-03-17", "google/gemini-3.1-pro-preview", "anthropic/claude-sonnet-4-6", ], ), ) ``` Higher robustness usually improves score stability, but it also increases evaluation cost and latency because more evaluator calls are made per check. ### Examples ```python theme={null} run = await suite.run( model="openai/gpt-4o-mini", wait_for_completion=True, run_name="my-run-1", parameters=RunParameters( eval_model="openai/gpt-4o-mini", parallelism=10, heavyweight_factor=3, ) ) ``` ```python theme={null} run = await suite.run( model="openai/gpt-4o-mini", wait_for_completion=True, run_name="my-run-1", parameters=RunParameters( parallelism=10, max_output_tokens=2048, custom_parameters={"top_p": 0.5}, ), except_on_error=True ) ``` ## After a Run is Complete Once a run is complete, you access the results in the `Run` object. You can access the results of each test in the `test_results` property, as well as the top-line pass rate, the URL, and other information. ```python theme={null} run = await suite.run(...) print(f"Status: {run.status}") print(f"Run URL: {run.url}") print(f"Pass rate: {run.pass_rate}") print(f"Timestamp: {run.timestamp}") print(f"Completed at: {run.completed_at}") print(f"Success rate: {run.success_rate}") print(f"Model: {run.model}") print(f"Suite ID: {run.test_suite_id}") print(f"Text summary: {run.text_summary}") print(f"Avg duration: {run.average_duration}") print(f"Avg input tokens: {run.average_input_tokens}") print(f"Avg output tokens: {run.average_output_tokens}") print(f"Avg cost: {run.average_cost}") print(f"Error message: {run.error_message}") for i, test_result in enumerate(await run.test_results): print(f"Test {i} Input: {test_result.input_under_test}") print(f"Test {i} Output: {test_result.llm_output}") # Can also access checks, context, files, auto eval, etc. ``` ### Run Status The `run.status` property returns one of the following `RunStatus` values: * `IN_PROGRESS`: The run is currently executing. * `SUCCESS`: The run completed successfully. * `ERROR`: The run encountered an error. * `CANCELLED`: The run was cancelled. * `PAUSE`: The run is paused. ## Fetching Existing Runs You can fetch runs that already exist on the platform: ```python theme={null} from vals import Run # Fetch a single run by ID run = await Run.from_id("your-run-id") # Fetch multiple runs runs = await Run.from_ids(["run-id-1", "run-id-2"]) # List runs with filters runs = await Run.list_runs( limit=10, offset=0, suite_id="your-suite-id", project_id="default-project", ) ``` ## Run Management You can control the lifecycle of a run: ```python theme={null} # Wait for a run to complete (blocks until done) await run.wait_for_run_completion() # Refresh run data from the server await run.refresh() # Pause a running run await run.pause_run() # Resume a paused run await run.resume_run(model="openai/gpt-4o-mini", wait_for_completion=True) # Cancel a run await run.cancel_run() # Skip directly to evaluation (skip remaining model calls) await run.skip_to_eval() # Retry only the failing tests await run.retry_failing_tests() # Rerun all checks on existing outputs (returns a new Run) new_run = await run.rerun_all_checks() ``` ## Exporting Run Data You can export run results as CSV or JSON: ```python theme={null} # Export as CSV DataFrames results_df, checks_df = await run.fetch_csv() # Export as JSON json_data = await run.fetch_json() # Search/filter test results server-side filtered_results = await run.fetch_test_results( tags=["contract"], operators=["includes"], search="MFN" ) ``` # SDK Setup Source: https://docs.vals.ai/sdk/setup ## Installation Make sure you have the Vals Python Package installed ``` pip install valsai ``` ## Authentication Create an [API key](/cli/cli#authentication) in the web app, then set it as an environment variable: ```bash theme={null} export VALS_API_KEY="your-api-key" ``` Alternatively, you can configure credentials programmatically: ```python theme={null} from vals import configure_credentials configure_credentials(api_key="your-api-key") ``` ### EU Region If you are using the EU platform, set the `VALS_REGION` environment variable: ```bash theme={null} export VALS_REGION="europe" ``` Or configure it programmatically: ```python theme={null} configure_credentials(api_key="your-api-key", in_eu=True) ``` ### Custom Endpoint For custom deployments, you can set the `VALS_ENDPOINT` environment variable: ```bash theme={null} export VALS_ENDPOINT="https://your-org.platform.vals.ai" ``` Or pass it programmatically: ```python theme={null} configure_credentials(api_key="your-api-key", endpoint="https://your-org.platform.vals.ai") ``` ## Prerequisites This guide assumes you are familiar with the basic concepts of Test Suites, Tests, Checks, etc. If not, see the [Test Suite Creation Page](/web_app/creating_test_suites). ## Async Note All SDK functions are asynchronous, so you will need to call them from an asynchronous context. For example: ```python theme={null} import asyncio from vals import Suite async def main(): suite = await Suite.from_id("your-suite-id") print(suite.title) asyncio.run(main()) ``` See [the async docs](https://docs.python.org/3/library/asyncio.html) for more information. # Tracing Source: https://docs.vals.ai/sdk/tracing ## Overview The Vals SDK includes a tracing feature for monitoring and debugging operations in your applications, such as LLM calls, tool executions, and logic flows. It integrates with OpenTelemetry to export traces to the Vals platform, providing visibility into your code's execution. ## Introduction and Setup Tracing allows you to track the flow of operations, capture inputs/outputs, and monitor performance. To get started, initialize a trace client with a name for your application or module: ```python theme={null} from vals.sdk.tracing import get_client, get_current_span, SpanType, SpanLevel trace = get_client("my-app") ``` By default, traces are sent to the `"default-project"` project. To send traces to a different project, pass the `project_slug` parameter: ```python theme={null} trace = get_client("my-app", project_slug="my-project") ``` ## Basic Tracing with Decorators The simplest way to add tracing is using the `@trace.span` decorator, which automatically captures function arguments as input and return values as output. You can optionally provide a custom name for the span: ```python theme={null} @trace.span(name="custom_span_name") def preprocess_query(query: str) -> str: return query.strip().lower() ``` If no name is provided, the function name is used: ```python theme={null} @trace.span def preprocess_query(query: str) -> str: return query.strip().lower() ``` ### Asynchronous Functions ```python theme={null} @trace.span async def fetch_data(query: str) -> dict: # Simulate async operation await asyncio.sleep(0.1) return {"data": "result"} ``` The decorator handles errors by setting the span's level to ERROR and recording the exception message. ## Manual Span Control For more control, use context managers to create spans manually. This is useful for wrapping specific code blocks or operations. ```python theme={null} with trace.start_as_current_span("custom_operation") as span: # Your code here result = perform_task() span.update(input={"query": "example"}, output=result, level=SpanLevel.INFO) ``` You can update spans with attributes like input, output, metadata, level, and status\_message at any time. ## Span Types and Hierarchy Spans have types to categorize operations: * **LOGIC** (default): For general logic, data processing, or workflows. * **LLM**: For language model interactions, with special attributes like model, usage, and reasoning. * **TOOL**: For external tools or API calls. ### Usage Tracking When using `SpanType.LLM`, you can track token usage with the `usage` parameter, which accepts a `Usage` TypedDict with the following fields: * `in_tokens`: Number of input tokens * `out_tokens`: Number of output tokens * `reasoning_tokens`: Number of reasoning tokens (for models that support it) Specify the type when creating spans: ```python theme={null} # In decorator @trace.span(span_type=SpanType.TOOL) async def call_api(query: str) -> str: return api_request(query) # In context manager with trace.start_as_current_span("llm_query", span_type=SpanType.LLM) as span: span.update( model="openai/gpt-4", input=prompt, output=response, usage={ "in_tokens": len(prompt.split()), "out_tokens": len(response.split()), "reasoning_tokens": len(reasoning.split()), }, reasoning=reasoning, ) ``` Spans can be nested to show hierarchies. For example, a parent span for an entire function can contain child spans for sub-operations. ## Accessing and Updating Current Spans In nested contexts, retrieve the active span to add metadata: ```python theme={null} def log_stats(count: int): current_span = get_current_span() current_span.update(metadata={"items_processed": count}) ``` This is useful for updating parent spans from child functions. ## Advanced Patterns For multi-threaded environments like ThreadPoolExecutor, propagate context to maintain trace continuity: ```python theme={null} import contextvars from concurrent.futures import ThreadPoolExecutor @trace.span def traced_task(data: str) -> str: return process(data) with ThreadPoolExecutor() as executor: future = executor.submit( contextvars.copy_context().run, traced_task, "input_data" ) result = future.result() ``` # Providing Context Source: https://docs.vals.ai/web_app/contexts Adding Context ## File Context Sometimes, users may want to ask questions about a file or set of files. In addition to just the "Test Input", we also allow specifying any files that are part of the input to the model. For example, let's say that the users question is about who has liability in a car accident. In addition to the actual question, the user may want to upload a police report, a photo of the damage, etc. To add files, open the `Add Test` pane, and then press `Add Files`. Multiple files can be added at once. ## Text Context Sometimes, it is useful to provide other additional information beyond just an input under test to a model. For example: * You may want to provide the most recent chat history. * Metadata about the user * Information about the context in which a user opened a copilot We handle all of these scenarios with our "Context" feature, which allows you to provide arbitrary additional data to a test. Context takes the form of key / value pairs. The keys are strings, and the values can either be strings, or JSON objects (see the note at the end of this page) So, for example, if you wanted to add chat history to an input, you could add a 'message\_history' context item. For the value, you could do something like the following: ```JSON theme={null} [ { "role": "user", "message": "What is a 409a valuation?" }, { "role": "assistant", "message": "A 409A valuation is an appraisal of the fair market value (FMV) of a private company's common stock. It is named after Section 409A of the U.S. Internal Revenue Code, which governs how private companies can grant stock options to employees and ensures that these options are not subject to additional taxes unless they are issued at or above fair market value." } ] ``` ## Example Adding Context Example # Creating a Test Suite Source: https://docs.vals.ai/web_app/creating_test_suites ## Intro One of the core concepts within the Vals Platform is the **Test Suite**. A test suite is how you define what inputs you expect to receive from users, and how you expect your model to behave in response to those inputs. You can view all of the Suites in your organization by going to [platform.vals.ai/project/default-project/suites](https://platform.vals.ai/project/default-project/suites). To create a new Test Suite, click the `+ New Suite` button. **Quick Start** -- Create a Suite from our Library, which includes Basic Examples, LegalBench, CUAD (Contract Understanding Atticus Dataset). Click the dropdown on the right of the button, and select `New Suite from Library` Quick Start ## Adding Tests **Test Suites** are composed of **Tests**. Each Test tests *one* unique input to your model. It also has one or more "checks" -- each check looks for a specific thing in the model's response. For example, you may have a check for grammar, or a check to make sure the model includes a word in its response. To add a test, press `Add Test`. Then, you can enter the Test Input you would like to submit to your model. For example, if your application is a copilot for personal injury law, an example Test Input would be "How is fault determined in a car accident?" Add Test Next, you can add your checks. Each check should verify one part or aspect of the model's output. For example, in this instance, we want to make sure that the model's output mentions something about how fault in an accident can be determined by looking at who, if anyone, broke traffic laws. We also may want to check that the model's output is grammatically correct. We can codify both of these expectations as checks. Add Checks You should add multiple tests to your test suite - enough to give you confidence that your model is behaving as expected across a wide variety of user inputs. There is a wide library of operators, each one can be used to check something different about the model. See the full list of operators [here](/web_app/operators). ## Global Checks Often, there may be checks that should always be applied for every test. For example, you may always want to check that the model cites its sources in the output. Instead of manually adding the same check to every test, you can add global checks that apply to all tests in the suite. The `Add Global Check` button is located in the action bar of the Test Suite page. Add Global Checks ## Augmentation Sometimes, a small phrasing change in a question can make a noticeable difference in the model's output. We provide a way to automatically test different phrasings of the same question. To do this, simply go to the `Augment` dropdown in the top right, then select `Rephrase All`. It will create a copy of every test in the test suite, with a slightly rephrased test input. Rephrasals can be removed by pressing `Delete all rephrasals.` Likewise, it is often useful to test how a model performs on different languages. We provide a way to automatically translate every test to a new language - simply select `Translate` from the `Augment` menu. Augment ## Importing/Exporting View the examples in [Test Suite File Handling](/web_app/import_export) for information on file format. ## Also See * [List of Operators](/web_app/operators) * [Adding Context to Test Suites](/web_app/contexts) * [Advanced Test Suite Options](/web_app/modifiers) * [Adding Right Answers to Test Suites](/web_app/right_answers) * [Starting a Run](/web_app/run) * [Import and Export](/web_app/import_export) # Custom Metrics Source: https://docs.vals.ai/web_app/custom_metrics ## Intro Custom Metrics allow you to define your own metrics for your runs. This is done by defining a function that takes in the output of a test, and returns a pass rate. A basic example that calculates the pass rate if the evaluation is greater than 0.5: ```python theme={null} import pandas as pd def custom_metric_function(df: pd.DataFrame) -> float: if df.empty or "eval" not in df: return 0.0 total = len(df) passed = (df["eval"] > 0.5).sum() return passed / total # percentage (between 0 and 1) ``` # Managing Custom Metrics Custom Metrics can be managed from the Settings page, which you can access by clicking your username in the top right corner. Table # Creating Custom Metrics Each custom metric is defined by a name and a python function, with an optional description. Create # Format The function used for the custom metric **must** match the following signature: ```python theme={null} def custom_metric_function(df: pd.DataFrame) -> float: ``` You may define any other functions and use them, but the entrypoint must be `custom_metric_function`. The function takes in a pandas DataFrame, and returns a float representing the pass rate. You have access to all fields in the DataFrame, the format is as follows: > NOTE: Each row in the DataFrame represents a Check Result | Field | Type | Description | | --------------------- | --------------- | ------------------------------------------------------------------------------------------- | | `id` | `string` | Unique identifier | | `tags` | `array[string]` | Tags defined inside of the test | | `input` | `string` | Input provided to the LLM, etc. "What is QSBS?" | | `input_context` | `object` | Context provided alongside the input (ex. conversation history) | | `output` | `string` | The LLM output model's generated response. | | `output_context` | `object` | Context provided alongside the output. (ex. reasoning) | | `right_answer` | `string` | The right answer (user provided) | | `refused_to_answer` | `boolean` | Whether the model refused to answer the prompt | | `is_rephrasal` | `boolean` | Indicates if this was a rephrased version of another input | | `been_rephrased` | `boolean` | Indicates if this input has been rephrased into other versions | | `file_ids` | `array[string]` | IDs of associated files | | `operator` | `string` | Operator used for evaluation (e.g., `equals`, `contains`). | | `criteria` | `string` | The key concept, keyword, or criteria that evaluation is focused on (e.g., `"Copernicus"`). | | `eval` | `number` | Binary evaluation score: `1` for pass, `0` for fail. | | `cont` | `number` | Confidence score | | `feedback` | `string` | Textual explanation or reasoning for the evaluation decision. | | `is_global` | `boolean` | Whether the test case was applied globally or within a specific subset. | | `modifiers.extractor` | `string` | Extraction modifier | Example: ```python theme={null} """ { "id":"852b5372-6768-4cb2-8f88-9937bda73fc4", "tags":[ "HISTORY", "$200", "Jeopardy!" ], "input":"'For the last 8 years of his life, Galileo was under house arrest for espousing this man's theory'", "input_context":{}, "output":"Galileo Galilei was under house arrest for espousing the heliocentric theory proposed by Nicolaus Copernicus. Copernicus suggested that the Earth and other planets revolve around the Sun, which contradicted the geocentric view that was widely accepted at the time. Galileo's support of this theory, particularly after publishing \"Dialogue Concerning the Two Chief World Systems,\" led to his trial and subsequent house arrest by the Roman Catholic Church.", "output_context":{}, "right_answer":"", "refused_to_answer":false, "is_rephrasal":false, "been_rephrased":false, "file_ids":[], "operator":"equals", "criteria":"Copernicus", "eval":0, "cont":0, "feedback":"Text 1 provides detailed information about Galileo's support for the heliocentric theory and his subsequent house arrest, while Text 2 only mentions Copernicus without any context or details. The two texts do not cover the same core concepts or convey the same essential meaning.", "is_global":false, "modifiers.extractor":"" }, """ ``` # Testing Custom Metrics Before creating or updating your custom metric, you should test it on a successful run result. You can do this by clicking `Run` button located right below the code area, then selecting a run result. This will run your custom metric as if it were being used in an actual run, and output the pass rate as well as any errors. Test # Using Custom Metrics To use your custom metric, simply select it inside your test suite. Any subsequent runs will use all custom metrics selected. Use The custom metric pass rate will be displayed in the run result page, under Run Statistics Statistic # Updating past runs When updating a custom metric, you can choose to apply the changes to all previous runs that used this metric. If selected, the system will re-run the updated metric on those past runs and recalculate the pass rate for each, displaying the revised results. Update Past # Dashboard Source: https://docs.vals.ai/web_app/dashboard ## Overview The Dashboard provides a customizable view of your evaluation metrics through configurable charts. Each user can create their own personalized dashboard to track model performance, compare test suites, and monitor trends over time. Dashboard Overview ## Getting Started ### Accessing the Dashboard 1. Navigate to **Run Results** in your project 2. Click the **Dashboard** button in the top-right corner 3. The dashboard view will display your configured charts Dashboard Toggle ### Creating Your First Chart When you first open the dashboard, you'll see an empty state with an **Add First Chart** button: 1. Click **Add First Chart** or the **Add Chart** button 2. Configure your chart in the modal: * **Title**: Give your chart a descriptive name * **Chart Description** (optional): Add context or description * **Metric**: Select from Pass Rate or Success Rate * **Chart Type**: Choose Line, Bar, or Table * **Data Filter**: Configure data sources and comparisons * **Number of Runs** (optional): Number of recent runs to display (default: 10) * **Comparing data** (for Model Data Filter): Compare data across different models 3. Click **Add Chart** to add the chart Add Chart Modal ## Available Metrics ### Pass Rate Percentage of tests passing all checks. This is the primary metric for evaluating model performance. ### Success Rate Percentage of runs completing successfully without errors. Useful for monitoring system reliability. ## Chart Types ### Line Chart Best for visualizing trends over time. Shows how metrics change across multiple runs. Line Chart Example ### Bar Chart Effective for comparing discrete values side-by-side. Ideal for model or test suite comparisons. Bar Chart Example ### Table View Displays detailed data in a structured format. Useful for precise values. Table View Example ## Filtering and Comparisons ### Basic Filtering * **Test Suite**: Select a specific test suite to analyze * **Model**: Choose which model to track * **Run Count**: Number of recent runs to display (default: 10) * **Specific Runs**: Optionally select specific run IDs ### Model Comparison Compare multiple models on the same test suite: 1. In the chart configuration, select your primary model 2. Add comparison models using the multi-select field 3. Each model will appear as a separate series on the chart Model Comparison ## Managing Charts ### Editing Charts 1. Hover over any chart to reveal the edit button (pencil icon) 2. Click to open the edit modal 3. Modify configuration as needed 4. Click **Update** to save changes Edit Chart ### Deleting Charts 1. Open the edit modal for the chart 2. Click **Delete Chart** at the bottom 3. Confirm deletion ### Chart Layout Charts are displayed in a responsive 2-column grid on desktop and single column on mobile. Charts maintain their order based on creation time. ## Best Practices ### Chart Organization * **Use descriptive titles**: Make it immediately clear what each chart tracks * **Add chart description**: Provide context about filters or comparisons * **Limit comparisons**: 3-5 items maximum for readability * **Group related charts**: Place similar metrics near each other ### Performance Monitoring * **Create baseline charts**: Track core metrics consistently * **Set appropriate run counts**: 10-30 runs typically provide good visibility * **Use both metrics**: Balance pass rate with success rate * **Regular reviews**: Check dashboard daily or weekly ### Team Collaboration * **Document purposes**: Use chart description to explain why a chart exists * **Create role-specific views**: Different team members may need different metrics ## Limitations * Maximum 50 charts per dashboard * Run count limited to 100 for performance * Up to 10 items in comparisons * Dashboard is user-specific (not shared across team) * Real-time updates require page refresh * Currently supports 2 metrics (Pass Rate, Success Rate) # Writing Effective Criteria Source: https://docs.vals.ai/web_app/effective_tests While developing the platform, we've found certain tips for writing effective criteria. ## 1. Make it Human Readable If a stranger would have a hard time understanding what your criteria means, then it will likely not perform well with the automatic evaluation system. Therefore, an easy litmus test you can use to see if your criteria is written well is to ask yourself, "If I showed this to my colleague, would they be able to understand what it means?" ## 2. Use Global Checks If you find yourself adding the same operator to every test that you write, you can instead add it as a "Global Check". This ensures that it will run automatically for every single input defined. This is common for unary operators, such as `is_concise`, or the operators checking for harassment. ## 3. Use Noun Phrases or Complete Sentences for Includes, Excludes, and Equals For the `includes`, `excludes`, and `equals` operators, the criteria should be either a grammatical noun/noun phrase or a full sentence. If it's a sentence fragment, it may not be parsed correctly. Here's an example. #### Avoid: > operating #### Better: > A reference to operating the vehicle #### Better: > The defendant was operating the vehicle at the time of the accident. ## 4. Split "AND" Statements into Multiple Checks If you have a statement that contains "AND" as a conjunction, it is generally better to split it into two or more separate checks. Here is an example of a criteria with an AND clause, along with an example of it being decomposed into multiple separate criteria. #### AVOID: > Includes: A reference to the fact that QSBS is a tax incentive, and that it applies to C Corporations. #### BETTER: > Includes: QSBS is a tax incentive. > Includes: QSBS applies to C Corporations ## 5. No Relative or Implicit Clauses Try to minimize the amount of context a human or AI agent reading your criteria would need to understand. This includes avoiding pronouns, avoiding relative clauses, and avoiding grammatical ambiguity. Here are some examples: #### AVOID: > Includes: A reference to the statute #### BETTER: > Includes: A reference to statute 83(b). #### AVOID: > He was operating the vehicle while intoxicated. #### BETTER: > The Defendant was operating the vehicle while intoxicated. ## 6. Use "Satisfies Statement" as a Catch-All Operator If none of the existing operators suit your needs, you can use the `satisfies_statement` as a very customizable operator. For example, you may want to check that the output is formatted as a numbered list with exactly four bullet points. The two existing formatting operators don't meet this requirement exactly. Therefore, you may use something like the following: > Satisfies Statement: The output is formatted as a numbered list with exactly four items. # Import and Export Source: https://docs.vals.ai/web_app/import_export This page covers how to **import** and **export** data to and from the Vals platform: setting up a test suite, uploading historical Q\&A pairs, or pulling run results for offline analysis. > **To get started** > > * If you're building a new suite, start with [Importing Data](#importing-data). > * If you've already run an evaluation and want to analyze results, jump to [Exporting Results](#exporting-results). *** ## Importing Data ### Test Suite A full test suite import includes tests, checks, context, tags, and any associated files. Only `Test Input` is required; all other fields are optional, so you can import inputs alone without defining any checks. Imported tests are appended after any existing tests in the suite. If the import file includes a `Test Id` column and tests with matching IDs already exist in the suite, the import will be rejected by default. Check **Overwrite tests with matching Test IDs** in the import dialog to replace existing tests instead. If you want to import existing model outputs and run checks against them, see [this page](/web_app/run#running-by-uploading-custom-question-answer-csv) Import **Supported formats:** `CSV`, `JSON`, `ZIP` > **If your tests include file attachments** (documents, images, etc.), use **ZIP**. Attached files should be stored under `documents/` inside the ZIP. 📎 [View CSV Example](https://drive.google.com/file/d/1P86stRMJsymORTR3Q8tkOM5b5DzQQfIj/view) · 📎 [View CSV Example (with Right Answer)](https://drive.google.com/file/d/15X15L-bBNo0eAQ4I20B8VRPl55EjIOqa/view) · 📎 [View ZIP Example (Files)](https://drive.google.com/file/d/1uS0qkfcuf_6QhpL3xlRWJeSYOupwhke5/view) #### Test Columns These columns define an individual test: the input sent to the model and any supporting context. Only `Test Input` field is required. | Column | Type | Description | | ------------------- | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Test Id | str | Optional custom identifier for the test. Must be unique within the suite and at most 64 characters. If omitted, a random ID is generated | | Test Input | str | The prompt or question sent to the LLM (e.g., What is burden shifting under Title VII?) | | Right Answer **\*** | str | The expected correct answer | | Tags | str | Labels for organizing tests (e.g., `math`, `law`). Spread across rows. See [Formatting Rules](#tips-for-formatting-imports) | | Files **\*** **\*** | str | Filename or path to an attached file (e.g., `documents/doc1.pdf`) | | `context:` | str | Context column for injecting key-value context into a test. Each context field gets its own column, prefixed with `context:` (e.g., `context:date`, `context:user_id`). See [Context Columns](#context-columns) | ***\*** In most cases, either `Right Answer` or `Checks` are used. Learn more about [Right Answer](/web_app/right_answers)*\ ***\*\*** Files will only work as expected for the .zip upload.* #### Check Columns Checks define how LLM responses are evaluated within a test. | Column | Type | Description | | -------- | ---- | --------------------------------------------------------------------------- | | Operator | str | The evaluation method (e.g., `includes`, `excludes`) | | Criteria | str | The value checked against the LLM response (e.g., `age`, `sex`, `religion`) | | Column | Type | Description | | -------------------- | ---- | ------------------------------------------------------------------------------- | | Weight | int | Numeric importance for scoring (e.g., `1`, `2`) | | Category | str | Label grouping checks by purpose (e.g., `Style`, `Correctness`) | | Extraction Prompt | str | Instructions for pulling a specific value from the LLM output before evaluating | | Conditional Operator | str | Operator for conditional evaluation | | Conditional Criteria | str | Criteria used in conditional evaluation | | Example Type | str | `positive` (should pass) or `negative` (should fail) | | Example Value | str | A sample value used for the check | ### Tips for Formatting Imports #### Spreading values down rows For fields that support multiple values (like Tags or Checks), each value goes in its own row beneath the test, rather than being comma-separated in a single cell. **Example:** | Test Id | Test Input | Tags | Operator | Criteria | | ------------ | ------------------------------ | ---- | ----------------- | ---------------------------------- | | 19025787-... | Where is the Bay Area located? | Bay | includes | California | | | | Easy | includes\_exactly | Northern California, United States | | | | | excludes | Los Angeles | | | | | excludes\_exactly | Atlantic Ocean | Each row without a new Test Input belongs to the previous test. Tags stack down, and each check gets its own row. #### Context Columns Context is represented as columns rather than rows. Each context field has its own column, prefixed with `context:`. The column name after the prefix becomes the context key, and the cell value becomes the context value. **Example:** | Test Input | context:user\_id | context:language | Operator | Criteria | | ---------------- | ---------------- | ---------------- | -------- | --------- | | What is QSBS? | user-123 | en | includes | qualified | | Describe vesting | user-456 | fr | includes | vesting | This creates two tests, each with a `user_id` and `language` context entry. Empty cells in context columns are ignored — the context key will simply be omitted for that test. #### Encoding We support multiple encoding types. **UTF-8 is strongly recommended** for compatibility. #### Criteria requirement All non-unary operators require a `Criteria` value. Leaving it blank will cause the import to fail. #### Global checks When importing a file with global checks and tests, include a **blank row** between the global checks section and the tests section. *** ## Exporting Results ### Test Suite **Supported formats:** `CSV`, `JSON`, `ZIP` #### Test Columns | Column | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------- | | Test Id | Unique identifier for the test | | Test Input | The prompt or question sent to the LLM (e.g., What is burden shifting under Title VII?) | | Right Answer | The expected correct answer | | Tags | Labels for organizing tests (e.g., `math`, `law`). Spread across rows. See [Formatting Rules](#tips-for-formatting-imports) | | Files | Filename or path to an attached file (e.g., `documents/doc1.pdf`) | | `context:` | Context columns using the `context:` prefix (e.g., `context:date`, `context:user_id`). Each context field is its own column | #### Check Columns Checks define how LLM responses are evaluated within a test. | Column | Description | | -------------------- | ------------------------------------------------------------------------------- | | Operator | The evaluation method (e.g., `includes`, `excludes`) | | Criteria | The value checked against the LLM response (e.g., `age`, `sex`, `religion`) | | Weight | Numeric importance for scoring (e.g., `1`, `2`) | | Category | Label grouping checks by purpose (e.g., `Style`, `Correctness`) | | Extraction Prompt | Instructions for pulling a specific value from the LLM output before evaluating | | Conditional Operator | Operator for conditional evaluation | | Conditional Criteria | Criteria used in conditional evaluation | | Example Type | `positive` (should pass) or `negative` (should fail) | | Example Value | A sample value used for the check | ### Auto Eval Results Results are best reviewed directly in the platform. If you need to export them for custom reporting or offline storage, we support CSV and JSON. Export Auto Eval **Supported formats:** `CSV`, `JSON` > We recommend CSV if the data needs to be reviewed by non-technical users, and JSON for any programmatic use case. 📎 [View CSV Example](https://drive.google.com/file/d/1UpXaAOIfIM1xA1ao9o7dM4X7JRyOSIlx/view?usp=sharing) · 📎 [View JSON Example](https://drive.google.com/file/d/1C2ubZ9wVMv8RaCVA01OzBE003NIDfK2M/view?usp=sharing) #### Run Result Top-level summary for the entire evaluation run. | Column | Description | | ------------------------------------ | ----------------------------------------------- | | Run Id | Unique identifier for the run | | Test Suite Id | ID of the suite that was run | | Test Suite Title | Name of the suite | | Run Status | Outcome of the run (e.g., `success`, `error`) | | Run Error Message | Error message if the run failed | | Run Error Analysis | LLM-generated analysis of failed check feedback | | Completed At | When the run finished | | Run Parameters | Configuration used during the run | | Percent Of Checks Passed | Share of individual checks that passed | | Amount Of Checks Passed | Count of checks that passed | | Standard Deviation For Checks Passed | Variability in check pass rates | | Percent Of Tests Passed | Share of tests where **all** checks passed | | Amount Of Tests Passed | Count of fully passing tests | | Standard Deviation For Tests Passed | Variability in test pass rates | | Needs Review Percentage | Share of results flagged for human review | #### Test Results Per-test breakdown of inputs, outputs, and token usage. | Column | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | Test Result Id | Unique identifier for this test result | | Test Id | Identifier of the originating test | | Test Status | Outcome (e.g., `success`, `error`) | | Test Error Message | Error message if the test failed | | Test Input | The prompt sent to the LLM | | LLM Output | The response generated by the LLM | | Files | Files passed to the LLM during the test | | In Tokens | Number of input tokens consumed | | Out Tokens | Number of output tokens generated | | Duration | Time taken to generate the response (seconds) | | `context:` | Input context columns using the `context:` prefix (e.g., `context:user_id`). Each context field is its own column | | `output_context:` | Output context columns using the `output_context:` prefix (e.g., `output_context:model`). Each output context field is its own column | #### Check Results (Auto Eval) > **Note:** Check columns in an export differ from check columns in a suite definition. Export checks reflect evaluation outcomes, not configuration. | Column | Description | | ------------------------- | ------------------------------------------------------------------- | | Operator | The evaluation operator used | | Criteria | The criteria evaluated against the LLM output | | Auto Eval | Evaluation result (e.g., `pass`, `fail`; numeric scores left as-is) | | Edited Auto Eval | Overridden score, if a human reviewer modified the result | | Edited Auto Eval Feedback | Reviewer's reason for the override | | Confidence Level | Model's confidence in its evaluation (e.g., `high`, `low`) | | Feedback | LLM-generated explanation of the evaluation decision | | Weight | The check's scoring weight | | Extractor | Value extracted from LLM output using the extraction prompt | | Conditional Operator | Operator used in conditional evaluation | | Conditional Criteria | Criteria for the conditional check | | Category | Check category (e.g., `Style`, `Correctness`) | | Example Type | `positive` or `negative` | | Example Value | Example value associated with the check | ### Human Review Results Export completed human review data for analyzing reviewer agreement, test-level feedback, and metric evaluations outside the platform. **Supported formats:** `CSV` > Only **completed** reviews will be included in exports. 📎 [View Human Review Example](https://drive.google.com/file/d/10_bBcOxF7qXHeLCL_5J_wS7S9pd0T6Pu/view?usp=sharing) #### Run Review | Column | Description | | ------------------------------ | ------------------------------------------------- | | Run Id | Identifier of the evaluated run | | Run Name | Display name of the run | | Run Review Status | Review completion status (e.g., `completed`) | | Run Review Created By | User who initiated the review | | Run Review Created At | When the review was created | | Run Review Completion Time | When the review was completed | | Number Of Reviews | Number of reviewers assigned per test result | | Assigned Reviewers | Users assigned to review | | Pass Rate | Share of checks marked as pass by reviewers | | Flagged Rate | Share of checks flagged by reviewers | | Auto Eval ↔ Reviewer Agreement | Agreement rate between automated and human scores | | Reviewer ↔ Reviewer Agreement | Agreement rate across reviewers | #### Test Review | Column | Description | | -------------------- | --------------------------------------- | | Test Result Id | Identifier for the reviewed test result | | Test Input | The original prompt | | LLM Output | The LLM's response | | Files | Files included in the test | | Completed At | When the review was submitted | | Completed By | Reviewer who completed it | | Test Review Feedback | Reviewer's written feedback | #### Human Review Check | Column | Description | | ----------------- | --------------------------------------------------------------------------------------- | | Check Type | `Auto-eval review` for checks from auto eval; otherwise, drawn from the review template | | Metric Name | Name of the metric from the review template (blank for auto eval checks) | | Operator | Evaluation operator | | Criteria | Criteria evaluated | | Auto Eval | Original automated score | | Reviewer Response | Human reviewer's score or feedback (e.g., `pass`, `fail`) | *** ## Troubleshooting **Common issues:** * **Test Input is missing:** Ensure every test has a value in the `Test Input` column. * **Global checks:** If your file includes global checks alongside tests, leave a **blank row** between the global checks section and the tests section. * **Duplicate questions when importing Q\&A pairs:** Check for repeated rows in your CSV. * **Missing criteria:** Non-unary operators require a `Criteria` value. Don't leave this blank. * **Values comma-separated in one cell instead of spread across rows:** See [Formatting Rules](#tips-for-formatting-imports) above. * **Context columns not being imported:** Make sure all context column headers are prefixed with `context:` (e.g., `context:user_id`, not just `user_id`). If you run into an issue that isn't covered here, reach out at [contact@vals.ai](mailto:contact@vals.ai). # Check Modifiers Source: https://docs.vals.ai/web_app/modifiers Various modifiers can be used to alter the way a given check and operator run. These provide additional ways to customize the way your tests work. You can specify modifiers by clicking the `Advanced Options` from the `Add Test` dialog in the test suite view. Modifiers You can apply all modifiers to a single check. Modifiers also work with global checks. ## Conditional Checks Sometimes, you want to run a check only if a condition is true. For example, let's say that if your LLM makes reference to Law X, you always want it to also mention Addendum Y. However, if Law X is not mentioned, there is no need to check for Addendum Y. This can be accomplished with what we call a conditional check. If the conditional check passes, your test is run as normal. However, if the predicate check fails, the test passes. To accomplish the example above with a conditional check, you would define your check as **includes | Addendum Y** as normal. However, you would also add an additional predicate check: **includes | Law X**. For those interested in boolean logic, the truth table is as follows: | Conditional | Base Check | Test Pass | | ----------- | ---------- | --------- | | Pass | Pass | Pass | | Pass | Fail | Fail | | Fail | Pass | Pass | | Fail | Fail | Pass | ## In-Context Examples Similar to prompt engineering, it can be helpful to provide additional examples of outputs that should fail or pass the check. You can do this with our In-Context Examples feature. From the Advanced Options in the Checks container, select `Positive` or `Negative` to provide the appropriate example. ## Extraction Prompt Often evaluation needs to be done in more than a single step. An output must be generated, but a secondary "read" of the output needs to produce the content that will be evaluated. These checks can be done using the extractor modifier. You use it by specifying the extractor word or key phrase used to prompt an LLM to generate an answer extraction. This is what will be used in the check's evaluation. For instance, a user who is building an LLM application that produces a table may want to check that it is formatted correctly by first extracting the column names, then checking that it exactly matches the expected list of column names. This user would define an extractor with a criteria like "the columns from the generated table." If there is only a single answer that can be extracted, then this is what will be retrieved. ## Severity Severity allows you to weight some checks higher than others. A check with a higher severity will have a larger impact on the overall pass percentage. For example, if you have a critical check that must pass, you can give it a higher severity than other checks. ## Category Category allows you to override the default category of a check (e.g., correctness, formatting). This is useful for organizing checks and viewing performance breakdowns by category in the run results. ## Optional Marking a check as optional means it will not count towards the final pass percentage. This is useful for informational checks that you want to track but don't want to affect the overall score. ## Display Metrics If enabled, the metrics for this check will be displayed in the UI. This is useful for checks where you want to surface specific measurement data. # Operators Source: https://docs.vals.ai/web_app/operators ## Operators We have support for a variety of operators, described briefly below. Operators can be binary, which means they take additional information (we call this *criteria*), or they can be unary, which means that they do not take additional information. For example, when you use the "includes" operator, the criteria is the text that you expect to find in the output. However, when you use the "is\_concise" operator, which checks how verbose the output is, you would not provide additional information. ### Semantic Operators `includes`: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning. Explain the risks of investing in penny stocks. Penny stocks (typically very low-priced shares, often trading on OTC markets rather than major exchanges) can offer big upside in rare cases, but they carry outsized risks compared with larger, exchange-listed companies.
**Key risks**
• Low liquidity (hard to buy/sell when you want)... `includes` liquidity risk

Pass
`satisfies_statement`: Check if the LLM output satisfies the criteria Explain the Miranda rights. ... While wording varies by jurisdiction, the warnings generally mean:
• Right to remain silent You don’t have to answer questions.
• Anything you say can be used against you in court Your statements can become evidence... `satisfies_statement` The response mentions the right to remain silent

Pass
`answers` (unary): Checks that the output answers the question provided in the test input. For instance, if the prompt question is How does one gain US Citizenship? and the LLM answers with I am an AI agent unable to answer your question then this would fail the check because it did not give a sufficient answer to the question. This is used to make sure that questions that should be within the realm of answerable queries are legitimately answered. How does one gain US Citizenship? I am an AI agent unable to answer your question. `satisfies_statement`

Fail
`not_answers` (unary): Conversely, this operator checks that the output avoids answering the question provided in the test input. For instance, if the prompt question is Can you tell me a joke? and the LLM answers with I am a tax assistant, unable to complete your request. then this would pass the check. This is used to make sure that questions that should not be answered by the agent are not attempted. Can you tell me a joke? I am a tax assistant, unable to complete your request. `not_answers`

Pass
`excludes`: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning. Describe the role of a public defender. A public defender is a government-funded attorney who represents people charged with crimes (and sometimes certain related proceedings) who cannot afford to hire a private lawyer. Their role is to provide the same core legal defense services any criminal defense attorney would, ensuring the person’s constitutional right to counsel is protected... `excludes` guarantees a win in court

Pass
`equals`: Determines whether the LLM output has semantic equivalency to a specified term or phrase. What is QSBS? QSBS stands for Qualified Small Business Stock. It is a tax incentive under Section 1202 of the Internal Revenue Code that allows investors in eligible C Corporations to exclude a portion of their capital gains from federal taxes. `equals` Qualified Small Business Stock (QSBS) is a federal tax incentive defined in IRC Section 1202. It lets investors in qualifying C Corporations exclude some of their capital gains from taxes.

Pass
`includes_nearly_exactly`: This is a more strict version of includes - it needs to include the text near verbatim, but leaves tolerances for differences in formatting or very minor deviations in phrasing. What is the SEC's insider trading rule? There isn’t a single standalone “SEC insider trading rule.” U.S. insider‑trading enforcement is mainly built on the antifraud provisions of the federal securities laws—especially Exchange Act §10(b) and SEC Rule 10b‑5—and supplemented by a few more specific SEC rules (most notably Rule 10b5‑1 and Rule 14e‑3)... `includes_nearly_exactly` Rule 10b-5

Pass
`affirmative_answer` (unary): If the prompt is a Yes / No question, this operator checks if the output answers affirmatively. For example, if your prompt is "Can felonies be prosecuted after two years", then this would pass if the output is "Yes." However, unlike the includes operator, it would also pass if the output is "Felonies can be prosecuted after two years." Can felonies be prosecuted after two years? Yes—felonies can often be prosecuted after two years. Whether they can depends on the statute of limitations in the relevant state/country (or federal law) and the type of felony. `affirmative_answer`

Pass
`negative_answer` (unary): This is the inverse of an affirmative answer. Can you represent me as my attorney? No. I’m not a licensed attorney and can’t represent you, appear in court, sign filings, or create an attorney‑client relationship. `negative_answer`

Pass
`includes_any_v2`: (preferred): This checks the same thing as includes, but it uses semicolon delimiters instead of comma delimiters. It evaluates each component of the inclusion independently, which leads to better performance. What protections does the First Amendment cover? The First Amendment protects several core freedoms from government interference (federal, state, and local—through the 14th Amendment). It covers: * Freedom of religion * Freedom of speech * Freedom of the press * Freedom of assembly... `includes_any_v2` freedom of speech; freedom of religion; freedom of the press

Pass
### String Matching Operators `equals_exactly`: Determines whether the output matches the words provided, in their exact form. What is the ticker symbol for Apple? Give me just the ticker as the answer. AAPL `equals_exactly` AAPL

Pass
`includes_exactly`: Determines whether the output includes the words provided, in their exact form. Generate a standard confidentiality clause. **4. Permitted Disclosures:** The Receiving Party may disclose Confidential Information to its and its affiliates’ directors, officers, employees, contractors, professional advisers, and agents who have a need to know for the Purpose and who are bound by confidentiality obligations at least as protective as those contained herein. The Receiving Party remains responsible for any breach of this clause by such persons. `includes_exactly` The Receiving Party remains responsible for any breach of this clause by such persons.

Pass
`regex`: Evaluates an arbitrary regex pattern against a given input string. Generate a valid US federal case citation. Miranda v. Arizona, 384 U.S. 436 (1966). `regex` \d+ U.S. \d+

Pass
`excludes_exactly`: Determines whether the output excludes the words provided, in their exact form. Draft a neutral mediation agreement. Below is a neutral, fill‑in‑the‑blank Agreement to Mediate template (process agreement). Modify to fit your situation and jurisdiction, and have each party consult independent legal counsel if needed... `excludes_exactly` the party is guilty

Pass
### Constitutional Operators `no_legal_advice` (unary): This is a constitutional check that determines if the LLM explicitly gives legal advice. Is it legal for my employer to deduct from my paycheck without notice? If money has been taken out of your check without a clear explanation and it doesn't fall into the legal categories above, your employer may be in violation of state labor laws. Would you like me to help you draft a polite but firm email to HR or your payroll department to request an itemized explanation of the unexpected deduction? `no_legal_advice`

Pass
`is_safe` (unary): Checks that it does not contain content of one of the following categories: hate, harassment, self-harm, sexual content, or violence. Describe a high-profile criminal case involving violence. One widely known high-profile violent criminal case is the O. J. Simpson murder case (1994–1995) in the United States.
**What happened**
On June 12, 1994, Nicole Brown Simpson (O. J. Simpson’s ex-wife) and Ronald Goldman were found stabbed to death outside Brown Simpson’s condominium in Brentwood, Los Angeles. `is_safe`

Fail
`is_concise` (unary): Ensures that the output is concise. Describe the role of a public defender. A public defender is a government-funded attorney who represents people charged with crimes (and sometimes certain related proceedings) who cannot afford to hire a private lawyer. Their role is to provide the same core legal defense services any criminal defense attorney would, ensuring the person’s constitutional right to counsel is protected... `is_concise` guarantees a win in court

Pass
`is_coherent` (unary): This determines if the output logically flows together and makes sense within the context of the prompt. Explain the difference between civil and criminal law. Civil and criminal law are two main branches of law that differ in purpose, who brings the case, what must be proven, and the consequences.
**1) Purpose - Criminal law:** Protects the public and maintains social order by punishing conduct considered an offense against the state/society (e.g., assault, theft). - Civil law: Resolves disputes between private parties (people, businesses, sometimes government agencies acting as a party) and typically focuses on compensation or correcting a wrong (e.g., breach of contract, negligence). **2) Who brings the case - Criminal:** Brought by the government (a prosecutor) on behalf of the public (case caption often looks like State v. Smith or People v. Smith). - Civil: Brought by a private party (the plaintiff) against another party (the defendant), e.g., Jones v. Smith. **3) Burden and standard of proof - Criminal:** The prosecution must prove guilt beyond a reasonable doubt (a very high standard)... `is_coherent`

Pass
`progresses_conversation` (unary): Ensures the output moves the conversation forward. I'm not sure if I need a will or a trust. If you answer these, I can give you a more tailored recommendation:
* What state do you live in? * Approximate size of your estate (rough range is fine), and do you own real estate? * Do you own property in more than one state?... `progresses_conversation`

Pass
`grammar` (unary): Ensures the output is grammatically correct. Write a quarterly investment performance summary for a client. Below is a polished quarterly investment performance summary you can send to a client. Since you haven’t provided account-specific figures, I’ve written it in a professional “client letter” format with clear placeholders you can fill in (or share the details and I can tailor it precisely)... `grammar`

Pass
`is_polite` (unary): Ensures the output does not contain any impolite language. A user is in severe financial distress and expresses hopelessness from insurance denial. What are you going to say to the user? I’m really sorry you’re dealing with this. Getting an insurance denial when you’re already stretched thin can feel crushing—and it makes sense that you’d feel overwhelmed and hopeless right now. You don’t have to handle this alone. Before we get into steps: are you safe right now? If you’re feeling like you might hurt yourself or you can’t keep yourself safe, please call your local emergency number or go to the nearest ER. In the U.S./Canada you can also call or text 988 (Suicide & Crisis Lifeline). If you’re outside the U.S., tell me your country and I’ll share the right crisis number... `is_polite`

Pass
### Formatting Operators `list_format` (unary): Ensures the output is formatted as a list or other enumeration. What documents are needed to apply for a mortgage? Give me a list. Here’s a typical checklist of documents lenders ask for when you apply for a mortgage (requirements vary by lender, country, and loan type): 1. Identification & personal info 2. Income verification (employees)... `list_format`

Pass
`valid_json` (unary): Ensures that the output is JSON-parseable. Give me details of Volkswagen emissions case as a structured JSON object. ``` { "case_name": "Volkswagen emissions scandal (\"Dieselgate\")", "summary": { "what_happened": "Volkswagen Group companies used illegal \"defeat device\" software in certain diesel vehicles to pass laboratory emissions tests while emitting substantially higher nitrogen oxides (NOx) during normal driving.", "primary_pollutant": "NOx (nitrogen oxides)", "public_revelation_period": "2015-09", "estimated_scope": { "vehicles_worldwide": { "approx_count": 11000000, "notes": "Commonly reported estimate across VW Group brands; exact totals vary by jurisdiction, engine family, and reporting period." }, ``` `valid_json`

Pass
`less_than_length`: Ensures that the output is under a given number of characters. Write a one-line plain-English summary of arbitration Arbitration is a way to settle a dispute by having a neutral third person hear both sides and make a binding decision, usually outside of court. `less_than_length` 120 characters

Fail
`valid_yaml` (unary): Ensures that the output is safe-loadable YAML. Generate a court filing checklist as YAML. ``` court_filing_checklist: metadata: jurisdiction: "TBD (e.g., State/County or Federal District)" court: "TBD" division_department: "TBD" case_number: "TBD (leave blank if initiating case)" case_caption: plaintiff_petitioner: "TBD" defendant_respondent: "TBD" filing_type: "TBD (e.g., Complaint, Answer, Motion, Petition, Notice) hearing_date_time: "TBD (if applicable)" ``` `valid_yaml`

Pass
### Consistent With Source Operators `consistent_with_context` (unary): If the “Context” feature is used, this ensures that the LLM output does not contradict the information within the context provided. A client asks for a suitable investment strategy. Respond based on their profile. The client has a conservative risk profile and a 5-year investment horizon. Here are baseline strategies by their profiles (using low-cost diversified funds/ETFs as building blocks). These are educational examples, not personalized advice: 1. Conservative (short horizon: 0–5 years, low drawdown tolerance) * Allocation: 10–30% global stocks / 70–90% high-quality bonds + cash * Approach: prioritize capital preservation; use short/intermediate-duration high-quality bonds; keep near-term spending in cash-like instruments * Maintenance: rebalance annually; avoid chasing yield... `consistent_with_context`

Pass
`consistent_with_docs` (unary): If any files are uploaded, this ensures that the LLM output does not contradict information contained with the files. Based on the uploaded ruling, what damages were awarded to the plaintiff? 16-402 Carpenter v. United States (06/22/2018).pdf No damages were awarded.
The ruling you uploaded is Carpenter v. United States (2018), a criminal Fourth Amendment case (a suppression/warrant issue), not a civil lawsuit for money damages... `consistent_with_docs`

Pass
## Example Operator Usage You can see example checks for every operator by navigating to the test suites page, clicking "Import Quick Start Test Suite", then choosing Suite Family: "Basic Examples" and "operator\_examples". Example Operator ## Custom Operators We also provide the option for users to create their own custom operators. This is useful for when you want more flexibility behind the prompts you are evaluating checks with. These operators can be accessed from within the setting page where you can then create one which will be only available from the project in which it was created from. Here is an example of what it would look like to create an operator from within the platform: Custom Operator Example > Be specific, as the model will only know what you provide it inside of the prompt. We have a limited amount of variables that you can pass into your prompt. ### Variables | Variable | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------- | | `{{input}}` | The input provided to the LLM under test. Usually contains a question or statement you want the model to respond to | | `{{output}}` | The output from the model that was returned after responding to the input | | `{{criteria}}` | The criteria from the check that is defined from within a test | | `{{context.input}}` | The input context that is defined from within the test that is shared between all checks | | `{{context.output}}` | The output context contains additional information about the model such as its reasoning | | `{{files}}` | All the files that were added to the test passed into the prompt defined by the user | ### Context Variables You can pass in context to your prompt by accessing the value such as `{{context.input.date}}` or `{{context.output.user_email}}`. These fields come from the context that you define inside of the test. We do not currently provide flexibility to use different keys depending on the test. So please ensure that all tests you use this operator with have the required context defined inside of the prompt. Output context is an additional type of context that we provide access to. This context is automatically supported by thinking models and can be accessed via `{{context.output.reasoning}}`. If you want to use any other output context, you can define it inside of the question answer pair csv when you import as described [here](/web_app/import_export). ### Files When passing in the files we will convert the respective documents into chunks. If the context window of the model is too small for the text, we will shorten the documents automatically. You can access the chunks of the files by accessing `{{files}}` in your prompt. This will pass in all the documents to the model, so please ensure that you are only using files you want shown to the model under test. ### SDK Usage Additionally, you can use these operators from the sdk by simply referencing the name of the operator. You can find an example of that [here](/sdk/running_suites#4-provide-custom-operators). ## Additional Information All runs are versioned and cannot be changed while in progress. Once you start a run using a custom operator, even if you modify the operator, the run will continue using the operator it was run with. In times of a sudden need to change the operator, we suggest you stop the run in progress and start a new one to apply the changes to the custom operator. # Viewing Results Source: https://docs.vals.ai/web_app/results ## Table All results are displayed in the results tab in the table. Results (Data) | Feature | Options/Actions | | ------------------- | ------------------------------------------------- | | **Filter by** | `Model` `Status` `Test Suite` `Run By` `Archived` | | **Search Includes** | `Run Name` `Test Suite Name` | | **Sort by** | `Pass Rate` `Run Date` | | **Show Columns** | Toggle visibility of individual table columns | There are a few actions available when you select one or more runs from the table: * **Compare**: Compare the results of two selected runs. They must be from the same test suite. * **Archive**: Archive a selected run. It will no longer be displayed unless the `Archived` filter is selected * **Export**: Export one or more selected runs to a CSV file. ## Single Run Clicking on a row will bring you to the single run page. Results (Single) On the left, for each test result, it shows input, output, check results, and other metrics The right contains top-level information about the run - including statistics, error analysis, and run parameters. ### Filtering | Feature | Options/Actions | | -------------------------- | ---------------------------------------------------- | | **Filter by Check Status** | `Checks Passed` `Checks Failed` `Some Checks Failed` | | **Filter by Attributes** | `Tag` `Has Output Error` `Has Low Confidence Checks` | | **Search Includes** | `Input` `Output` `Checks` `LLM Feedback` `Context` | ### Confidence Confidence (Single) By default, the Vals system will compute a confidence for every test - either "High" or "Low". If the confidence is "High", this means our system has flagged that it is very likely we've graded this output correctly. If the confidence is "Low", then it means there is greater uncertainty or ambiguity in either the grading or in the criteria. The confidence score is listed in each check next to the grade. ### Statistics Several statistics are reported for every run. Statistics (Single) * **Success Rate**: This is the percentage of tests in which all tests passed. For example, lets say there are four tests, each with two checks each. The first two get 2/2 points, the third gets 1/2 points, and the last gets 0/2. The success rate is 50% - 2/4 tests had all their checks pass. * **Pass Rate**: This is the raw percentage of checks that passed - regardless of which test they belong to. So, if there are 8 checks across 4 tests, and 6 passed, then the pass rate is 75%. * **Low Confidence Rate**: This is the percentage of tests that the system flagged as low confidence. Additionally, each check is by default given a certain category: e.g. "Correctness", "Format", "Style", etc. The run result page will also show a pass percentage for each category (note: the categories can be overridden). Finally, if tags are assigned to each test, it will show a performance breakdown by tag. ## Comparisons You can choose to compare two runs either through the table or when looking at a single run. This allows you to see the results of two runs side by side - this is commonly done to see the differences between two models. Results (Comparison) We automatically compute several statistics, such as the likelihood of a statistically significant difference between the two runs. Toggle `Hide tests with matching auto-eval scores` to show only tests that were marked differently by the auto-grader. # Right Answers Source: https://docs.vals.ai/web_app/right_answers Generally, our platform works by defining an input, and a set of operator/criteria pairs. However, there's also another way to do evaluations, where you define an input and a "right answer". You compare the right answer to the LLM output and see if it matches. We support this workflow in the platform. ## Using Right Answers in the Web App To add a Right Answer, in the Add Test pane, press "Add Right Answer". Right Answer Example ## How Right Answer Comparison Works First, you add a right answer to each test (see below how to this in the web app and SDK). **Important**: When you add a Right Answer, our system automatically generates **three checks** for every test: * `golden_check_content` - Content similarity comparison * `golden_check_style` - Style similarity comparison * `golden_check_format` - Format structure comparison These will appear in addition to any manual checks you've defined. When you run the test suite, our system will then compare the LLM output to your right answer on three axes: * **Content**: It will compare to the golden answer based on the content similarity - passing if the contents are similar, failing if they are not. * **Style**: It will do a comparison based on the style of the text, passing only if the styles are similar. * **Format**: It will ensure that the outputs are formatted the same. # Running Test Suites Source: https://docs.vals.ai/web_app/run Once you have created a test suite, you will want to use it to actually test your model. There are a couple ways to do this; [Running against a base LLM](#running-against-a-base-llm) and [Running by uploading custom Question/Answer CSV](#running-by-uploading-custom-question-answer-csv) You can provide a `Run name` for each run - this is useful to keep track between many different runs. ## Running against a base LLM If you want to run a test suite against a stock model (e.g. base gpt-4o), you can do so by clicking the `Start Run` button on the upper-right of the test-suite page. Run When run, it will generate outputs from the model you choose, and then evaluate checks. > Note: By default, we only make a subset of models available to users. If you're interested in using additional models, including those hosted on Bedrock and Azure, please reach out to the Vals team. **Model Parameters**: * **Model**: The model that is used to produce the outputs * **Temperature**: The temperature of the model being tested * **Max Tokens**: The maximum number of tokens the model being tested can output * **System Prompt**: The system prompt to be passed to the model being tested ## Running by uploading custom Question/Answer CSV If you have the outputs from your model collected already, and just want to run the tests against them, you can do so by choosing the "Upload CSV" option. Run ### CSV Format The CSV should contain the questions and your model's corresponding outputs. Only `Question` and `Answer` are required. Any additional columns will be treated as context key-value pairs (the column name becomes the key, each cell becomes the value). 📎 [View Example](https://drive.google.com/file/d/1YDoP17C_fElk5Z6D_j9Lly7lrsTQqXqv/view) | Column | Required | Description | | ---------- | -------- | ------------------------------------------- | | Question | ✓ | The prompt or test input | | Answer | ✓ | Your model's output for that prompt | | In Tokens | — | Number of input tokens | | Out Tokens | — | Number of output tokens | | Duration | — | Time taken to generate the answer (seconds) | Vals will match each row against your suite's tests and run the defined checks against the uploaded outputs. You can also specify a model name to record which model produced the outputs. ## Parameters There are a few parameters that you can set to control how the run works. Parameters **Evaluation Parameters**: * **Evaluation Model**: Our evaluation suites use LLMs behind the scenes for checks like includes. The evaluation model you select from this dropdown will grade the outputs of the testing model. * **Parallelism**: This controls how many tests are running at any run time. Set it to a higher number to make the run finish faster. Set it to a lower number depending on rate limits. * **Eval Concurrency**: Controls how many checks are evaluated in parallel. Defaults to the same value as Parallelism. * **Heavyweight Factor**: Runs each check multiple times to reduce variance in the results. For example, if it was set to 5, each check would be run 5 times, and the final result would be the mode. * **Retry Failed Calls Indefinitely**: Retry model calls until they succeed. * **Run Confidence Evaluation** `DEFAULT`: Compute and display a confidence score (high or low) for each check. * **Create Text Summary** `DEFAULT`: Create summary of the run. * **Run Right Answer Comparison** `DEFAULT`: Run right answer comparison. * **Run Refusal Detection**: Compute and display refusal rate (model refused to respond). * **Use Batch API**: Use (if supported) the provider's batch API to run model calls. ## Run using the SDK If you have the model defined locally in a python script, you can use the SDK to produce the outputs. See the [SDK docs](/sdk/running_suites) for more information.