> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vals.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Glossary

## Tests

**Test Suite** - A grouping of one or more **tests** –- useful for keeping related tests organized

**Test** - An **input** combined with a grouping of **checks** against which an **output** will be evaluated

**Input** - A question posed to the LLM (aka **question**)

**Output** - The LLM’s response to the **input** (aka **answer**)

**Check** - An **operator/criteria** pair used to evaluate whether the output meets certain expectations

**Operator** - One of `includes`, `excludes`, etc. This is either `unary` like `is_safe` or `binary` (meaning it takes a criteria filled in by the user). Each operator is a tool we've defined to check one specific item in the **output**

**Criteria** - Free text filled in by the user that the **operator** will be evaluated against

**Right answer** - An alternative way to evaluate an output. The user writes the “right answer” to a given prompt/question and we compare that right answer to the answer the LLM produces.

## Platform

**Custom operators** - A custom operator you define via prompt engineering. Can be configured on the settings page.

**LLM (Large Language Model)** - The underlying model used to generate responses.

**LLM as Judge** - A process where one LLM evaluates the output LLM.

**SME (Subject matter expert)** - An expert in the domain under question (e.g. lawyer, CPA, etc.)

## Runs

**Run result** - The result of evaluating a test suite. A run has two phases:

1. Output gathering - The model generates responses for each test **input** to gather **outputs**
2. Evaluation - Each **check** is evaluated based on the **outputs**

**Test result** - The result of evaluating a single **test**

**Auto eval** - The result of grading an **output** against its respective **checks**

**Model** - The model used to produce **outputs**. Can either be a foundation model or your own custom model.

**Eval model** - Some of our **operators** employ LLM as judge. This is the model that will be used as judge.

**Run parameters** - These are the knobs available to you to tune either how the evaluation is conducted (evaluation parameters) or how the model responses are produced (model parameters).

**Heavyweight factor** - Each auto eval will be run this many times, and the result is the mode.

**Duration/Latency** - The time it takes for a model to produce a response.

**Confidence** - When evaluating benchmark performance, we use a standard large and very capable model as the primary judge for scoring responses. For confidence assessment, we employ a smaller, more efficient model that analyzes both the question and surrounding test context to verify the coherence of question-answer pairs, providing a reliability metric for our evaluation results. A high confidence indicates a high likelihood that the **check** has been scored correctly.

## Human Review

**Auto eval re-review (aka Single run review)** - Another pass by a human reviewer. Each of the **checks** posed to the **model** must be answered or flagged.

**Pairwise review** - Two **outputs** are compared side by side. The reviewer selects the better one.

**Human review template** - A user configured review question that will only be answered by a human reviewer. This allows you to evaluate **outputs** in ways that may be hard to codify solely using checks.