Check Modifiers

Various modifiers can be used to alter the way a given check and operator run. These provide additional ways to customize the way your tests work. You can specify modifiers by clicking the Advanced Options from the Add Test dialog in the test suite view. You can apply all modifiers to a single check. Modifiers also work with global checks.

​

Conditional Checks

Sometimes, you want to run a check only if a condition is true. For example, let’s say that if your LLM makes reference to Law X, you always want it to also mention Addendum Y. However, if Law X is not mentioned, there is no need to check for Addendum Y. This can be accomplished with what we call a conditional check. If the conditional check passes, your test is run as normal. However, if the predicate check fails, the test passes. To accomplish the example above with a conditional check, you would define your check as includes | Addendum Y as normal. However, you would also add an additional predicate check: includes | Law X. For those interested in boolean logic, the truth table is as follows:

​

In-Context Examples

Similar to prompt engineering, it can be helpful to provide additional examples of outputs that should fail or pass the check. You can do this with our In-Context Examples feature. From the Advanced Options in the Checks container, select Positive or Negative to provide the appropriate example.

​

Extraction Prompt

Often evaluation needs to be done in more than a single step. An output must be generated, but a secondary “read” of the output needs to produce the content that will be evaluated. These checks can be done using the extractor modifier. You use it by specifying the extractor word or key phrase used to prompt an LLM to generate an answer extraction. This is what will be used in the check’s evaluation. For instance, a user who is building an LLM application that produces a table may want to check that it is formatted correctly by first extracting the column names, then checking that it exactly matches the expected list of column names. This user would define an extractor with a criteria like “the columns from the generated table.” If there is only a single answer that can be extracted, then this is what will be retrieved.

​

Severity

Severity allows you to weight some checks higher than others. A check with a higher severity will have a larger impact on the overall pass percentage. For example, if you have a critical check that must pass, you can give it a higher severity than other checks.

​

Category

Category allows you to override the default category of a check (e.g., correctness, formatting). This is useful for organizing checks and viewing performance breakdowns by category in the run results.

​

Optional

Marking a check as optional means it will not count towards the final pass percentage. This is useful for informational checks that you want to track but don’t want to affect the overall score.

​

Display Metrics

If enabled, the metrics for this check will be displayed in the UI. This is useful for checks where you want to surface specific measurement data.