​
Operators

We have support for a variety of operators, described briefly below. Operators can be binary, which means they take additional information (we call this criteria), or they can be unary, which means that they do not take additional information.

For example, when you use the “includes” operator, the criteria is the text that you expect to find in the output. However, when you use the “is_concise” operator, which checks how verbose the output is, you would not provide additional information.

​
Semantic Operators

includes: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning. (default severity: 1).

satisfies_statement: Check if the LLM output satisfies the criteria

answers (unary): Checks that the output answers the question provided in the test input. For instance, if the prompt question is How does one gain US Citizenship? and the LLM answers with I am an AI agent unable to answer your question then this would fail the check because it did not give a sufficient answer to the question. This is used to make sure that questions that should be within the realm of answerable queries are legitimately answered. (default severity: 1)

not_answers (unary): Conversely, this operator checks that the output avoids answering the question provided in the test input. For instance, if the prompt question is Can you tell me a joke? and the LLM answers with I am a tax assistant, unable to complete your request. then this would pass the check. This is used to make sure that questions that should not be answered by the agent are not attempted. (default severity: 1)

excludes: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning. (default severity: 1).

equals: Determines whether the LLM output has semantic equivalency to a specified term or phrase. (default severity: 1)

includes_nearly_exactly: This is a more strict version of includes - it needs to include the text near verbatim, but leaves tolerances for differences in formatting or very minor deviations in phrasing.

negative_answer (unary): This is the inverse of an affirmative answer.

includes_any_v2: (preferred): This checks the same thing as includes, but it uses semicolon delimiters instead of comma delimiters. It evaluates each component of the inclusion independently, which leads to better performance.

​
String Matching Operators

equals_exactly: Determines whether the output matches the words provided, in their exact form.

includes_exactly: Determines whether the output includes the words provided, in their exact form.

regex: Evaluates an arbitrary regex pattern against a given input string.

excludes_exactly: Determines whether the output excludes the words provided, in their exact form.

​
Constitutional Operators

is_not_hallucinating (unary): Ensures that any entities found within the output can also be found verbatim via a Google search. This is especially useful for legal cases: for example, if the LLM mentions “Jeffords v. NY Police Department”, the LLM will determine if this case exists.

no_legal_advice (unary): This is a constitutional check that determines if the LLM explicitly gives legal advice.

is_safe (unary): Checks that it does not contain content of one of the following categories: hate, harassment, self-harm, sexual content, or violence.

is_concise (unary): Ensures that the output is concise.

is_coherent (unary): This determines if the output logically flows together and makes sense within the context of the prompt.

progresses_conversation (unary): Ensures the output moves the conversation forward.

grammar (unary): Ensures the output is gramatically correct.

is_polite (unary): Ensures the output does not contain any impolite language.

​
Formatting Operators

list_format (unary): Ensures the output is formatted as a list or other enumeration.

valid_json (unary): Ensures that the output is JSON-parseable.

less_than_length: Ensures that the output is under a given number of characters.

valid_yaml (unary): Ensures that the output is YAML parseable.

​
Consistent With Source Operators

consistent_with_context (unary): If the “Context” feature is used, this ensures that the LLM output does not contradict the information within the context provided.

consistent_with_docs (unary): If any files are uploaded, this ensures that the LLM output does not contradict information contained with the files.

​
Example Operator Usage

You can see example checks for every operator by navigating to the test suites page, clicking “Import Quick Start Test Suite”, then choosing Suite Family: “Basic Examples” and “operator_examples”.