​
Modifiers

Various modifiers can be used to alternate the way a given check and operator run. These provide additional ways to customize the way your tests.

​
Conditional Checks

Sometimes, you want to run a check only if a condition is true. For example, let’s say that if your LLM makes reference to Law X, you always want it to also mention Addendum Y. However, if Law X is not mentioned, there is no need to check for Addendum Y.

This can be accomplished with what we call a conditional check. If the conditional check passes, your test is run as normal. However, if the predicate check fails, the test passes.

To accomplish the example above with a conditional check, you would define your check as includes | Addendum Y as normal. However, you would also add an additional predicate check: includes | Law X.

For those interested in boolean logic, the truth table is as follows:

ConditionalBase CheckTest Pass
PassPassPass
PassFailFail
FailPassPass
FailFailPass

​
Positive and Negative Examples

Similar to when prompt engineering, it can be helpful to provide additional examples of outputs that should fail the check, and outputs that should pass the check. You can do this with our examples feature. In the Web App, press “Add Example” in Advanced Options.

​
Extractor

Often evaluation needs to be done in more than a single step. An output must be generated, but a secondary “read” of the output needs to produce the content that will be evaluated. These checks can be done using the extractor modifier. You use it by specifying the extractor word or key phrase used to prompt an LLM to generate an answer extraction. This is what will be used in the check’s evaluation.

For instance, a user who is building an LLM application that produces a table may want to check that it is formatted correctly by first extracting the column names, then checking that it exactly matches the expected list of column names. This user would define an extractor with a criteria like “the columns from the generated table.”

If there is only a single answer that can be extracted, then this is what will be retrieved.

​
Modifiers in the Web App

You can specify modifiers by clicking the “Advanced Options” from the Add Test Dialog in the test suite view.

​
Modifiers in the SDK Tool

To add a conditional check:

check = Check(
    operator="includes",
    criteria="...",
    modifiers=CheckModifiers(
        conditional=ConditionalCheck(
            operator="includes", 
            criteria="..."
        )
    )
)

To add examples:

check = Check(
    operator="includes",
    criteria="...",
    modifiers=CheckModifiers(
        examples = [
            Example(
                text="...",
                type="positive"
            )
        ]
    )
)

To add an extractor:

check = Check(
    operator="includes",
    criteria="...",
    modifiers=CheckModifiers(
        extractor="all section headers"
    )
)

You can apply all modifiers to a single check. Modifiers also work with global checks.