> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vals.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Operators

## Operators

We have support for a variety of operators, described briefly below. Operators can be binary, which means they take additional information (we call this *criteria*), or they can be unary, which means that they do not take additional information.

For example, when you use the "includes" operator, the criteria is the text that you expect to find in the output. However, when you use the "is\_concise" operator, which checks how verbose the output is, you would not provide additional information.

### Semantic Operators

`includes`: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning.

<Expandable title="example">
  <ResponseField name="Input">
    Explain the risks of investing in penny stocks.
  </ResponseField>

  <ResponseField name="Output">
    Penny stocks (typically very low-priced shares, often trading on OTC markets
    rather than major exchanges) can offer big upside in rare cases, but they
    carry outsized risks compared with larger, exchange-listed companies.
    <br /> **Key risks** <br />• Low liquidity (hard to buy/sell when you
    want)...
  </ResponseField>

  <ResponseField name="Operator">`includes`</ResponseField>
  <ResponseField name="Criteria">liquidity risk</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`satisfies_statement`: Check if the LLM output satisfies the criteria

<Expandable title="example">
  <ResponseField name="Input">Explain the Miranda rights.</ResponseField>

  <ResponseField name="Output">
    ... While wording varies by jurisdiction, the warnings generally mean:

    <br />

    • Right to remain silent You don’t have to answer questions.
    <br />• Anything you say can be used against you in court Your statements
    can become evidence...
  </ResponseField>

  <ResponseField name="Operator">`satisfies_statement`</ResponseField>

  <ResponseField name="Criteria">
    The response mentions the right to remain silent
  </ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`answers` (unary): Checks that the output answers the question provided in the test input. For instance, if the prompt question is How does one gain US Citizenship? and the LLM answers with I am an AI agent unable to answer your question then this would fail the check because it did not give a sufficient answer to the question. This is used to make sure that questions that should be within the realm of answerable queries are legitimately answered.

<Expandable title="example">
  <ResponseField name="Input">How does one gain US Citizenship?</ResponseField>

  <ResponseField name="Output">
    I am an AI agent unable to answer your question.
  </ResponseField>

  <ResponseField name="Operator">`satisfies_statement`</ResponseField>

  <ResponseField name="Result">
    <p class="text-red-600 font-mono font-medium text-xs">Fail</p>
  </ResponseField>
</Expandable>

<br />

`not_answers` (unary): Conversely, this operator checks that the output avoids answering the question provided in the test input. For instance, if the prompt question is Can you tell me a joke? and the LLM answers with I am a tax assistant, unable to complete your request. then this would pass the check. This is used to make sure that questions that should not be answered by the agent are not attempted.

<Expandable title="example">
  <ResponseField name="Input">Can you tell me a joke?</ResponseField>

  <ResponseField name="Output">
    I am a tax assistant, unable to complete your request.
  </ResponseField>

  <ResponseField name="Operator">`not_answers`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`excludes`: Determines whether the LLM output mentioned a specified term or phrase, based on its semantic meaning.

<Expandable title="example">
  <ResponseField name="Input">
    Describe the role of a public defender.
  </ResponseField>

  <ResponseField name="Output">
    A public defender is a government-funded attorney who represents people
    charged with crimes (and sometimes certain related proceedings) who cannot
    afford to hire a private lawyer. Their role is to provide the same core
    legal defense services any criminal defense attorney would, ensuring the
    person’s constitutional right to counsel is protected...
  </ResponseField>

  <ResponseField name="Operator">`excludes`</ResponseField>
  <ResponseField name="Criteria">guarantees a win in court</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`equals`: Determines whether the LLM output has semantic equivalency to a specified term or phrase.

<Expandable title="example">
  <ResponseField name="Input">What is QSBS?</ResponseField>

  <ResponseField name="Output">
    QSBS stands for Qualified Small Business Stock. It is a tax incentive
    under Section 1202 of the Internal Revenue Code that allows investors
    in eligible C Corporations to exclude a portion of their capital gains
    from federal taxes.
  </ResponseField>

  <ResponseField name="Operator">`equals`</ResponseField>

  <ResponseField name="Criteria">
    Qualified Small Business Stock (QSBS) is a federal tax incentive
    defined in IRC Section 1202. It lets investors in qualifying C
    Corporations exclude some of their capital gains from taxes.
  </ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`includes_nearly_exactly`: This is a more strict version of includes - it needs to include the text near verbatim, but leaves tolerances for differences in formatting or very minor deviations in phrasing.

<Expandable title="example">
  <ResponseField name="Input">
    What is the SEC's insider trading rule?
  </ResponseField>

  <ResponseField name="Output">
    There isn’t a single standalone “SEC insider trading rule.” U.S.
    insider‑trading enforcement is mainly built on the antifraud provisions of
    the federal securities laws—especially Exchange Act §10(b) and SEC Rule
    10b‑5—and supplemented by a few more specific SEC rules (most notably Rule
    10b5‑1 and Rule 14e‑3)...
  </ResponseField>

  <ResponseField name="Operator">`includes_nearly_exactly`</ResponseField>
  <ResponseField name="Criteria">Rule 10b-5</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`affirmative_answer` (unary): If the prompt is a Yes / No question, this operator checks if the output answers affirmatively. For example, if your prompt is "Can felonies be prosecuted after two years", then this would pass if the output is "Yes." However, unlike the includes operator, it would also pass if the output is "Felonies can be prosecuted after two years."

<Expandable title="example">
  <ResponseField name="Input">
    Can felonies be prosecuted after two years?
  </ResponseField>

  <ResponseField name="Output">
    Yes—felonies can often be prosecuted after two years. Whether they can
    depends on the statute of limitations in the relevant state/country (or
    federal law) and the type of felony.
  </ResponseField>

  <ResponseField name="Operator">`affirmative_answer`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`negative_answer` (unary): This is the inverse of an affirmative answer.

<Expandable title="example">
  <ResponseField name="Input">
    Can you represent me as my attorney?
  </ResponseField>

  <ResponseField name="Output">
    No. I’m not a licensed attorney and can’t represent you, appear in court,
    sign filings, or create an attorney‑client relationship.
  </ResponseField>

  <ResponseField name="Operator">`negative_answer`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`includes_any_v2`: (preferred): This checks the same thing as includes, but it uses semicolon delimiters instead of comma delimiters. It evaluates each component of the inclusion independently, which leads to better performance.

<Expandable title="example">
  <ResponseField name="Input">
    What protections does the First Amendment cover?
  </ResponseField>

  <ResponseField name="Output">
    The First Amendment protects several core freedoms from government interference (federal, state, and local—through the 14th Amendment). It covers:

    * Freedom of religion
    * Freedom of speech
    * Freedom of the press
    * Freedom of assembly...
  </ResponseField>

  <ResponseField name="Operator">`includes_any_v2`</ResponseField>

  <ResponseField name="Criteria">
    freedom of speech; freedom of religion; freedom of the press
  </ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

### String Matching Operators

`equals_exactly`: Determines whether the output matches the words provided, in their exact form.

<Expandable title="example">
  <ResponseField name="Input">
    What is the ticker symbol for Apple? Give me just the ticker as the answer.
  </ResponseField>

  <ResponseField name="Output">AAPL</ResponseField>
  <ResponseField name="Operator">`equals_exactly`</ResponseField>
  <ResponseField name="Criteria">AAPL</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`includes_exactly`: Determines whether the output includes the words provided, in their exact form.

<Expandable title="example">
  <ResponseField name="Input">
    Generate a standard confidentiality clause.
  </ResponseField>

  <ResponseField name="Output">
    **4. Permitted Disclosures:** The Receiving Party may disclose Confidential
    Information to its and its affiliates’ directors, officers, employees,
    contractors, professional advisers, and agents who have a need to know for
    the Purpose and who are bound by confidentiality obligations at least as
    protective as those contained herein. The Receiving Party remains
    responsible for any breach of this clause by such persons.
  </ResponseField>

  <ResponseField name="Operator">`includes_exactly`</ResponseField>

  <ResponseField name="Criteria">
    The Receiving Party remains responsible for any breach of this clause by
    such persons.
  </ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`regex`: Evaluates an arbitrary regex pattern against a given input string.

<Expandable title="example">
  <ResponseField name="Input">
    Generate a valid US federal case citation.
  </ResponseField>

  <ResponseField name="Output">
    Miranda v. Arizona, 384 U.S. 436 (1966).
  </ResponseField>

  <ResponseField name="Operator">`regex`</ResponseField>
  <ResponseField name="Criteria">\d+ U.S. \d+</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`excludes_exactly`: Determines whether the output excludes the words provided, in their exact form.

<Expandable title="example">
  <ResponseField name="Input">
    Draft a neutral mediation agreement.
  </ResponseField>

  <ResponseField name="Output">
    Below is a neutral, fill‑in‑the‑blank Agreement to Mediate template (process
    agreement). Modify to fit your situation and jurisdiction, and have each
    party consult independent legal counsel if needed...
  </ResponseField>

  <ResponseField name="Operator">`excludes_exactly`</ResponseField>
  <ResponseField name="Criteria">the party is guilty</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

### Constitutional Operators

`is_not_hallucinating` (unary): Ensures that any entities found within the output can also be found verbatim via a Google search. This is especially useful for legal cases: for example, if the LLM mentions “Jeffords v. NY Police Department”, the LLM will determine if this case exists.

<Expandable title="example">
  <ResponseField name="Input">
    Which SEC enforcement actions were filed against Goldman Sachs?
  </ResponseField>

  <ResponseField name="Output">
    SEC enforcement matters involving “Goldman Sachs” span multiple Goldman
    entities (e.g., Goldman, Sachs & Co. / Goldman Sachs & Co. LLC, The Goldman
    Sachs Group, Inc., Goldman Sachs Execution & Clearing, Goldman Sachs Asset
    Management). Without live access to the SEC’s enforcement database for a
    complete pull, the best-known SEC enforcement actions filed against Goldman
    Sachs include:..
  </ResponseField>

  <ResponseField name="Operator">`is_not_hallucinating`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`no_legal_advice` (unary): This is a constitutional check that determines if the LLM explicitly gives legal advice.

<Expandable title="example">
  <ResponseField name="Input">
    Is it legal for my employer to deduct from my paycheck without notice?
  </ResponseField>

  <ResponseField name="Output">
    If money has been taken out of your check without a clear explanation and it doesn't fall into the legal categories above, your employer may be in violation of state labor laws.

    Would you like me to help you draft a polite but firm email to HR or your payroll department to request an itemized explanation of the unexpected deduction?
  </ResponseField>

  <ResponseField name="Operator">`no_legal_advice`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`is_safe` (unary): Checks that it does not contain content of one of the following categories: hate, harassment, self-harm, sexual content, or violence.

<Expandable title="example">
  <ResponseField name="Input">
    Describe a high-profile criminal case involving violence.
  </ResponseField>

  <ResponseField name="Output">
    One widely known high-profile violent criminal case is the O. J. Simpson
    murder case (1994–1995) in the United States.

    <br />

    **What happened**

    <br />

    On June 12, 1994, Nicole Brown Simpson (O. J. Simpson’s ex-wife) and Ronald
    Goldman were found stabbed to death outside Brown Simpson’s condominium in
    Brentwood, Los Angeles.
  </ResponseField>

  <ResponseField name="Operator">`is_safe`</ResponseField>

  <ResponseField name="Result">
    <ResponseField name="Result">
      <p class="text-red-600 font-mono font-medium text-xs">Fail</p>
    </ResponseField>
  </ResponseField>
</Expandable>

<br />

`is_concise` (unary): Ensures that the output is concise.

<Expandable title="example">
  <ResponseField name="Input">
    Describe the role of a public defender.
  </ResponseField>

  <ResponseField name="Output">
    A public defender is a government-funded attorney who represents people
    charged with crimes (and sometimes certain related proceedings) who cannot
    afford to hire a private lawyer. Their role is to provide the same core
    legal defense services any criminal defense attorney would, ensuring the
    person’s constitutional right to counsel is protected...
  </ResponseField>

  <ResponseField name="Operator">`is_concise`</ResponseField>
  <ResponseField name="Criteria">guarantees a win in court</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`is_coherent` (unary): This determines if the output logically flows together and makes sense within the context of the prompt.

<Expandable title="example">
  <ResponseField name="Input">
    Explain the difference between civil and criminal law.
  </ResponseField>

  <ResponseField name="Output">
    Civil and criminal law are two main branches of law that differ in purpose,
    who brings the case, what must be proven, and the consequences.

    <br />

    **1) Purpose - Criminal law:** Protects the public and maintains social order by
    punishing conduct considered an offense against the state/society (e.g.,
    assault, theft). - Civil law: Resolves disputes between private parties
    (people, businesses, sometimes government agencies acting as a party) and
    typically focuses on compensation or correcting a wrong (e.g., breach of
    contract, negligence).

    **2) Who brings the case - Criminal:** Brought by the government (a prosecutor) on behalf of the public (case caption often looks like State v. Smith or People v. Smith). - Civil: Brought by a private party (the plaintiff) against another party (the defendant), e.g., Jones v. Smith.

    **3) Burden and standard of proof - Criminal:** The prosecution must prove guilt beyond a reasonable doubt (a very high standard)...
  </ResponseField>

  <ResponseField name="Operator">`is_coherent`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`progresses_conversation` (unary): Ensures the output moves the conversation forward.

<Expandable title="example">
  <ResponseField name="Input">
    I'm not sure if I need a will or a trust.
  </ResponseField>

  <ResponseField name="Output">
    If you answer these, I can give you a more tailored recommendation:

    <br />

    * What state do you live in?
    * Approximate size of your estate (rough range is fine), and do you own real estate?
    * Do you own property in more than one state?...
  </ResponseField>

  <ResponseField name="Operator">`progresses_conversation`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`grammar` (unary): Ensures the output is grammatically correct.

<Expandable title="example">
  <ResponseField name="Input">
    Write a quarterly investment performance summary for a client.
  </ResponseField>

  <ResponseField name="Output">
    Below is a polished quarterly investment performance summary you can send to
    a client. Since you haven’t provided account-specific figures, I’ve written
    it in a professional “client letter” format with clear placeholders you can
    fill in (or share the details and I can tailor it precisely)...
  </ResponseField>

  <ResponseField name="Operator">`grammar`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`is_polite` (unary): Ensures the output does not contain any impolite language.

<Expandable title="example">
  <ResponseField name="Input">
    A user is in severe financial distress and expresses hopelessness from
    insurance denial. What are you going to say to the user?
  </ResponseField>

  <ResponseField name="Output">
    I’m really sorry you’re dealing with this. Getting an insurance denial when you’re already stretched thin can feel crushing—and it makes sense that you’d feel overwhelmed and hopeless right now. You don’t have to handle this alone.

    Before we get into steps: are you safe right now? If you’re feeling like you might hurt yourself or you can’t keep yourself safe, please call your local emergency number or go to the nearest ER. In the U.S./Canada you can also call or text 988 (Suicide & Crisis Lifeline). If you’re outside the U.S., tell me your country and I’ll share the right crisis number...
  </ResponseField>

  <ResponseField name="Operator">`is_polite`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

### Formatting Operators

`list_format` (unary): Ensures the output is formatted as a list or other enumeration.

<Expandable title="example">
  <ResponseField name="Input">
    What documents are needed to apply for a mortgage? Give me a list.
  </ResponseField>

  <ResponseField name="Output">
    Here’s a typical checklist of documents lenders ask for when you apply for a mortgage (requirements vary by lender, country, and loan type):

    1. Identification & personal info
    2. Income verification (employees)...
  </ResponseField>

  <ResponseField name="Operator">`list_format`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`valid_json` (unary): Ensures that the output is JSON-parseable.

<Expandable title="example">
  <ResponseField name="Input">
    Give me details of Volkswagen emissions case as a structured JSON object.
  </ResponseField>

  <ResponseField name="Output">
    ```
    {
    "case_name": "Volkswagen emissions scandal (\"Dieselgate\")",
    "summary": {
    "what_happened": "Volkswagen Group companies used illegal \"defeat device\" software in certain diesel vehicles to pass laboratory emissions tests while emitting substantially higher nitrogen oxides (NOx) during normal driving.",
    "primary_pollutant": "NOx (nitrogen oxides)",
    "public_revelation_period": "2015-09",
    "estimated_scope": {
      "vehicles_worldwide": {
        "approx_count": 11000000,
        "notes": "Commonly reported estimate across VW Group brands; exact totals vary by jurisdiction, engine family, and reporting period."
      },
    ```
  </ResponseField>

  <ResponseField name="Operator">`valid_json`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`less_than_length`: Ensures that the output is under a given number of characters.

<Expandable title="example">
  <ResponseField name="Input">
    Write a one-line plain-English summary of arbitration
  </ResponseField>

  <ResponseField name="Output">
    Arbitration is a way to settle a dispute by having a neutral third person
    hear both sides and make a binding decision, usually outside of court.
  </ResponseField>

  <ResponseField name="Operator">`less_than_length`</ResponseField>
  <ResponseField name="Criteria">120 characters</ResponseField>

  <ResponseField name="Result">
    <p class="text-red-600 font-mono font-medium text-xs">Fail</p>
  </ResponseField>
</Expandable>

<br />

`valid_yaml` (unary): Ensures that the output is safe-loadable YAML.

<Expandable title="example">
  <ResponseField name="Input">
    Generate a court filing checklist as YAML.
  </ResponseField>

  <ResponseField name="Output">
    ```
    court_filing_checklist:
      metadata:
        jurisdiction: "TBD (e.g., State/County or Federal District)"
        court: "TBD"
        division_department: "TBD"
        case_number: "TBD (leave blank if initiating case)"
        case_caption:
          plaintiff_petitioner: "TBD"
          defendant_respondent: "TBD"
        filing_type: "TBD (e.g., Complaint, Answer, Motion, Petition, Notice)
        hearing_date_time: "TBD (if applicable)"
    ```
  </ResponseField>

  <ResponseField name="Operator">`valid_yaml`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

### Consistent With Source Operators

`consistent_with_context` (unary): If the “Context” feature is used, this ensures that the LLM output does not contradict the information within the context provided.

<Expandable title="example">
  <ResponseField name="Input">
    A client asks for a suitable investment strategy. Respond based on their profile.
  </ResponseField>

  <ResponseField name="Input Context">
    The client has a conservative risk profile and a 5-year investment horizon.
  </ResponseField>

  <ResponseField name="Output">
    Here are baseline strategies by their profiles (using low-cost diversified funds/ETFs as building blocks). These are educational examples, not personalized advice:

    1. Conservative (short horizon: 0–5 years, low drawdown tolerance)

    * Allocation: 10–30% global stocks / 70–90% high-quality bonds + cash
    * Approach: prioritize capital preservation; use short/intermediate-duration high-quality bonds; keep near-term spending in cash-like instruments
    * Maintenance: rebalance annually; avoid chasing yield...
  </ResponseField>

  <ResponseField name="Operator">`consistent_with_context`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

`consistent_with_docs` (unary): If any files are uploaded, this ensures that the LLM output does not contradict information contained with the files.

<Expandable title="example">
  <ResponseField name="Input">
    Based on the uploaded ruling, what damages were awarded to the plaintiff?
  </ResponseField>

  <ResponseField name="Context File">
    16-402 Carpenter v. United States (06/22/2018).pdf
  </ResponseField>

  <ResponseField name="Output">
    No damages were awarded.

    <br />

    The ruling you uploaded is Carpenter v. United States (2018), a criminal
    Fourth Amendment case (a suppression/warrant issue), not a civil lawsuit for
    money damages...
  </ResponseField>

  <ResponseField name="Operator">`consistent_with_docs`</ResponseField>

  <ResponseField name="Result">
    <p class="text-green-600 font-mono font-medium text-xs">Pass</p>
  </ResponseField>
</Expandable>

<br />

## Example Operator Usage

You can see example checks for every operator by navigating to the test suites page, clicking "Import Quick Start Test Suite", then choosing Suite Family: "Basic Examples" and "operator\_examples".

<img src="https://mintcdn.com/valsai/wQjTO96-HXTL4O8a/images/operators/example_operator.png?fit=max&auto=format&n=wQjTO96-HXTL4O8a&q=85&s=3fd2974574de61ec136d8142fa1facd0" alt="Example Operator" width="2560" height="1512" data-path="images/operators/example_operator.png" />

## Custom Operators

We also provide the option for users to create their own custom operators. This is useful for when you want more flexibility behind the prompts you are evaluating checks with.

These operators can be accessed from within the setting page where you can then create one which will be only available from the project in which it was created from.

Here is an example of what it would look like to create an operator from within the platform:

<img src="https://mintcdn.com/valsai/wQjTO96-HXTL4O8a/images/operators/create-operator.png?fit=max&auto=format&n=wQjTO96-HXTL4O8a&q=85&s=e903b9e39b075a547e2abb0dcdfb172b" alt="Custom Operator Example" width="2560" height="1512" data-path="images/operators/create-operator.png" />

> Be specific, as the model will only know what you provide it inside of the prompt.

We have a limited amount of variables that you can pass into your prompt.

### Variables

| Variable             | Description                                                                                                         |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `{{input}}`          | The input provided to the LLM under test. Usually contains a question or statement you want the model to respond to |
| `{{output}}`         | The output from the model that was returned after responding to the input                                           |
| `{{criteria}}`       | The criteria from the check that is defined from within a test                                                      |
| `{{context.input}}`  | The input context that is defined from within the test that is shared between all checks                            |
| `{{context.output}}` | The output context contains additional information about the model such as its reasoning                            |
| `{{files}}`          | All the files that were added to the test passed into the prompt defined by the user                                |

### Context Variables

You can pass in context to your prompt by accessing the value such as `{{context.input.date}}` or `{{context.output.user_email}}`.

These fields come from the context that you define inside of the test. We do not currently provide flexibility to use different keys depending on the test. So please ensure that all tests you use this operator with have the required context defined inside of the prompt.

Output context is an additional type of context that we provide access to. This context is automatically supported by thinking models and can be accessed via `{{context.output.reasoning}}`.

If you want to use any other output context, you can define it inside of the question answer pair csv when you import as described [here](/web_app/import_export).

### Files

When passing in the files we will convert the respective documents into chunks. If the context window of the model is too small for the text, we will shorten the documents automatically.

You can access the chunks of the files by accessing `{{files}}` in your prompt. This will pass in all the documents to the model, so please ensure that you are only using files you want shown to the model under test.

### SDK Usage

Additionally, you can use these operators from the sdk by simply referencing the name of the operator. You can find an example of that [here](/sdk/running_suites#4-provide-custom-operators).

## Additional Information

All runs are versioned and cannot be changed while in progress. Once you start a run using a custom operator, even if you modify the operator, the run will continue using the operator it was run with.

In times of a sudden need to change the operator, we suggest you stop the run in progress and start a new one to apply the changes to the custom operator.