While developing the platform, we’ve found that certain tips for writing effective criteria.

​
1. Make it Human Readable

If a stranger would have a hard time understanding what your criteria means, then it will likely not perform well with the automatic evaluation system. Therefore, an easy litmus test you can use to see if your criteria is written well is to ask yourself, “If I showed this to my colleague, would they be able to understand what it means?“

​
2. Use Global Checks

If you find yourself adding the same operator to every test that you write, you can instead add it as a “Global Check”. This ensures that it will run automatically for every single input defined. This is common for unary operators, such as is_concise, or the operators checking for harassment.

​
3. Use Noun Phrases or Complete Sentences for Includes, Excludes, and Equals

For the includes, excludes, and equals operators, the criteria should be either a grammatical noun/noun phrase or a full sentence. If it’s a sentence fragment, it may not be parsed correctly. Here’s an example.

​
Avoid:

operating

​
Better:

A reference to operating the vehicle

​
Better:

The defendant was operating the vehicle at the time of the accident.

​
4. Split “AND” Statements into Multiple Checks

If you have a statement that contains “AND” as a conjunction, it is generally better to split it into two or more separate checks. Here is an example of a criteria with an AND clause, along with an example of it being decomposed into multiple separate criteria.

​
AVOID:

Includes: A reference to the fact that QSBS is a tax incentive, and that it applies to C Corporations.

​
BETTER:

Includes: QSBS is a tax incentive.

Includes: QSBS applies to C Corporations

​
5. No Relative or Implicit Clauses

Try to minimize the amount of context a human or AI agent reading your criteria would need to understand. This includes avoiding pronouns, avoiding relative clauses, and avoiding grammatical ambiguity. Here are some examples:

​
AVOID:

Includes: A reference to the statute

​
BETTER:

Includes: A reference to statute 83(b).

​
AVOID:

He was operating the vehicle while intoxicated.

​
BETTER:

The Defendant was operating the vehicle while intoxicated.

​
6. Use “Satisfies Statement” as a Catch-All Operator

If none of the existing operators suit your needs, you can use the satisfies_statement as a very customizable operator. For example, you may want to check that the output is formatted as a numbered list with exactly four bullet points. The two existing formatting operators don’t meet this requirement exactly. Therefore, you may use something like the following:

Satisfies Statement: The output is formatted as a numbered list with exactly four items.