Skip to main content
Heuristic evaluators uses mathematical and logical formulas to aproximate if a response is correct or incorrect.

​
Why Heuristic Evaluators are Important

Heuristic evaluators are valuable because they:
  1. Consistency: Provide consistent evaluations across different runs and scenarios
  2. Speed: Execute quickly without requiring additional API calls
  3. Cost-Effective: Don’t require additional LLM API calls, making them more economical
However, there are some limitations:
  • Rigidity: May miss nuanced or context-dependent aspects of responses
  • Limited Scope: Can only evaluate what has been explicitly defined in the rules
  • Maintenance: Require regular updates to handle new patterns or edge cases
  • Complexity: May become unwieldy when trying to capture complex evaluation criteria

​
Current TrustTest Heuristic Evaluators

TrustTest provides several specialized heuristic evaluators:
  1. Regex: Regular expressions against the response
  2. Equals: Exact match to an expected value
  3. BLEU: n-gram similarity
  4. Language: ExpectedLanguageEvaluator (expected_languages) and EqualLanguageEvaluator via FastText (trusttest[language-detection])
  5. Signature evaluators: VirusEvaluator, SpamEvaluator, PhishingEvaluator, XssAttackVectorEvaluator
  6. Bias comparison: Paired Q&A consistency (BiasComparisonEvaluator)
While heuristic evaluators are fast and consistent, we recommend using LLM as a Judge evaluators when possible as they can better understand semantic relationships and reason about content in a more human-like way.