NeuralTrust | Platform for Agent Security.

LLM Clients

TrustTest provides a flexible abstraction layer for working with different LLM providers through its LLMClient interface. This architecture allows for seamless integration with various LLM services while maintaining a consistent interface for generating questions, evaluations, and other LLM-powered features.

Architecture

The core of this system is the LLMClient abstract base class, which defines two main methods:

complete(instructions, system_prompt): For single-prompt completions
complete_chat(messages): For multi-turn conversations

Each implementation handles provider-specific details while exposing a unified interface.

Supported Providers

get_llm_client(provider=..., model=..., **kwargs) supports: openai, azure, google, anthropic, ollama, vllm, groq, deepseek, http.

Provider	Extra	Notes
`openai`	`trusttest[openai]`	Default models in `set_config`
`azure`	`trusttest[azure]` or `trusttest[openai]`	Azure OpenAI judge / generator
`google`	`trusttest[google]`
`anthropic`	`trusttest[anthropic]`
`ollama`	`trusttest[ollama]`	Local
`vllm`	`trusttest[vllm]`	OpenAI-compatible
`groq`	`trusttest[openai]`	Uses `OpenAiClient` against Groq
`deepseek`	`trusttest[deepseek]`
`http`	none	Judge/generator against any HTTP endpoint — not the same as `HttpTarget` (the system under test)

uv add "trusttest[openai]"
uv add "trusttest[anthropic]"
uv add "trusttest[google]"
uv add "trusttest[ollama]"
uv add "trusttest[vllm]"
uv add "trusttest[azure]"
uv add "trusttest[deepseek]"

HTTP judge / generator

HTTPClient is an LLM client. HttpTarget is the model you are testing. Do not mix them.

from trusttest.llm_clients import get_llm_client
from trusttest.evaluators import TrueFalseEvaluator

judge = get_llm_client(
    provider="http",
    model="ignored",
    url="https://your-judge.example.com/v1/complete",
    payload_config={
        "messages": [
            {"role": "system", "content": "{{ system_prompt }}"},
            {"role": "user", "content": "{{ instructions }}"},
        ]
    },
    concatenate_field="choices.0.message.content",
    headers={"Authorization": "Bearer ..."},
)

evaluator = TrueFalseEvaluator(llm_client=judge)

concatenate_field supports dot paths. Placeholders default to {{ instructions }} and {{ system_prompt }}. You can also pass llm_client= on individual probes and evaluators instead of changing global config.

Usage Example

import asyncio
import os
from trusttest.llm_clients import get_llm_client

# Set up environment variables for the provider
os.environ["DEEPSEEK_BASE_URL"] = "https://api.deepseek.com"
os.environ["DEEPSEEK_API_KEY"] = "<your-api-key>"

# Initialize the client
llm = get_llm_client(model="deepseek-chat", provider="deepseek")

# Make a completion request
response = asyncio.run(llm.complete("Return a json saying hello"))
print(response)

The abstraction allows for easy switching between providers while maintaining consistent behavior across the application.

Embeddings Clients

TrustTest provides a flexible abstraction layer for working with different embedding providers through its EmbeddingsModel interface. This architecture allows for seamless integration with various embedding services while maintaining a consistent interface for generating vector representations of text.

Architecture

The core of this system is the EmbeddingsModel abstract base class, which defines the main method:

embed(texts): Converts a sequence of texts into numerical vector representations

Each implementation handles provider-specific details while exposing a unified interface.

Supported Providers

get_embeddings_model(provider=..., model=...) supports openai, azure, google, and ollama.

from trusttest.embeddings import get_embeddings_model

embeddings = get_embeddings_model(
    provider="azure",
    model="text-embedding-3-small",
)

Use trusttest[azure] (or trusttest[rag-azure]) for Azure embeddings. Vector knowledge bases need embeddings plus a topic_summarizer LLM via set_config.

Usage Example

import os

from trusttest.embeddings import get_embeddings_model

os.environ["OPENAI_API_KEY"] = "<your-api-key>"

embeddings = get_embeddings_model(
    provider="openai",
    model="text-embedding-3-small",
)

texts = ["Hello world", "TrustTest is great"]
vectors = embeddings.embed(texts)
print(vectors.shape)

The abstraction allows for easy switching between providers while maintaining consistent behavior across the application.

Global Configuration

TrustTest provides a global configuration system to manage LLM and embeddings settings across your application. The configuration can be set using the set_config() function, which accepts a dictionary with settings for different components:

import trusttest
from trusttest.config import get_config, load_config, TrustTestConfigError

trusttest.set_config({
    "evaluator": {
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.2,
        "retry_config": {"attempts": 3, "initial_delay": 1.0, "max_delay": 10.0, "exp_base": 2.0},
        "extra_args": {},
    },
    "question_generator": {
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.5,
    },
    "translation": {
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.2,
    },
    "embeddings": {
        "provider": "openai",
        "model": "text-embedding-3-small",
    },
    "topic_summarizer": {
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.2,
    },
})

Tasks: evaluator, question_generator, translation (used by StaticDatasetProbe.translate_into_language), topic_summarizer, embeddings. LLM fields: provider, model, temperature, optional retry_config (attempts, initial_delay, max_delay, exp_base), optional extra_args. Config provider literals are openai, azure, deepseek, vllm, google, anthropic, ollama, http. Use get_llm_client(provider="groq", ...) for Groq. Embeddings fields: provider (openai, azure, google, ollama) and model. Defaults when you do not set config: gpt-4o-mini (all LLM tasks) and text-embedding-3-small.

Config files

On import, TrustTest looks in the current working directory for .trusttest_config.json, then trusttest_config.json. The JSON object uses the same keys as set_config. If neither file exists and you never called set_config, get_config() raises TrustTestConfigError.

from trusttest.config import get_config, load_config, TrustTestConfigError

load_config()  # re-read CWD files
try:
    cfg = get_config()
except TrustTestConfigError:
    trusttest.set_config({...})

Implementing Custom Clients

Both LLM and Embeddings clients can be easily extended by implementing custom providers. The base classes provide a clear interface that you need to implement.

Custom LLM Client

To create a custom LLM client, inherit from LLMClient and implement the required methods:

from trusttest.llm_clients.base import LLMClient, ChatMessage, BaseLLMResponse

class CustomLLMClient(LLMClient):
    async def complete(
        self,
        instructions: str,
        system_prompt: Optional[str] = None,
        response_schema: Type[BaseModel] = BaseLLMResponse,
    ) -> Dict[str, Any]:
        # implement your custom logic here
        raise NotImplementedError

    async def complete_chat(
        self,
        messages: Sequence[ChatMessage],
        response_schema: Type[BaseModel] = BaseLLMResponse,
    ) -> Dict[str, Any]:
        # implement your custom logic here
        raise NotImplementedError

The LLMClient expects to define the response schema, this is a pydantic model that will be used to parse the response from the LLM. Once implemented, you are ready to use them:

custom_llm = CustomLLMClient()
evaluator = CorrectnessEvaluator(llm_client=custom_llm)

Custom Embeddings Client

To create a custom embeddings client, inherit from EmbeddingsModel and implement the required method:

from trusttest.embeddings import EmbeddingsModel
import numpy as np

class CustomEmbeddingsModel(EmbeddingsModel):
    def embed(self, texts: list[str]) -> np.ndarray:
        # Implement your custom embedding logic
        pass

Getting Started

Core Concepts

Connect your app

Create tests

Evaluate results

LLMs & Embeddings

LLM Clients

Architecture

Supported Providers

HTTP judge / generator

Usage Example

Embeddings Clients

Architecture

Supported Providers

Usage Example

Global Configuration

Config files

Implementing Custom Clients

Custom LLM Client

Custom Embeddings Client

​LLM Clients

​Architecture

​Supported Providers

​HTTP judge / generator

​Usage Example

​Embeddings Clients

​Architecture

​Supported Providers

​Usage Example

​Global Configuration

​Config files

​Implementing Custom Clients

​Custom LLM Client

​Custom Embeddings Client

LLM Clients

Architecture

Supported Providers

HTTP judge / generator

Usage Example

Embeddings Clients

Architecture

Supported Providers

Usage Example

Global Configuration

Config files

Implementing Custom Clients

Custom LLM Client

Custom Embeddings Client