> ## Documentation Index
> Fetch the complete documentation index at: https://neuraltrust-92b43583-develop.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# From RAG

> Generate functional tests from your Retrieval-Augmented Generation knowledge base

Generate functional tests directly from your RAG knowledge base to validate that your model correctly retrieves and synthesizes information from your documents.

## Overview

Testing RAG applications requires validating that:

1. **Retrieval works correctly**: Relevant documents are found
2. **Synthesis is accurate**: Information is correctly combined
3. **Responses are grounded**: Answers are based on the knowledge base
4. **No hallucinations**: Model doesn't make up information

## How It Works

TrustTest automatically:

1. Connects to your knowledge base (vector store, database, etc.)
2. Retrieves document chunks
3. Generates question-answer pairs based on the content
4. Creates test cases with expected responses
5. Evaluates your model's actual responses against expectations

***

## Supported Knowledge Bases

| Connector                                                                     | Description                      |
| ----------------------------------------------------------------------------- | -------------------------------- |
| [In-Memory](/trusttest/create/knowledge-base/connectors/in-memory)            | Local vector store for testing   |
| [Azure AI Search](/trusttest/create/knowledge-base/connectors/azure)          | Azure's cognitive search         |
| [Neo4j](/trusttest/create/knowledge-base/connectors/neo4j)                    | Graph database                   |
| [PostgreSQL + pgvector](/trusttest/create/knowledge-base/connectors/postgres) | PostgreSQL with vector extension |
| [Upstash](/trusttest/create/knowledge-base/connectors/upstash)                | Serverless Redis vector store    |

***

## Code Example

### Using In-Memory Knowledge Base

```python theme={null}
from trusttest.knowledge_base import InMemoryKnowledgeBase, Document
from trusttest.probes.rag import RAGProbe
from trusttest.targets.http import HttpTarget, PayloadConfig
from trusttest.evaluators import CorrectnessEvaluator
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluation_scenarios import EvaluationScenario

documents = [
    Document(id="1", content="TrustTest is a framework for testing AI models for safety and reliability.", topic="product"),
    Document(id="2", content="TrustTest supports Azure, Neo4j, PostgreSQL, and Upstash knowledge bases.", topic="connectors"),
    Document(id="3", content="Probes in TrustTest generate test cases to evaluate model behavior.", topic="probes"),
]

kb = InMemoryKnowledgeBase(documents=documents)

# Configure target
target = HttpTarget(
    url="https://your-rag-endpoint.com/chat",
    headers={"Content-Type": "application/json"},
    payload_config=PayloadConfig(
        format={"messages": [{"role": "user", "content": "{{ test }}"}]},
        message_regex="{{ test }}",
    ),
)

# Create RAG probe
probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=20,
)

# Evaluate with correctness judge
evaluator = CorrectnessEvaluator()
suite = EvaluatorSuite(evaluators=[evaluator])
scenario = EvaluationScenario(evaluator_suite=suite)

test_set = probe.get_test_set()
results = scenario.evaluate(test_set)
results.display_summary()
```

### Using Azure AI Search

```python theme={null}
from azure.core.credentials import AzureKeyCredential
from trusttest.knowledge_base.azure_search import AzureKnowledgeBase

kb = AzureKnowledgeBase(
    credentials=AzureKeyCredential("your-api-key"),
    service_endpoint="https://your-search.search.windows.net",
    index_name="your-index",
)

probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=50,
)
```

### Using PostgreSQL with pgvector

```python theme={null}
from trusttest.knowledge_base.pgvector import PgVectorKnowledgeBase

kb = PgVectorKnowledgeBase(
    connection_string="postgresql://user:pass@localhost/db",
    table_name="documents",
    fields_mapping={
        "id": "id",
        "content": "content",
        "embeddings": "embedding",
    },
)

probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=50,
)
```

***

## Configuration Options

| Parameter        | Type            | Default                      | Description                          |
| ---------------- | --------------- | ---------------------------- | ------------------------------------ |
| `target`         | `Target`        | Required                     | The RAG model to test                |
| `knowledge_base` | `KnowledgeBase` | Required                     | Your knowledge base connector        |
| `num_questions`  | `int`           | `20`                         | Number of test questions to generate |
| `question_types` | `List[str]`     | `["factual", "inferential"]` | Types of questions to generate       |
| `language`       | `LanguageType`  | `"English"`                  | Language for generated questions     |

***

## Question Types

TrustTest generates different types of questions:

| Type            | Description                    | Example                                                |
| --------------- | ------------------------------ | ------------------------------------------------------ |
| **Factual**     | Direct fact retrieval          | "What connectors does TrustTest support?"              |
| **Inferential** | Requires combining information | "How would you test a RAG app with Azure?"             |
| **Comparative** | Comparing entities             | "What's the difference between probes and evaluators?" |

***

## Evaluating RAG Responses

For RAG applications, use these evaluators:

```python theme={null}
from trusttest.evaluators import (
    CorrectnessEvaluator,
    CompletenessEvaluator,
    RAGPoisoningEvaluator,
)

evaluators = [
    CorrectnessEvaluator(),      # Is the answer factually correct?
    CompletenessEvaluator(),     # Does it cover all relevant points?
    RAGPoisoningEvaluator(),     # Is the response grounded in context?
]
```

***

## Gold answers from retrieved docs

Use `RagContextBuilder` when you want an `ExpectedResponseContext` dataset grounded in retrieved chunks:

```python theme={null}
from trusttest.dataset_builder import RagContextBuilder, QuestionTopic

builder = RagContextBuilder(questions=[QuestionTopic(question="...", topic="...")], knowledge_base=kb)
dataset = builder.build()
```

See [Automatic test generation](/trusttest/create/automatic-test-generation#ragcontextbuilder).

Configure `embeddings` and `topic_summarizer` with `set_config` before clustering unlabeled documents. Clustering (UMAP / HDBSCAN) is not Azure-only.

## Related Topics

* [Knowledge Base Connectors](/trusttest/create/knowledge-base/overview)
* [Automatic Test Generation](/trusttest/create/automatic-test-generation)
* [Answer Relevance](/trusttest/evaluate-result/llm-as-a-judge/answer-relevance)
* [RAG Poisoning Evaluation](/trusttest/evaluate-result/llm-as-a-judge/rag-poisoning)