> ## Documentation Index > Fetch the complete documentation index at: https://neuraltrust-92b43583-develop.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > These docs cover three products: TrustGate (AI agent gateway), TrustGuard (runtime security), and TrustTest (AI red teaming). Start from each product overview for the definition and How it works. Prefer the .md URL next to a page in /llms.txt when you need the full article. Use /llms-full.txt for a single-file dump of the site. # High availability > The four availability tiers, from one cluster across zones to two active regions — what each one survives, and exactly what you change in the chart to get there.

Availability is a ladder, not a single design. Climb it only as far as the failure you actually have to survive — each rung costs more to operate than the one below it.

This page covers every self-hosted model: [Hybrid](/neuraltrust/deployment/hybrid), [External](/neuraltrust/deployment/external), and [Central control plane](/neuraltrust/deployment/central). The tiers are the same for all three. What changes per model is small and listed at the [end](#what-changes-per-deployment-model). ## The ladder

The four tiers plotted on cost and operational complexity against availability. Tier 1, zone resilient, sits lowest on both axes and is boxed and badged 99.95% cloud SLA. Tier 2, cluster resilient, costs more. Tier 3, region resilient, costs more again. Tier 4, always on, is the most expensive. The line through them flattens towards the top right, because each rung costs more and adds less. A footnote records that 99.95% is the monthly uptime GKE, EKS and AKS each commit to for a regional, zone-redundant cluster, that it covers the Kubernetes API server, and that your own figure also depends on your nodes and datastores.

| Tier | Survives | Costs you | | ---------------------------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------ | | **[1. Zone resilient](#tier-1-one-cluster-across-zones)**
one cluster, across zones | A node, or an entire availability zone | Node pools in three zones; nothing else to run | | **[2. Cluster resilient](#tier-2-twin-clusters-in-one-region)**
twin clusters, one region | Losing a cluster, and bad upgrades | A second cluster and a traffic switch | | **[3. Region resilient](#tier-3-two-regions-active/passive)**
two regions, active/passive | Losing an entire region | The above, plus DNS promotion and a datastore runbook | | **[4. Always on](#tier-4-two-regions-active/active)**
two regions, active/active | Losing a region, with no promotion of traffic | Each region sized for all the traffic, geo-aware routing, and per-region counters for good | **Tier 1 is enough for most deployments.** It is also the only rung the public clouds put a number on, because it is exactly what they sell: a regional, zone-redundant Kubernetes cluster. GKE, EKS and AKS each commit to **99.95%** monthly uptime for one. Read that as the ceiling your design works within rather than a promise about your traffic, because it covers the Kubernetes API server and not the workloads on it. Tiers 2 to 4 are topologies you build yourself, so no vendor underwrites them. Start at tier 1 and stop there unless a regional requirement says otherwise. The tiers are additive: each one assumes you have already done the ones below it, so tier 3 is tier 1 inside each region plus a promotion procedure, and tier 4 is tier 3 with both regions taking traffic. The chart now ships most of tier 1: the request path defaults to two replicas and spreads itself across zones. What is left to you is [the detector replicas, the disruption budgets and the datastores](#what-the-chart-does-and-what-you-still-set). ## Two invariants Everything below is easier to reason about once these are fixed. They hold at every tier, in every deployment model. A single writer is what prevents split-brain. Two clusters cannot diverge if there is only one place to write, which turns failover into a datastore operation rather than a distributed decision. Two DataAgents on the same scope register duplicate streams. This is why `dataagent.replicas` is `1` and must stay `1` — it is a correctness constraint, not a capacity setting. ## Tier 1: one cluster, across zones Spread the node pool over three availability zones and use managed PostgreSQL and Redis with automatic failover inside the region. The chart already runs the request path at two replicas and spreads it across those zones. There is no promotion procedure and no DNS work.

Tier 1: one regional Kubernetes cluster whose node pool spans three availability zones. Zone a, zone b and zone c each run agentgateway, trustguard and the firewall. Below the cluster, managed PostgreSQL is multi-AZ with primary and standby in different zones, and managed Redis holds cache and counters that rebuild in seconds. A note records that the chart ships two replicas and spreads them across zones, leaving you the firewall worker replicas, the disruption budgets and the managed datastores.

### What the chart does, and what you still set The chart now ships tier 1 most of the way: the request path defaults to two replicas and spreads itself across zones. Three things are still yours to set — the replica count on the detectors, the disruption budgets, and [the datastores](#managed-datastores-are-what-make-tier-1-real), which is the one that decides whether you are actually at tier 1. #### Replica counts The request path defaults to two replicas, so a fresh install already survives losing a node. These are the numbers worth checking before you call a cluster tier 1. | Workload | Chart key | Default | | ---------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------- | | Gateway proxy (LLM) | `agentgateway.dataPlane.replicas` | `2` | | Gateway MCP | `agentgateway.mcp.replicas` | `2` | | Gateway admin — External only | `agentgateway.controlPlane.replicas` | `2` | | TrustGuard data | `trustguard.dataPlane.replicas` | `2` | | TrustGuard control — External only | `trustguard.controlPlane.replicas` | `2` | | Firewall gateway | `firewall.firewall.gateway.replicas` | `2` | | Firewall workers | `firewall.firewall.workerDefaults.replicas` | `1` — **raise it**, these are the detectors | | Data-plane API | `data-plane-api.dataPlane.components.api.replicas` | `2` | | Control-plane API | `control-plane-api.controlPlane.components.api.replicaCount` | `2` | | Console | `control-plane-app.controlPlane.components.app.replicaCount` | `2` | | DataBridge | `databridge.replicas` | `2` | | DataAgent | `dataagent.replicas` | `1` — **leave at 1**, see [the invariants](#two-invariants) | The firewall workers hold the SLM detectors and are the expensive pods, which is why they still ship as singletons. On a GPU node pool, raising them is a cost decision as much as an availability one. `autoscaling.enabled` is `false` everywhere. Leave it that way until you have a capacity reason — autoscaling is a throughput decision, not an availability one, and a minimum replica count is what tier 1 actually depends on. #### Zone spread, and budgets for draining Replicas alone do not survive a zone loss: nothing stops Kubernetes putting them both on one node, and nothing stops a drain taking them at the same time. Two separate settings cover those. **Spreading is on by default.** Every workload renders `topologySpreadConstraints` across `topology.kubernetes.io/zone` and then `kubernetes.io/hostname`. They use `whenUnsatisfiable: ScheduleAnyway`, which makes them a preference rather than a requirement — so they cannot block scheduling on a single-zone or single-node cluster, and there is nothing to turn on for the common case. **Disruption budgets are opt-in, behind one switch.** A budget stops a node drain evicting every replica at once: ```yaml theme={null} global: highAvailability: podDisruptionBudget: enabled: true ``` That covers every component. Resolution is the same everywhere — an explicit per-component value wins, then `global.highAvailability`, then the built-in default — so one component can opt out again: ```yaml theme={null} agentgateway: dataPlane: podDisruptionBudget: enabled: false # this one component stays out # maxUnavailable: 1 # or shape the budget instead ``` A budget never renders on a single replica. One replica with a budget pins `disruptionsAllowed` at 0 and deadlocks every node drain on that node, so the chart suppresses it below two replicas — including for DataAgent, which must stay a singleton. The same three keys — `podDisruptionBudget`, `topologySpreadConstraints` and `affinity` — are accepted on every component, and `global.affinity` merges underneath a per-component `affinity` the way `global.nodeSelector` does. Then confirm the replicas really did spread: ```bash theme={null} # which node each gateway replica landed on kubectl -n neuraltrust get pods \ -l app.kubernetes.io/name=agentgateway,app.kubernetes.io/component=data-plane \ -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName --sort-by=.spec.nodeName # and which zone those nodes are in — two replicas in one zone is not tier 1 kubectl get nodes -o custom-columns=\ NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone ``` #### Managed datastores are what make tier 1 real The chart deploys PostgreSQL and Redis **in-cluster by default**, so a first install needs nothing provisioned. That default is for a proof of concept. Each store runs as a single pod, directly under the request path, so spreading the gateway over three zones buys you nothing while one Redis pod can still take the whole path down with it. **Tier 1 is not reached until both datastores are managed services** with automatic failover inside the region. Set `deploy: false` and point at instances outside the cluster: ```yaml theme={null} global: postgresql: deploy: false host: redis: deploy: false host: ``` See [managed stores](/neuraltrust/deployment/configuration#managed-stores) for the full values, and [datastore sizing floors](/neuraltrust/deployment/architecture#datastore-sizing-floors) for the minimum shapes. ## Tier 2: twin clusters in one region Two clusters side by side, both pointed at **one PostgreSQL primary and one Redis primary**. Because they are in the same region, sharing both stores adds no network hop and loses nothing on a switch: state is identical from either side.

Tier 2: a traffic switch you operate holds one hostname pointed at one cluster at a time. Cluster 1 is serving production traffic and cluster 2 is the standby where you roll the next version first. Both run agentgateway, trustguard and the firewall with replicas across three zones on the same chart version. Because they sit in one region they share one set of stores: one writable PostgreSQL primary and one Redis primary.

What it buys you is an escape from a broken cluster or a bad upgrade: roll the second cluster, move traffic, and keep the first as your way back. Keep both clusters on the same chart version and the same values. A twin cluster that has drifted is not a rollback target — it is a second thing to debug during an incident. ## Tier 3: two regions, active/passive Both clusters are full installs. The passive one is **warm**: its workloads run, pass health checks, and keep synchronizing, but it receives no production traffic.

Tier 3: two regions in active and passive. A traffic manager you operate holds one hostname per endpoint resolving to the active region, re-pointed on promotion. Region A is active, serves every request and runs the only DataAgent. Region B is passive and warm, Ready but taking no production traffic, with its DataAgent stopped. Each region has its own region-local Redis. Below both, one writable PostgreSQL primary streams to a cross-region read replica. Rate limits are counted per region because Redis is region-local by design.

Both read and write **one** PostgreSQL primary, which streams to a cross-region read replica promoted only if the primary's region fell over. ### Why the two datastores are treated differently **PostgreSQL — one writable primary, one cross-region read replica.** It holds the durable state, so a single writer is what prevents split-brain. Promotion is a datastore operation rather than a distributed decision, and your RPO is the replication lag. **Redis — region-local, one per cluster.** Redis carries semantic cache, rate-limit counters and evaluation progress, and has [no persistence requirement](/neuraltrust/deployment/architecture#datastore-sizing-floors). Reaching across a region for that on every request costs latency on the hot path and buys nothing, because the data is worth seconds. After a promotion the new active cluster starts with cold counters and rebuilds them in seconds. The consequence is that **rate limits are counted per region**. With one active region at a time that is invisible, except in the switchover window, when a client could briefly get a fresh allowance. With two regions active at once the same behaviour becomes permanent — that is the trade [tier 4](#tier-4-two-regions-active/active) asks you to accept. ### What each failure looks like | Failure | Effect | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | A node or a zone | Remaining replicas absorb the traffic; no operator action | | The active cluster | Your traffic manager and promotion automation switch the endpoints to the passive cluster | | The PostgreSQL primary | Managed failover inside the region; losing the region means promoting the read replica and repointing both clusters | | A region's Redis | That cluster loses cache and counters and rebuilds them; the other region is untouched | | The control plane, or your egress to it | Data planes keep enforcing their last synchronized configuration. Changes apply once sync resumes. Hybrid and Central only — External has no such dependency | ### Restarts during a control-plane outage Running pods survive a control-plane outage from their in-memory configuration, but a pod that **restarts** during one needs the last-known-good file on disk. The chart mounts it on an `emptyDir`, which does not survive a restart, so a restarted pod with no control-plane connectivity will not become Ready. If a cluster has to tolerate restarts mid-outage, move that mount to persistent storage. The `CONFIG_SYNC_LKG_KEY` that encrypts it is generated per cluster and needs no distribution. See [Secrets](/neuraltrust/deployment/secrets) for the key, and [Config sync](/neuraltrust/deployment/config-sync) for the mechanism. Do not make configuration changes directly in one data plane during a control-plane outage. The control plane remains the source of truth. ### Promote the passive cluster Use regional health checks backed by Kubernetes readiness on every published endpoint. Remove the failed cluster from all of them and stop its DataAgent, so it cannot come back mid-recovery and register a second stream. If the primary is in the failed region, promote the read replica and update `global.postgresql.host` in the surviving cluster. If the primary is unaffected, there is nothing to do here — this is the case the single-primary design is buying you. Redis needs no attention either way, because the surviving cluster already has its own. Confirm the request path is Ready with a current or last-known-good configuration, then enable DataAgent. Only one DataAgent may be active. Route the published hostnames to the promoted cluster. Keep all protocols on the same cluster, and remember that DNS-based promotion is delayed by resolver and client caching. Run representative allowed and blocked requests, then confirm policy decisions and telemetry before declaring the failover complete. Requests already in flight in the failed cluster fail. Clients should use bounded retries appropriate for their LLM or MCP operation. ### Before you rely on it * [ ] Both clusters are deployed, health checked, and on the same chart version. * [ ] Every request-path workload has redundant replicas across zones in each cluster, with PodDisruptionBudgets enabled. * [ ] PostgreSQL is managed, with one primary and a cross-region read replica. * [ ] Each region has its own managed Redis, and you accept per-region counters. * [ ] Last-known-good configuration is on persistent storage. * [ ] Only the active cluster runs DataAgent. * [ ] Every published hostname resolves to exactly one cluster. * [ ] You have rehearsed: block egress to the control plane and confirm traffic still flows; restart a pod during that outage; promote the passive cluster and switch every endpoint. ## Tier 4: two regions, active/active Both regions take production traffic at the same time. Nothing is promoted when a region fails: your traffic manager stops sending clients there, and the survivor carries everything. That is the whole gain — and the reason it is the most expensive rung to run honestly.

Tier 4: two regions both active. A traffic manager routes clients to whichever region is nearer, and every hostname resolves to both. Region A hosts the PostgreSQL primary so its writes stay in-region, and runs the only DataAgent. Region B also serves traffic, but every PostgreSQL write it makes crosses the region boundary, and its DataAgent stays stopped. There is still one writable primary streaming to a cross-region read replica. Each region must carry all of the traffic alone, and per-region rate limits become permanent.

**Talk to us before you build this.** Tier 4 is not a values change. It is a design conversation about your traffic manager, your write latency budget and how your clients are distributed. The platform supports it inside the same two invariants as every other tier, and those invariants are exactly what make it cost more than it looks. ### What stays the same **Still one writable PostgreSQL primary.** Both regions write to it. The region that does not host it pays a cross-region round trip on every write — raw payloads in [Hybrid](/neuraltrust/deployment/hybrid), and additionally configuration and console state in [External](/neuraltrust/deployment/external). Measure that latency against your budget before committing, and put the primary in the region that writes most. Losing the primary's region is the same promotion as at tier 3; the difference is that the surviving region is already serving, so only the datastore step and DataAgent remain. **Still exactly one DataAgent.** It reads the single PostgreSQL primary, so one agent already serves both regions' data. Run it in the primary's region, and start the other one only as part of a promotion. **Still region-local Redis, and still independent configuration.** Each region pulls its own configuration and keeps its own last-known-good copy, exactly as at tier 3. ### What it costs you for good | Consequence | What to do about it | | ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Rate limits are counted per region, permanently.** A client whose requests land in both regions can get up to twice its allowance. | Route by geography or client identity so that each client always lands in one region. Do not use round-robin. There is no globally exact counter across regions. | | **The semantic cache is per region.** Hit rates fall until each region has warmed its own. | Accept it. Pinning clients to a region also keeps their cache hits in one place. | | **Each region must carry all of the traffic on its own.** The day one region fails, the other takes everything. | Size and load-test each region for the whole peak, not half of it. Two half-sized regions are a tier 3 without the runbook. | | **Health checks decide who serves.** A region that is Ready but slow keeps taking traffic. | Back the traffic manager's checks with Kubernetes readiness on every published endpoint, and check the request path rather than a static page. | ### Before you rely on tier 4 Everything on the [tier 3 checklist](#before-you-rely-on-it), plus: * [ ] Every client is routed to exactly one region under normal conditions, and you know how. * [ ] Each region has been load-tested carrying the whole peak alone. * [ ] You have measured cross-region write latency from the far region and accepted it. * [ ] You have rehearsed withdrawing a region from the traffic manager and watched the other absorb the traffic. ## What changes per deployment model The ladder is identical in all three. These are the only differences, and each model's page covers its own in detail. | Model | What is specific to it | | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **[Hybrid](/neuraltrust/deployment/hybrid#high-availability)** | Both clusters pull the same gateway scope from NeuralTrust SaaS independently and keep their own last-known-good copy. DataAgent runs in the active cluster only. | | **[External](/neuraltrust/deployment/external#high-availability)** | Configuration lives in PostgreSQL, so a second cluster sees the same policies the moment it starts. ClickHouse is in-cluster and **does not follow a promotion** — at tier 4 each console permanently sees only its own region's telemetry unless you replicate it. Console sessions live in the region that issued them, so at tier 4 pin the console hostname to one region. Upgrade one cluster at a time — both consoles run migrations against one database. | | **[Central control plane](/neuraltrust/deployment/central#high-availability)** | Two questions, not one: the remote data planes climb this ladder, and the central cluster usually sits on tier 1 because it is not on the request path. If it does go to tier 3, four published hostnames have to follow the promotion, certificates must be valid from both central clusters, and the platform Secret's signing keys must be identical in both. | ## Recovery time Recovery time depends on health-check intervals, traffic-manager convergence, client DNS behavior, and datastore promotion — all of them yours, not the chart's. **Measure yours rather than assuming it**, and re-measure after any change to the traffic manager or the datastore topology. ## Next steps Node pools, datastore sizing floors, and the ports between components. Managed stores, ingress, TLS, and every values switch. The checks that confirm a cluster is actually serving what you think. What to look at when a replica will not become Ready.