Architecture

• 1: Nodes
• 2: Infrastructure
• 3: PGSQL Arch

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars: { pg_cluster: pg-test }

$ bin/pgsql-add pg-test  # Initialize cluster 'pg-test'

1 - Nodes

A node represents hardware resources and an operating system. It can be a physical server, virtual machine, container, or pod.

A node needs a Linux OS, systemd, and standard CPU, memory, disk, and network resources.

Install modules on nodes. The installed modules determine the node type:

In a single-node deployment, one node has all roles: regular, administration, infrastructure, ETCD, and database.

Regular Node

Use node.yml to configure managed nodes to the target state. A regular node can run these services:

node_exporter, vector, and haproxy are enabled by default. They export host metrics, collect logs, and balance traffic. Docker, keepalived, and keepalived_exporter are optional.

ADMIN Node

A deployment has exactly one administration node that runs Ansible playbooks and issues deployment or management commands.

This node has ssh/sudo access to every other node. Restrict access carefully. See Security Model: Trust Boundaries for its trusted access and critical assets.

During single-node installation and configuration, the current node becomes the administration node. A laptop with Ansible and SSH access to all managed nodes can also serve this role. This is not recommended for production.

For example, a laptop managing a cloud Pigsty VM is the administration node.

Production installations usually reserve 1–2 DBA machines for administration. With limited resources, INFRA nodes can also perform administration. All infrastructure nodes have Ansible by default.

INFRA Node

A Pigsty deployment may have 1 or more INFRA nodes; large production environments typically have 2-3.

The infra group in the inventory defines which nodes are INFRA nodes. These nodes run the INFRA module with these components:

Nginx provides the module Web UI and software repository. Services on multiple infrastructure nodes are independent. Each Grafana can access all monitoring data sources.

Pigsty is licensed under Apache-2.0, though embedded Grafana component uses AGPLv3.

ETCD Node

The ETCD module provides Distributed Consensus Service (DCS) for PostgreSQL high availability.

The etcd group in the inventory defines ETCD nodes. These nodes run etcd servers on two ports:

MINIO Node

The MINIO module provides optional backup storage for PostgreSQL.

The minio inventory group defines MINIO module nodes. In v4.5.0, these nodes run Silo servers on:

PGSQL Node

A PGSQL node has the PGSQL module. Each node runs one PostgreSQL instance, giving a 1:1 mapping.

node_id_from_pg lets a PGSQL node use its PostgreSQL instance name. The default is true.

PGSQL nodes run these additional components beyond regular node services:

vip-manager runs only when a PG VIP is configured. Define custom services through pg_services. haproxy exposes them on additional ports.

Node Relationships

Regular nodes normally select an INFRA node with admin_ip. For example, global admin_ip = 10.10.10.10 directs nodes to services at that address.

Parameters that reference ${admin_ip}:

The administration node usually also provides infrastructure. With multiple infrastructure nodes, the first usually administers the environment and others provide backups.

Large deployments can separate administration from infrastructure. For example, use 1–2 small DBA-managed administration hosts and 2–3 high-capacity physical monitoring servers.

Typical node counts by deployment scale:

2 - Infrastructure

Production HA PostgreSQL needs monitoring, alerts, logs, time synchronization, DNS, and software repositories. The INFRA module provides these services. It is optional but strongly recommended.

Overview

The following single-node deployment diagram shows INFRA components on the right:

Nginx

Nginx provides access to Pigsty WebUI services. HTTP and HTTPS use 80 and 443 by default. See the Live Demo.

Nginx exposes the Grafana, VictoriaMetrics (VMUI), AlertManager, and HAProxy web interfaces. It also serves the local software repository and other static files.

Nginx configures local web servers or reverse proxy servers based on definitions in infra_portal.

infra_portal:
  home : { domain: i.pigsty }

The default administration homepage is i.pigsty. Its endpoints proxy these components:

Configure Nginx as a file server or reverse proxy. HTTPS supports self-signed and publicly trusted certificates.

For more information, see: Tutorial: Nginx—Expose Web Services via Proxy and Tutorial: Certbot—Request and Renew HTTPS Certificates

Repo

During installation, Pigsty creates a local software repository on the infrastructure node to speed later installations. See the Live Demo.

The default directory is /www/pigsty. Nginx serves it at /pigsty:

For offline installation, copy a prepared local software repository into the target environment. If /www/pigsty/repo_complete exists, deployment skips upstream downloads and uses that repository. sow generates this file as a completion marker and SHA-256 manifest. To force a rebuild, run ./infra.yml -t repo_build -e repo_build=true.

For more information, see: Config: INFRA - REPO

Grafana

Grafana visualizes Pigsty metrics, logs, and other monitoring data. See the Live Demo.

Grafana listens on port 3000 by default and is proxied via Nginx at the /ui path:

Pigsty includes dashboards for VictoriaMetrics, Logs, and Traces. URL links let you move between summary and detail views for diagnosis.

Grafana also supports low-code visualization. Pigsty installs ECharts, victoriametrics-datasource, and victorialogs-datasource plugins by default. Vector/Victoria data sources use the names vmetrics-*, vlogs-*, and vtraces-* for custom dashboards.

For more information, see: Config: INFRA - GRAFANA.

VictoriaMetrics

VictoriaMetrics collects and stores Pigsty monitoring metrics as time-series data. See the Live Demo.

The default port is 8428. Access it through Nginx at /vmetrics or the p.pigsty domain:

VictoriaMetrics supports the Prometheus API, PromQL, remote read/write protocols, and the Alertmanager API. Its VMUI supports direct queries for metric exploration. It also supplies data to Grafana.

For more information, see: Config: INFRA - VMETRICS

VictoriaLogs

VictoriaLogs centrally stores structured logs from all Pigsty nodes. See the Live Demo.

It listens on port 9428 by default, mounted at Nginx /vlogs path:

Managed nodes run Vector by default. It collects system, PostgreSQL, Patroni, Pgbouncer, and other logs, structures them, and sends them to VictoriaLogs. The built-in Web UI supports search and filtering. Use the Grafana victorialogs-datasource plugin for visual analysis.

For more information, see: Config: INFRA - VLOGS

VictoriaTraces

VictoriaTraces is used for collecting trace data and slow SQL records. Live Demo

It listens on port 10428 by default, mounted at Nginx /vtraces path:

VictoriaTraces provides a Jaeger-compatible interface for service traces and slow SQL queries. Use it with Grafana to locate performance limits and trace their causes.

For more information, see: Config: INFRA - VTRACES

VMAlert

VMAlert evaluates alert rules and sends triggered events to Alertmanager. See the Live Demo.

It listens on port 8880 by default, mounted at Nginx /vmalert path:

VMAlert periodically evaluates rules against VictoriaMetrics data. Pigsty includes rules for common failures in PGSQL, NODE, REDIS, and other modules.

For more information, see: Config: INFRA - VMALERT

AlertManager

AlertManager handles alert event aggregation, deduplication, grouping, and dispatch. Live Demo

It listens on port 9059 by default, mounted at Nginx /alertmgr path, and also accessible via the a.pigsty domain:

AlertManager supports email, Webhook, Slack, PagerDuty, WeChat Work, and other notification channels. Routing rules select destinations by severity and module. Silencing and inhibition are also supported.

For more information, see: Config: INFRA - AlertManager

BlackboxExporter

Blackbox Exporter probes targets to check reachability.

It listens on port 9115 by default, mounted at Nginx /blackbox path:

Probe methods include ICMP Ping, TCP ports, and HTTP/HTTPS endpoints. Check VIP reachability, service ports, and external dependencies to determine the scope of a failure.

For more information, see: Config: INFRA - BLACKBOX

Ansible

Ansible runs Pigsty deployment, configuration, and management playbooks.

Installation adds Ansible to the administration/infrastructure node. Declarative configuration specifies the target state. Idempotent playbooks can run repeatedly to reach that state without additional side effects.

Ansible provides:

• Agentless execution: Connect through SSH without installing an agent on managed nodes.
• Declarative configuration: Describe the target state; configuration also documents it.
• Idempotent operations: Repeated runs produce consistent results and can retry partial failures.

For more information, see: Playbooks: Pigsty Playbook

DNSMASQ

DNSMASQ provides DNS resolution on INFRA nodes, resolving domain names to their corresponding IP addresses.

DNSMASQ listens on port 53 (UDP/TCP) by default, providing DNS resolution for all nodes. Records are stored in the /etc/dnsmasq.d/pigsty directory.

Other modules register domain names with DNSMASQ during deployment. Client nodes can use infrastructure nodes as DNS servers and access services by name. DNS is optional; Pigsty also works without it.

• dns_records: Default DNS records written to INFRA nodes
• node_dns_servers: Configure DNS servers for nodes, defaults to INFRA node via admin_ip (can also be disabled)

For more information, see: Config: INFRA - DNS and Tutorial: DNS—Configure Domain Resolution

Chronyd

Chronyd synchronizes node clocks through NTP. Its time-source service uses UDP port 123 by default.

Distributed systems need synchronized clocks. Logs need comparable timestamps, certificate checks need accurate time, and PostgreSQL streaming replication is sensitive to clock drift. In isolated networks, other nodes can synchronize to an infrastructure node acting as their NTP server.

All nodes run chronyd by default, using public pool.ntp.org servers. The Node module manages chronyd. In isolated networks, use admin_ip to select the INFRA node as the internal time source.

For more information, see: Config: NODE - TIME

INFRA Node vs Regular Node

In Pigsty, the relationship between nodes and infrastructure is a weak circular dependency: node_monitor → infra → node

NODE does not itself require INFRA. Its node_monitor functions require the infrastructure monitoring platform and services.

infra.yml and deploy resolve this dependency through these steps:

• Initialize NODE on regular nodes. Skip monitoring because the infrastructure is not ready.
• Initialize INFRA on the INFRA node to provide monitoring.
• Configure monitoring on regular nodes to connect to that platform.

Alternatively, use phased deployment: initialize infrastructure nodes before regular nodes.

How Are Nodes Coupled to Infrastructure?

Regular nodes use admin_ip to select an INFRA node for infrastructure services.

For example, global admin_ip = 10.10.10.10 normally directs all nodes to infrastructure at that address.

Change this address to switch infrastructure providers for many nodes. These parameters may reference ${admin_ip}:

For example, local uses the Nginx software repository at admin_ip:80/pigsty. DNS uses DNSMASQ at admin_ip:53. Both choices are optional. Nodes can install from Internet repositories, as most single-node templates do. They can also leave DNS unconfigured because Pigsty does not require it.

INFRA Node vs ADMIN Node

The ADMIN node usually also serves as the INFRA node, including in single-node deployments. With multiple infrastructure nodes, the first infra member usually administers the environment and others provide backups. You can also separate the roles.

For example, large production deployments can use 1–2 small DBA-managed VMs for administration. Use 2–3 high-capacity physical servers, or more, for monitoring infrastructure. Set admin_ip to an infrastructure node address. The parameter retains its historical name from when administration and infrastructure roles were combined.

For local management of cloud nodes, install Ansible on your laptop and select cloud servers as managed targets. The laptop is the administration node. Cloud servers provide infrastructure.

all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 , ansible_host: your_ssh_alias } } }  # <--- Use ansible_host to point to cloud node (fill in ssh alias)
    etcd:    { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }    # SSH connection will use: ssh your_ssh_alias
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } }
  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default

Multiple INFRA Nodes

One infrastructure node usually meets the requirements. Failure of its INFRA module does not interrupt database services on other nodes.

For higher monitoring and alerting availability, use multiple infrastructure nodes. Two nodes can provide redundancy and monitor each other. A distributed Victoria cluster can use more nodes for horizontal scaling without a fixed limit.

Each infrastructure node operates independently. Nginx uses local services, and VictoriaMetrics collects metrics from every environment service. Nodes send logs to all VictoriaLogs endpoints by default. Grafana is the exception: each instance registers every VictoriaMetrics, Logs, Traces, and PostgreSQL instance as a data source. Thus, each Grafana can access all monitoring data.

Dashboard and data-source changes affect only the Grafana instance you edit. For shared configuration, use a PostgreSQL database as Grafana storage. See Configure Grafana High Availability.

3 - PGSQL Arch

PGSQL organizes PostgreSQL as clusters. Each cluster is a logical group of database instances connected through primary-replica relationships.

Overview

The PGSQL module combines these components for production HA clusters:

Enable vip-manager when needed. PGSQL also uses components from other modules:

The PGSQL module combines the PostgreSQL engine with supporting services. Patroni and Etcd provide HA. pgBackRest and optional Silo provide backups. HAProxy, Pgbouncer, and vip-manager provide access. Exporters and Vector provide monitoring.

You can select other database kernels and extensions. The diagram represents the engine as a CPU and the complete service as a computer.

Component Interaction

• Cluster DNS is resolved by DNSMASQ on infra nodes
• Cluster VIP is managed by vip-manager, which binds pg_vip_address to the cluster primary node.
  • vip-manager gets cluster leader info written by patroni from the etcd cluster
• Cluster services are exposed by HAProxy on nodes, different services distinguished by node ports (543x).
  • HAProxy port 9101: Monitoring metrics & statistics & admin page
  • HAProxy port 5433: Routes to primary pgbouncer: read-write service
  • HAProxy port 5434: Routes to replica pgbouncer: read-only service
  • HAProxy port 5436: Routes to primary postgres: default service
  • HAProxy port 5438: Routes to offline postgres: offline service
  • HAProxy routes traffic based on health check info from patroni.
• Pgbouncer is connection pooling middleware, listening on port 6432 by default, buffering connections, exposing additional metrics, and providing extra flexibility.
  • Pgbouncer is stateless and deployed 1:1 with Postgres via local Unix socket.
  • Production traffic (primary/replica) goes through pgbouncer by default (can specify bypass via pg_default_service_dest)
  • Default/offline services always bypass pgbouncer and connect directly to target Postgres.
• PostgreSQL listens on port 5432, providing relational database services
  • Installing PGSQL module on multiple nodes with the same cluster name automatically forms an HA cluster via streaming replication
  • PostgreSQL process is managed by patroni by default.
• Patroni listens on port 8008 by default, supervising PostgreSQL server processes
  • Patroni starts Postgres server as child process
  • Patroni uses etcd as DCS: stores config, failure detection, and leader election.
  • Patroni provides Postgres info (e.g., primary/replica) via health checks, HAProxy uses this to distribute traffic
• pg_exporter exposes postgres monitoring metrics on port 9630
• pgbouncer_exporter exposes pgbouncer metrics on port 9631
• pgBackRest uses local backup repository by default (pgbackrest_method = local)
  • If using local (default), pgBackRest creates local repository under pg_fs_bkup on primary node
  • If using minio, pgBackRest creates the backup repository on dedicated Silo or an external S3 service
• Vector collects Postgres-related logs (postgres, pgbouncer, patroni, pgbackrest)
  • vector listens on port 9598, also exposes its own metrics to VictoriaMetrics on infra nodes
  • vector sends logs to VictoriaLogs on infra nodes

HA Subsystem

Patroni and etcd provide the HA subsystem. They detect failures, coordinate failover, and manage configuration.

Operation: Patroni manages each local PostgreSQL process. It writes the leader, members, and configuration to etcd. After primary failure, Patroni coordinates an election and promotes the healthiest replica. Recovery is automatic, with RTO typically below 45 seconds.

Key Interactions:

• PostgreSQL: Starts, stops, reloads PG as parent process, controls its lifecycle
• etcd: External dependency, writes/watches leader key for distributed consensus and failure detection
• HAProxy: Provides health checks via REST API (:8008), reporting instance role
• vip-manager: Watches leader key in etcd, auto-migrates VIP

For more information, see: High Availability and Config: PGSQL - PG_BOOTSTRAP

Access Subsystem

HAProxy, Pgbouncer, and vip-manager expose services, route traffic, and pool connections.

There are multiple access methods. A typical traffic path is: Client → DNS/VIP → HAProxy (543x) → Pgbouncer (6432) → PostgreSQL (5432)

Service Ports:

• 5433 primary: Read-write service, routes to primary Pgbouncer
• 5434 replica: Read-only service, routes to replica Pgbouncer
• 5436 default: Default service, direct to primary (bypasses pool)
• 5438 offline: Offline service, direct to offline replica (ETL/analytics)

Key Features:

• HAProxy uses Patroni REST API to determine instance role, auto-routes traffic
• Pgbouncer uses transaction-level pooling, absorbs connection spikes, reduces PG connection overhead
• vip-manager watches etcd leader key, auto-migrates VIP during failover

For more information, see: Service Access and Config: PGSQL - PG_ACCESS

Backup Subsystem

pgBackRest provides backups and PITR. Optional Silo or external S3 storage can serve as the remote repository.

Backup Types:

• Full backup: Complete database copy
• Incremental/differential backup: Only backs up changed data blocks
• WAL archiving: Continuous transaction log archiving, enables any point-in-time recovery

Storage Backends:

• local (default): Local disk, backups stored at pg_fs_bkup mount point
• minio: S3-compatible object storage, supports centralized backup management and off-site DR

Key Interactions:

• pgBackRest → PostgreSQL: Executes backup commands, manages WAL archiving
• pgBackRest → Patroni: Recovery can bootstrap replicas as new primary or standby
• pgbackrest_exporter → VictoriaMetrics: Exports backup status metrics through the Prometheus-compatible protocol to monitor backup health

For more information, see: PITR, Backup & Recovery, and Config: PGSQL - PG_BACKUP

Observability Subsystem

Three Exporters collect metrics, and Vector collects logs.

Data Flow:

• Metrics: Exporter → VictoriaMetrics (INFRA) → Grafana dashboards
• Logs: Vector → VictoriaLogs (INFRA) → Grafana log queries

pg_exporter and pgbouncer_exporter use local Unix sockets, independently of HA topology. Slim installation can disable them.

For more information, see: Config: PGSQL - PG_MONITOR

PostgreSQL

PostgreSQL is the PGSQL database engine. It listens on 5432 by default, with one instance per node.

Pigsty supports PostgreSQL 14–18 through binary packages from the PGDG repository. You can replace the default engine with a PG-compatible kernel. Up to 600 extensions are available.

Patroni manages PostgreSQL processes for HA by default. A single-node cluster has one primary. Additional nodes automatically join as replicas and receive changes through real-time physical replication. Replicas serve read-only traffic and can automatically replace a failed primary.

You can access PostgreSQL directly, or through HAProxy and Pgbouncer connection pool.

For more information, see: Config: PGSQL - PG_BOOTSTRAP

Patroni

Patroni is the PostgreSQL HA control component, listening on port 8008 by default.

Patroni manages PostgreSQL startup, shutdown, configuration, and health. It writes leader and member records to etcd. It coordinates failover, replication factor, and parameter changes. Its REST API serves HAProxy, monitoring, and administrators.

HAProxy checks Patroni endpoints to identify instance roles and route primary or replica traffic. vip-manager watches the etcd leader key and moves the VIP when the primary changes.

For more information, see: Config: PGSQL - PG_BOOTSTRAP

Pgbouncer

Pgbouncer provides connection pooling on port 6432 by default. Each node has one Pgbouncer instance for its PostgreSQL instance.

Pgbouncer is stateless and connects to local PostgreSQL through a Unix socket. Its default transaction pooling handles connection bursts, stabilizes session counts, and reduces lock contention. This improves performance under high concurrency.

By default, read/write 5433 and read-only 5434 traffic use Pgbouncer. Default 5436 and offline 5438 services connect directly to PostgreSQL.

pgbouncer_poolmode defaults to transaction. Disable pooling with pgbouncer_enabled.

For more information, see: Config: PGSQL - PG_ACCESS

pgBackRest

pgBackRest provides full, incremental, and differential PostgreSQL backups and WAL archiving.

Pigsty uses pgBackRest for PITR to any time within the backup retention window.

pgBackRest coordinates with PostgreSQL for backups and archiving. By default, it creates a repository on the primary: pgbackrest_method = local. Configure Silo or external S3 storage for centralized backup management.

pgbackrest_init_backup can trigger the first full backup after initialization. Recovery integrates with Patroni to initialize replicas as new primaries or standbys.

For more information, see: Backup & Recovery and Config: PGSQL - PG_BACKUP

HAProxy

HAProxy is the service entry point and load balancer, exposing multiple database service ports.

HAProxy checks Patroni REST endpoints to identify roles and route primary or replica traffic. pg_default_services and pg_services define services.

By default, local HAProxy instances expose services. For higher traffic, use pg_service_provider to select a dedicated HAProxy node group.

For more information, see: Service Access and Config: PGSQL - PG_ACCESS

vip-manager

vip-manager binds L2 VIP to the current primary node. This is an optional component; enable it if your network supports L2 VIP.

vip-manager runs on each PG node. It watches the leader key that Patroni writes to etcd. It binds pg_vip_address to the primary network interface. During failover, it releases the old binding and immediately binds the VIP to the new primary.

Enable this optional component with pg_vip_enabled. All nodes must share one VLAN or VIP migration fails. Public clouds usually do not support L2 VIP. Use it for on-premises or private cloud deployments.

For more information, see: Tutorial: VIP Configuration and Config: PGSQL - PG_ACCESS

pg_exporter

pg_exporter exports PostgreSQL monitoring metrics, listening on port 9630 by default.

pg_exporter connects to local PostgreSQL through a Unix socket. It exports sessions, buffer hits, replication lag, transaction rates, and other metrics. VictoriaMetrics on infrastructure nodes collects them.

Set collection rules with pg_exporter_config. Use pg_exporter_auto_discovery for automatic database discovery. pg_exporter_cache_ttls controls different cache lifetimes.

Disable this component through parameters. Slim installation leaves it disabled.

For more information, see: Config: PGSQL - PG_MONITOR

pgbouncer_exporter

pgbouncer_exporter exports Pgbouncer connection pool metrics, listening on port 9631 by default.

pgbouncer_exporter uses the pg_exporter binary with separate metric configuration. It supports pgbouncer 1.8–1.25+. Pgbouncer statistics views provide pool utilization, wait queue, and hit rate metrics.

Disabling Pgbouncer also disables its exporter. Slim installation leaves the exporter disabled.

For more information, see: Config: PGSQL - PG_MONITOR

pgbackrest_exporter

pgbackrest_exporter exports backup status metrics, listening on port 9854 by default.

pgbackrest_exporter reads pgBackRest status to report the latest backup time, size, type, and other metrics. Alert rules can detect expired or failed backups. Large repositories or many backups can make collection expensive. The default collection interval is 2 minutes, so a completed backup can take 2 minutes to appear in monitoring.

For more information, see: Config: PGSQL - PG_MONITOR

etcd

etcd is a distributed consistent store (DCS), providing cluster metadata storage and leader election capability for Patroni.

The separate ETCD module manages etcd. PGSQL relies on it for HA. Patroni stores cluster state, leader information, and configuration there so nodes can reach consensus. vip-manager reads the leader key to move the VIP automatically.

For more information, see: ETCD Module

vector

The NODE module deploys Vector to collect PostgreSQL-related logs efficiently.

Vector reads node log directories for PostgreSQL, Pgbouncer, Patroni, and pgBackRest. It sends structured records to infrastructure-node VictoriaLogs for central storage and queries.

For more information, see: NODE Module