Configuration

Before deployment, define the etcd cluster in the configuration inventory. Typical choices are:

• One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
• Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
• Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

An etcd cluster can have an even number of members. It has the same failure tolerance as a cluster with one fewer member. The extra member increases deployment and quorum costs. Thus, production clusters usually have one, three, or five members. Clusters with more than five members are uncommon.

One Node

To define a single etcd instance in Pigsty, use one line of configuration:

etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

All single-node configuration templates include this line. The configuration process replaces placeholder IP 10.10.10.10 with the current admin node IP.

Only etcd_seq and etcd_cluster are required. Together, these parameters uniquely identify each etcd instance.

Three Nodes

The most common configuration has 3 etcd nodes. It tolerates 1 node failure and is suitable for small and medium production environments.

Example: Pigsty’s 3-node templates trio and safe use 3-node etcd:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq (instance number) required
    10.10.10.11: { etcd_seq: 2 }  # positive integers, sequential from 0 or 1
    10.10.10.12: { etcd_seq: 3 }  # immutable for life, never recycled
  vars: # cluster-level params
    etcd_cluster: etcd    # default cluster name: 'etcd', don't change unless deploying multiple etcd clusters
    etcd_safeguard: false # enable safeguard? Enable after prod init to prevent accidental deletion
    etcd_clean: true      # force remove existing during init? Enable for testing for true idempotency

Five Nodes

A 5-node cluster tolerates 2 node failures. It is suitable for large production environments.

Example: Pigsty’s production simulation template ha/simu uses a 5-node etcd cluster:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

Services Using etcd

Services using etcd in Pigsty:

After a permanent etcd membership change, reload the related service configurations. This updates the endpoints that services use to access etcd.

Update Patroni’s etcd endpoint ref:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager’s etcd endpoint ref (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

RBAC Authentication Config

Since v4.0, Pigsty enables etcd RBAC authentication by default. Related parameters:

Production recommendations:

all:
  vars:
    etcd_root_password: 'YourSecureEtcdPassword'  # change default

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true    # enable safeguard for production

Filesystem Layout

The module creates these directories and files on target hosts: