Module: ETCD

• 1: Configuration
• 2: Parameters
• 3: Administration
• 4: Playbook
• 5: Monitoring
• 6: Metrics
• 7: FAQ

flowchart LR
    subgraph PGSQL [PGSQL]
        patroni[Patroni]
        vip[VIP Manager]
    end

    subgraph ETCD [ETCD]
        etcd[DCS Service]
    end

    subgraph NODE [NODE]
        node[Software Repo]
    end

    PGSQL -->|depends| ETCD -->|depends| NODE

    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style ETCD fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style patroni fill:#2d4a66,stroke:#1e3347,color:#fff
    style vip fill:#2d4a66,stroke:#1e3347,color:#fff
    style etcd fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

1 - Configuration

Before deployment, define etcd cluster in config inventory. Typical choices:

• One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
• Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
• Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

Even-numbered clusters don’t make sense; 5+ node clusters uncommon. Typical configs: single, 3-node, 5-node.

One Node

Define singleton etcd instance in Pigsty—single line of config:

etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

All single-node config templates include this line. Placeholder IP 10.10.10.10 replaced with current admin node’s IP.

Only required params: etcd_seq and etcd_cluster—uniquely identify each etcd instance.

Three Nodes

Most common config: 3-node etcd cluster tolerates 1 node failure, suitable for small-medium prod.

Example: Pigsty’s 3-node templates trio and safe use 3-node etcd:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq (instance number) required
    10.10.10.11: { etcd_seq: 2 }  # positive integers, sequential from 0 or 1
    10.10.10.12: { etcd_seq: 3 }  # immutable for life, never recycled
  vars: # cluster-level params
    etcd_cluster: etcd    # default cluster name: 'etcd', don't change unless deploying multiple etcd clusters
    etcd_safeguard: false # enable safeguard? Enable after prod init to prevent accidental deletion
    etcd_clean: true      # force remove existing during init? Enable for testing for true idempotency

Five Nodes

5-node cluster tolerates 2 node failures, suitable for large prod.

Example: Pigsty’s prod sim template prod uses 5-node etcd:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

Services Using etcd

Services using etcd in Pigsty:

When etcd cluster membership changes permanently, reload related service configs to ensure correct access.

Update Patroni’s etcd endpoint ref:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager’s etcd endpoint ref (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

RBAC Authentication Config

v4.0 enables etcd RBAC auth by default. Related params:

Prod recommendations:

all:
  vars:
    etcd_root_password: 'YourSecureEtcdPassword'  # change default

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true    # enable safeguard for production

Filesystem Layout

Module creates these directories/files on target hosts:

2 - Parameters

The ETCD module has 13 parameters, divided into two sections:

• ETCD: 10 parameters for etcd cluster deployment and configuration
• ETCD_REMOVE: 3 parameters for controlling etcd cluster removal

Parameter Overview

The ETCD parameter group is used for etcd cluster deployment and configuration, including instance identification, cluster name, data directory, ports, and authentication password.

The ETCD_REMOVE parameter group controls etcd cluster removal behavior, including safeguard protection, data cleanup, and package uninstallation.

ETCD

This section contains parameters for the etcd role, which are used by the etcd.yml playbook.

Parameters are defined in roles/etcd/defaults/main.yml

#etcd_seq: 1                      # etcd instance identifier, explicitly required
etcd_cluster: etcd                # etcd cluster & group name, etcd by default
etcd_learner: false               # run etcd instance as learner? default is false
etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
etcd_port: 2379                   # etcd client port, 2379 by default
etcd_peer_port: 2380              # etcd peer port, 2380 by default
etcd_init: new                    # etcd initial cluster state, new or existing
etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
etcd_root_password: Etcd.Root     # etcd root user password for RBAC authentication (please change!)

etcd_seq

Parameter: etcd_seq, Type: int, Level: I

etcd instance identifier. This is a required parameter—you must assign a unique identifier to each etcd instance.

Here is an example of a 3-node etcd cluster with identifiers 1 through 3:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

etcd_cluster

Parameter: etcd_cluster, Type: string, Level: C

etcd cluster & group name, default value is the hard-coded etcd.

You can modify this parameter when you want to deploy an additional etcd cluster for backup purposes.

etcd_learner

Parameter: etcd_learner, Type: bool, Level: I/A

Initialize etcd instance as learner? Default value is false.

When set to true, the etcd instance will be initialized as a learner, meaning it cannot participate in voting elections within the etcd cluster.

Use Cases:

• Cluster Expansion: When adding new members to an existing cluster, using learner mode prevents affecting cluster quorum before data synchronization completes
• Safe Migration: In rolling upgrade or migration scenarios, join as a learner first, then promote after confirming data synchronization

Workflow:

1. Set etcd_learner: true to initialize the new member as a learner
2. Wait for data synchronization to complete (check with etcdctl endpoint status)
3. Use etcdctl member promote <member_id> to promote it to a full member

etcd_data

Parameter: etcd_data, Type: path, Level: C

etcd data directory, default is /data/etcd.

etcd_port

Parameter: etcd_port, Type: port, Level: C

etcd client port, default is 2379.

etcd_peer_port

Parameter: etcd_peer_port, Type: port, Level: C

etcd peer port, default is 2380.

etcd_init

Parameter: etcd_init, Type: enum, Level: C

etcd initial cluster state, can be new or existing, default value: new.

Option Values:

Important Notes:

Usage Examples:

# Create new cluster (default behavior)
./etcd.yml

# Add new member to existing cluster
./etcd.yml -l <new_ip> -e etcd_init=existing

# Or use the convenience script (automatically sets etcd_init=existing)
bin/etcd-add <new_ip>

etcd_election_timeout

Parameter: etcd_election_timeout, Type: int, Level: C

etcd election timeout, default is 1000 (milliseconds), i.e., 1 second.

etcd_heartbeat_interval

Parameter: etcd_heartbeat_interval, Type: int, Level: C

etcd heartbeat interval, default is 100 (milliseconds).

etcd_root_password

Parameter: etcd_root_password, Type: password, Level: G

etcd root user password for RBAC authentication, default value is Etcd.Root.

Pigsty v4.0 enables etcd RBAC (Role-Based Access Control) authentication by default. During cluster initialization, the etcd_auth task automatically creates the root user and enables authentication.

Password Storage Location:

• Password is stored in /etc/etcd/etcd.pass file
• File permissions are 0640 (owned by root, readable by etcd group)
• The etcdctl environment script /etc/profile.d/etcdctl.sh automatically reads this file

Integration with Other Components:

• Patroni uses the pg_etcd_password parameter to configure the password for connecting to etcd
• If pg_etcd_password is empty, Patroni will use the cluster name as password (not recommended)
• VIP-Manager also requires the same authentication credentials to connect to etcd

Security Recommendations:

etcd_root_password: 'YourSecurePassword'

ETCD_REMOVE

This section contains parameters for the etcd_remove role, which are action flags used by the etcd-rm.yml playbook.

Parameters are defined in roles/etcd_remove/defaults/main.yml

etcd_safeguard: false             # prevent purging running etcd instances?
etcd_rm_data: true                # remove etcd data and config files during removal?
etcd_rm_pkg: false                # uninstall etcd packages during removal?

etcd_safeguard

Parameter: etcd_safeguard, Type: bool, Level: G/C/A

Safeguard to prevent purging running etcd instances? Default value is false.

When enabled, the etcd-rm.yml playbook will abort when detecting running etcd instances, preventing accidental deletion of active etcd clusters.

Recommended Settings:

In emergencies, you can override the configuration with command-line parameters:

./etcd-rm.yml -e etcd_safeguard=false

etcd_rm_data

Parameter: etcd_rm_data, Type: bool, Level: G/C/A

Remove etcd data and configuration files during removal? Default value is true.

When enabled, the etcd-rm.yml playbook will delete the following contents when removing a cluster or member:

• /etc/etcd/ - Configuration directory (including certificates and password files)
• /var/lib/etcd/ - Alternate data directory
• {{ etcd_data }} - Primary data directory (default /data/etcd)
• {{ systemd_dir }}/etcd.service - Systemd service unit file
• /etc/profile.d/etcdctl.sh - Client environment script
• /etc/vector/etcd.yaml - Vector log collection config

Use Cases:

# Stop service only, preserve data
./etcd-rm.yml -e etcd_rm_data=false

etcd_rm_pkg

Parameter: etcd_rm_pkg, Type: bool, Level: G/C/A

Uninstall etcd packages during removal? Default value is false.

When enabled, the etcd-rm.yml playbook will uninstall etcd packages when removing a cluster or member.

Use Cases:

# Uninstall packages during removal
./etcd-rm.yml -e etcd_rm_pkg=true

3 - Administration

Common etcd admin SOPs:

• Create Cluster: Initialize an etcd cluster
• Destroy Cluster: Destroy an etcd cluster
• CLI Environment: Configure etcd client to access server cluster
• RBAC Authentication: Use etcd RBAC auth
• Reload Config: Update etcd server member list for clients
• Add Member: Add new member to existing etcd cluster
• Remove Member: Remove member from etcd cluster
• Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster

For prod etcd clusters, enable safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use dedicated etcd-rm.yml playbook to destroy etcd cluster. Use caution!

./etcd-rm.yml                         # remove entire etcd cluster
./etcd-rm.yml -e etcd_safeguard=false # override safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

Removal playbook respects etcd_safeguard. If true, playbook aborts to prevent accidental deletion.

CLI Environment

Uses etcd v3 API by default (v2 removed in v3.6+). Pigsty auto-configures env script /etc/profile.d/etcdctl.sh on etcd nodes, loaded on login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

v4.0 enables RBAC auth by default—user auth required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After configuring client env, run etcd CRUD ops:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

v4.0 enables etcd RBAC auth by default. During cluster init, etcd_auth task auto-creates root user and enables auth.

Root user password set by etcd_root_password, default: Etcd.Root. Stored in /etc/etcd/etcd.pass with 0640 perms (root-owned, etcd-group readable).

Strongly recommended to change default password in prod:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password to configure etcd connection password. If empty, Patroni uses cluster name as password (not recommended). Configure separate etcd password per PG cluster in prod.

Reload Config

If etcd cluster membership changes (add/remove members), refresh etcd service endpoint references. These etcd refs in Pigsty need updates:

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script auto-performs:

• Validates IP address validity
• Executes etcd.yml playbook (auto-sets etcd_init=existing)
• Provides safety warnings and countdown
• Prompts config refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

1. Update config inventory: Add new instance to etcd group
2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
3. Initialize new member: Run playbook with etcd_init=existing parameter
4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
5. Reload config: Update etcd endpoint references for all clients

# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

$ bin/etcd-add 10.10.10.11

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script auto-performs:

• Gracefully removes members from cluster
• Stops and disables etcd service
• Cleans up data and config files
• Deregisters from monitoring system

Manual: Step-by-Step

Remove member instance from etcd cluster:

1. Remove from config inventory: Comment out or delete instance, and reload config
2. Kick from cluster: Use etcdctl member remove command
3. Clean up instance: Use etcd-rm.yml playbook to clean up

# Use dedicated removal playbook (recommended)
./etcd-rm.yml -l <ip>

# Or manual
etcdctl member remove <server_id>      # kick from cluster
./etcd-rm.yml -l <ip>                  # clean up instance

$ bin/etcd-rm 10.10.10.12

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    # 10.10.10.12: { etcd_seq: 3 }   # <---- comment out this member
  vars: { etcd_cluster: etcd }

$ ./etcd-rm.yml -l 10.10.10.12

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

• Validates IP addresses in config inventory
• Auto-sets etcd_init=existing parameter
• Executes etcd.yml playbook to complete member addition
• Prompts config refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

• Provides safety warnings and confirmation countdown
• Auto-executes etcd-rm.yml playbook
• Gracefully removes members from cluster
• Cleans up data and config files

4 - Playbook

The ETCD module provides two core playbooks: etcd.yml for installing and configuring etcd clusters, and etcd-rm.yml for removing etcd clusters or members.

etcd.yml

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group, then launches the etcd service.

The following subtasks are available in etcd.yml:

• etcd_assert : Validate etcd identity parameters (etcd_seq must be defined as a non-negative integer)
• etcd_install : Install etcd packages
• etcd_dir : Create etcd data and configuration directories
• etcd_config : Generate etcd configuration
  • etcd_conf : Generate etcd main config file /etc/etcd/etcd.conf
  • etcd_cert : Generate etcd TLS certificates (CA, server cert, private key)
• etcd_member : Add new member to existing cluster (only runs when etcd_init=existing)
• etcd_launch : Launch etcd service
• etcd_auth : Enable RBAC authentication (create root user and enable auth)
• etcd_register : Register etcd to VictoriaMetrics/Prometheus monitoring

etcd-rm.yml

Playbook source: etcd-rm.yml

A dedicated playbook for removing etcd clusters or individual members. The following subtasks are available in etcd-rm.yml:

• etcd_safeguard : Check safeguard and abort if enabled
• etcd_pause : Pause for 3 seconds, allowing user to abort with Ctrl-C
• etcd_deregister : Remove etcd registration from VictoriaMetrics monitoring targets
• etcd_leave : Try graceful leaving etcd cluster before purge
• etcd_svc : Stop and disable etcd service with systemd
• etcd_data : Remove etcd data (disable with etcd_rm_data=false)
• etcd_pkg : Uninstall etcd packages (enable with etcd_rm_pkg=true)

The removal playbook uses the etcd_remove role with the following configurable parameters:

• etcd_safeguard: Prevents accidental removal when set to true
• etcd_rm_data: Controls whether ETCD data is deleted (default: true)
• etcd_rm_pkg: Controls whether ETCD packages are uninstalled (default: false)

Demo

Cheatsheet

Etcd Installation & Configuration:

./etcd.yml                                      # Initialize etcd cluster
./etcd.yml -t etcd_launch                       # Restart entire etcd cluster
./etcd.yml -t etcd_conf                         # Refresh /etc/etcd/etcd.conf with latest state
./etcd.yml -t etcd_cert                         # Regenerate etcd TLS certificates
./etcd.yml -l 10.10.10.12 -e etcd_init=existing # Scale out: add new member to existing cluster

Etcd Removal & Cleanup:

./etcd-rm.yml                                   # Remove entire etcd cluster
./etcd-rm.yml -l 10.10.10.12                    # Remove single etcd member
./etcd-rm.yml -e etcd_safeguard=false           # Override safeguard to force removal
./etcd-rm.yml -e etcd_rm_data=false             # Stop service only, preserve data
./etcd-rm.yml -e etcd_rm_pkg=true               # Also uninstall etcd packages

Convenience Scripts:

bin/etcd-add <ip>                               # Add new member to existing cluster (recommended)
bin/etcd-rm <ip>                                # Remove specific member from cluster (recommended)
bin/etcd-rm                                     # Remove entire etcd cluster

Safeguard

To prevent accidental deletion, Pigsty’s ETCD module provides a safeguard mechanism controlled by the etcd_safeguard parameter, which defaults to false (safeguard disabled).

For production etcd clusters that have been initialized, it’s recommended to enable the safeguard to prevent accidental deletion of existing etcd instances:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true  # Enable safeguard protection

When etcd_safeguard is set to true, the etcd-rm.yml playbook will detect running etcd instances and abort to prevent accidental deletion. You can override this behavior using command-line parameters:

./etcd-rm.yml -e etcd_safeguard=false  # Force override safeguard

Unless you clearly understand what you’re doing, we do not recommend arbitrarily removing etcd clusters.

5 - Monitoring

Dashboards

ETCD module provides one monitoring dashboard: Etcd Overview.

ETCD Overview Dashboard

ETCD Overview: Overview of ETCD cluster

Dashboard provides key ETCD status info. Notable: ETCD Aliveness—shows overall etcd cluster service status.

Red bands = instance downtime; blue-gray below = cluster unavailable.

Alert Rules

Pigsty provides 5 preset alert rules for etcd, defined in files/prometheus/rules/etcd.yml:

• EtcdServerDown: etcd node down, CRIT alert
• EtcdNoLeader: etcd cluster no leader, CRIT alert
• EtcdQuotaFull: etcd quota > 90%, WARN alert
• EtcdNetworkPeerRTSlow: etcd network latency slow, INFO alert
• EtcdWalFsyncSlow: etcd disk fsync slow, INFO alert

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# etcd server instance down
- alert: EtcdServerDown
  expr: etcd_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      etcd_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# Etcd no Leader triggers P0 alert immediately
# if dcs_failsafe mode not enabled, may cause global outage
- alert: EtcdNoLeader
  expr: min(etcd_server_has_leader) by (cls) < 1
  for: 15s
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      etcd_server_has_leader[cls={{ $labels.cls }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

#==============================================================#
#                        Saturation                            #
#==============================================================#
- alert: EtcdQuotaFull
  expr: etcd:cls:quota_usage > 0.90
  for: 1m
  labels: { level: 1, severity: WARN, category: etcd }
  annotations:
    summary: "WARN EtcdQuotaFull: {{ $labels.cls }}"
    description: |
      etcd:cls:quota_usage[cls={{ $labels.cls }}] = {{ $value | printf "%.3f" }} > 90%
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Latency                              #
#==============================================================#
# etcd network peer rt p95 > 200ms for 1m
- alert: EtcdNetworkPeerRTSlow
  expr: etcd:ins:network_peer_rt_p95_5m > 0.200
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdNetworkPeerRTSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:network_peer_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 200ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}
# Etcd wal fsync rt p95 > 50ms
- alert: EtcdWalFsyncSlow
  expr: etcd:ins:wal_fsync_rt_p95_5m > 0.050
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdWalFsyncSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:wal_fsync_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 50ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

6 - Metrics

The ETCD module has 177 available metrics.

7 - FAQ

What is etcd’s role in Pigsty?

etcd is a distributed, reliable key-value store for critical system data. Pigsty uses etcd as DCS (Distributed Config Store) service for Patroni, storing PG HA status.

Patroni uses etcd for: cluster failure detection, auto failover, primary-replica switchover, and cluster config management.

etcd is critical for PG HA. etcd’s availability and DR ensured through multiple distributed nodes.

What’s the appropriate etcd cluster size?

If more than half (including exactly half) of etcd instances unavailable, etcd cluster enters unavailable state—refuses service.

Example: 3-node cluster allows max 1 node failure while 2 others continue; 5-node cluster tolerates 2 node failures.

Note: Learner instances don’t count toward members—3-node cluster with 1 learner = 2 actual members, zero fault tolerance.

In prod, use odd number of instances. For prod, recommend 3-node or 5-node for reliability.

Impact of etcd unavailability?

If etcd cluster unavailable, affects PG control plane but not data plane—existing PG clusters continue running, but Patroni management ops fail.

During etcd failure: PG HA can’t auto failover, can’t use patronictl for PG management (config changes, manual failover, etc.).

Ansible playbooks unaffected by etcd failure: create DB, create user, refresh HBA/Service config. During etcd failure, operate PG clusters directly.

Note: Behavior applies to Patroni >=3.0 (Pigsty >=2.0). With older Patroni (<3.0, Pigsty 1.x), etcd/consul failure causes severe global impact:

All PG clusters demote: primaries → replicas, reject writes, etcd failure amplifies to global PG failure. Patroni 3.0 introduced DCS Failsafe—significantly improved.

What data does etcd store?

In Pigsty, etcd is PG HA only—no other config/state data.

PG HA component Patroni auto-generates and manages etcd data. If lost in etcd, Patroni auto-rebuilds.

Thus, by default, etcd in Pigsty = “stateless service”—destroyable and rebuildable, simplifies maintenance.

If using etcd for other purposes (K8s metadata, custom storage), backup etcd data yourself and restore after cluster recovery.

Recover from etcd failure?

Since etcd in Pigsty = PG HA only = “stateless service”—disposable, rebuildable. Failures? “restart” or “reset” to stop bleeding.

Restart etcd cluster:

./etcd.yml -t etcd_launch

Reset etcd cluster:

./etcd.yml

For custom etcd data: backup and restore after recovery.

Etcd maintenance considerations?

Simple answer: don’t fill up etcd.

Pigsty v2.6+ enables etcd auto-compaction and 16GB backend quota—usually fine.

etcd’s data model = each write generates new version.

Frequent writes (even few keys) = growing etcd DB size. At capacity limit, etcd rejects writes → PG HA breaks.

Pigsty’s default etcd config includes optimizations:

auto-compaction-mode: periodic      # periodic auto compaction
auto-compaction-retention: "24h"    # retain 24 hours history
quota-backend-bytes: 17179869184    # 16 GiB quota

More details: etcd official maintenance guide.

Enable etcd auto garbage collection?

Earlier Pigsty (v2.0 - v2.5)? Enable etcd auto-compaction in prod to avoid quota-based unavailability.

Edit etcd config template: roles/etcd/templates/etcd.conf.j2:

auto-compaction-mode: periodic
auto-compaction-retention: "24h"
quota-backend-bytes: 17179869184

Then set related PG clusters to maintenance mode and redeploy etcd with ./etcd.yml.

This increases default quota from 2 GiB → 16 GiB, retains last 24h writes—avoids infinite growth.

Where is PG HA data stored in etcd?

By default, Patroni uses pg_namespace prefix (default: /pg) for all metadata keys, followed by PG cluster name.

Example: PG cluster pg-meta stores metadata under /pg/pg-meta.

etcdctl get /pg/pg-meta --prefix

Sample data:

/pg/pg-meta/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":false,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":16,"pg_cluster":"pg-meta","pg_shard":"pg-meta","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":100,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":"on","wal_level":"logical","wal_log_hints":"on","max_worker_processes":16,"max_wal_senders":50,"max_replication_slots":50,"password_encryption":"scram-sha-256","ssl":"on","ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/pg/cert/ca.crt","shared_buffers":"7969MB","maintenance_work_mem":"1993MB","work_mem":"79MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"7GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"7GB","max_wal_size":"28GB","max_slot_wal_keep_size":"42GB","wal_buffers":"16MB","wal_writer_delay":"20ms","wal_writer_flush_after":"1MB","commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":"on","archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-meta archive-push %p","max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":"on","wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"23907MB","default_statistics_target":200,"log_destination":"csvlog","logging_collector":"on","l...
ode=prefer"}}
/pg/pg-meta/failsafe
{"pg-meta-2":"http://10.10.10.11:8008/patroni","pg-meta-1":"http://10.10.10.10:8008/patroni"}
/pg/pg-meta/initialize
7418384210787662172
/pg/pg-meta/leader
pg-meta-1
/pg/pg-meta/members/pg-meta-1
{"conn_url":"postgres://10.10.10.10:5432/postgres","api_url":"http://10.10.10.10:8008/patroni","state":"running","role":"primary","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"timeline":1}
/pg/pg-meta/members/pg-meta-2
{"conn_url":"postgres://10.10.10.11:5432/postgres","api_url":"http://10.10.10.11:8008/patroni","state":"running","role":"replica","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"replication_state":"streaming","timeline":1}
/pg/pg-meta/status
{"optime":184549376,"slots":{"pg_meta_2":184549376,"pg_meta_1":184549376},"retain_slots":["pg_meta_1","pg_meta_2"]}

Use external existing etcd cluster?

Config inventory hardcodes etcd group—members used as DCS servers for PGSQL. Initialize with etcd.yml or assume external cluster exists.

To use external etcd: define as usual. Skip etcd.yml execution since cluster exists—no deployment needed.

Requirement: external etcd cluster certificate must use same CA as Pigsty—otherwise clients can’t use Pigsty’s self-signed certs.

Add new member to existing etcd cluster?

For detailed process, refer to Add member to etcd cluster

Recommended: Utility script

# First add new member to config inventory, then:
bin/etcd-add <ip>      # add single new member
bin/etcd-add <ip1>     # add multiple new members

Manual method:

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380 # announce new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing                                 # initialize new member
etcdctl member promote <new_ins_server_id>                                       # promote to full member

Recommend: add one new member at a time.

Remove member from existing etcd cluster?

For detailed process, refer to Remove member from etcd cluster

Recommended: Utility script

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm                   # remove entire etcd cluster

Manual method:

./etcd-rm.yml -l <ins_ip>                    # use dedicated removal playbook
etcdctl member remove <etcd_server_id>       # kick from cluster
./etcd-rm.yml -l <ins_ip>                    # clean up instance

Configure etcd RBAC authentication?

Pigsty v4.0 enables etcd RBAC auth by default. Root password set by etcd_root_password, default: Etcd.Root.

Prod recommendation: change default password

all:
  vars:
    etcd_root_password: 'YourSecurePassword'

Client auth:

# On etcd nodes, env vars auto-configured
source /etc/profile.d/etcdctl.sh
etcdctl member list

# Manual auth config
export ETCDCTL_USER="root:YourSecurePassword"
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

More: RBAC Authentication.