Administration

Use these standard procedures to administer etcd:

• Create Cluster: Initialize an etcd cluster
• Destroy Cluster: Destroy an etcd cluster
• CLI Environment: Configure etcd client to access server cluster
• RBAC Authentication: Use etcd RBAC auth
• Reload Config: Update etcd server member list for clients
• Add Member: Add new member to existing etcd cluster
• Remove Member: Remove member from etcd cluster
• Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster

For production etcd clusters, enable the safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use the dedicated etcd-rm.yml playbook to destroy an Etcd cluster. The default etcd_rm_data: true deletes local data and configuration. First, confirm that no PostgreSQL cluster still uses it as DCS. Verify a recent backup and check the exact target name.

./etcd-rm.yml -l etcd                            # Destroy the entire Etcd cluster after confirmation
./etcd-rm.yml -l etcd -e etcd_safeguard=false   # Override only when the inventory enables the safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

The removal playbook respects etcd_safeguard. If this parameter is true, the playbook aborts before any cluster-leave, deregistration, service-stop, or file-deletion operation. The default is false. The absence of an explicit override does not confirm removal.

CLI Environment

Pigsty uses the etcd v3 API by default. Etcd v3.6+ removed the v2 API. Pigsty automatically configures /etc/profile.d/etcdctl.sh on etcd nodes. The shell loads this environment script at login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

Since v4.0, Pigsty enables RBAC authentication for etcd by default. User authentication is still required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After you configure the client environment, run etcd CRUD operations:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

Since v4.0, Pigsty enables etcd RBAC authentication by default. During cluster initialization, the etcd_auth task creates the root user and enables authentication.

Set the root user password with etcd_root_password. The default is Etcd.Root. Pigsty stores it in /etc/etcd/etcd.pass with 0640 permissions. Root owns the file, and the etcd group can read it.

We strongly recommend that you change the default password in production:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password as the etcd connection password. If it is empty, Patroni uses the cluster name as the password. We do not recommend this default. In production, set a separate etcd password for each PostgreSQL cluster.

Reload Config

After you add or remove etcd members, update the service endpoint references. Update these etcd references in Pigsty:

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

The script attempts these operations in order:

• Validates the IP address
• Sets etcd_init=existing automatically and runs the etcd.yml playbook
• Provides safety warnings and countdown
• Shows configuration refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

1. Update config inventory: Add new instance to etcd group
2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
3. Initialize new member: Run playbook with etcd_init=existing parameter
4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
5. Reload config: Update etcd endpoint references for all clients

# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

$ bin/etcd-add 10.10.10.11

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

The script automatically performs these operations:

• Gracefully removes members from cluster
• Stops and disables etcd service
• Cleans up data and config files
• Deregisters from monitoring system

The underlying removal role tolerates some leave and cleanup errors. After the script finishes, inspect etcdctl member list, endpoint health, and remaining quorum. Also inspect the actual service and data-directory state on the target.

Manual: Step-by-Step

Remove member instance from etcd cluster:

1. Keep the member in the inventory: The removal playbook needs its etcd_seq, cluster members, and connection endpoints
2. Clean up the instance: Run etcd-rm.yml against the target. It attempts member remove and stops the service. It then cleans up according to the removal flags.
3. Update the inventory: Comment out or delete the member only after the playbook succeeds
4. Reload references: Follow Reload Config for the remaining etcd members and the Patroni/VIP-Manager endpoints

# <ip> must still be in the etcd inventory group
./etcd-rm.yml -l <ip>                  # Leave the cluster and clean up automatically
# After success, remove the member from pigsty.yml and refresh the remaining members and clients

Do not delete the target from the inventory before you run the removal playbook. Without the target, the hosts: etcd scope in etcd-rm.yml cannot select it. The playbook also cannot get the instance identity or cluster endpoints from the inventory.

You do not need to repeat etcdctl member remove before or after the playbook.

$ bin/etcd-rm 10.10.10.12

$ ./etcd-rm.yml -l 10.10.10.12

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

• Validates IP addresses in config inventory
• Auto-sets etcd_init=existing parameter
• Executes etcd.yml playbook to complete member addition
• Shows configuration refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

• Provides safety warnings and confirmation countdown
• Auto-executes etcd-rm.yml playbook
• Gracefully removes members from cluster
• Cleans up data and config files

Manage Etcd Password

The etcd_root_password parameter defines the password of the etcd root user.

To change it, use an INFRA node or ETCD node that can reach an etcd endpoint. Run this command as the admin user:

e user passwd root  # change the etcd root user password

Update every reference to the etcd root password. Include the Patroni client configuration on INFRA nodes and the etcdctl client environment:

./infra.yml -t env_patroni    # refresh the etcd root password in /infra/conf/patronictl.yml
./etcd.yml  -t etcd_conf      # refresh /etc/etcd/etcd.pass and /etc/profile.d/etcdctl.sh