CA and Certificates

Pigsty maintains a self-signed Certificate Authority (CA) on the admin node by default. It signs certificates for PostgreSQL, Patroni, etcd, Silo, Nginx, and other internal services. Public Nginx entries can use Certbot/Let’s Encrypt certificates configured through infra_portal.

Self-Signed CA

The ca stage of infra.yml creates or reuses the CA locally on the admin node running Ansible. It does not create the CA on a remote Infra node. The default layout is:

files/pki/
├── ca/                       # CA key, certificate, and OpenSSL CA state
│   ├── ca.key
│   └── ca.crt
├── csr/                      # certificate signing requests
├── misc/                     # generic certificates issued by cert.yml
├── etcd/
├── infra/
├── kafka/
├── minio/                    # MINIO module (Silo) certificates
├── mongo/
├── mysql/
├── nginx/
└── pgsql/

The core defaults match the v4.5.0 roles:

The role hard-codes the CA certificate lifetime to 36500d (about 100 years). These long-lived certificates serve a controlled internal trust domain. Public browsers do not trust them by default. Clients must explicitly trust ca.crt. Public endpoints should use a publicly trusted CA.

Initialize the local CA stage:

./infra.yml -t ca

Running ./infra.yml -t ca may create a missing key or certificate and therefore changes PKI state. Confirm the admin node, configuration, and existing CA backup first.

Use an External CA

To reuse an enterprise CA:

1. Set ca_create: false in pigsty.yml.
2. Place a matching files/pki/ca/ca.key and files/pki/ca/ca.crt pair on the admin node.
3. Set permissions and verify that the public-key digests match.

chmod 700 files/pki/ca
chmod 600 files/pki/ca/ca.key
chmod 644 files/pki/ca/ca.crt

# The public-key digests must match; neither command prints private-key material
openssl pkey -in files/pki/ca/ca.key -pubout -outform PEM | openssl sha256
openssl x509 -in files/pki/ca/ca.crt -pubkey -noout | openssl sha256

ca_create: false prevents generation of a new private key only when ca.key is missing. If the key exists and ca.crt is missing, the role still creates a new self-signed CA certificate from the key. Always restore the pair together instead of relying on certificate regeneration.

Before running the CA stage, verify the files that will be used, the existing CA backup, and the admin node.

Back Up and Restore the CA

Retain at least:

• files/pki/ca/ca.key and ca.crt
• CA state such as ca.srl, index.txt, and CRL files if issuance/revocation management uses them
• backup time, the CA certificate SHA-256 fingerprint, and restoration instructions

# Inspect the public CA certificate only
openssl x509 -in files/pki/ca/ca.crt -noout -subject -issuer -dates -fingerprint -sha256

Encrypt the backup and store it on controlled offline media or in a secrets-management system. Do not keep an unencrypted tar archive.

First, restore it to an isolated temporary directory. Before replacement, verify the file count, types, permissions, public-key match, and certificate fingerprint.

Loss of ca.key does not immediately prevent verification of existing certificates. Verification still works if clients trust ca.crt and the certificates remain valid and unrevoked.

You can no longer issue, renew, or revoke certificates with the original CA. Recovery usually requires a new CA, reissuance of every certificate, and deployment of a new trust chain.

Issue Certificates with cert.yml

cert.yml runs locally on the admin node and issues generic certificates with the Pigsty CA. Pass cn explicitly instead of relying on the generic script default:

./cert.yml -e cn=dbuser_dba

Default outputs:

files/pki/misc/<cn>.key   # 0600
files/pki/misc/<cn>.crt   # 0600
files/pki/csr/<cn>.csr

Advanced examples:

# Pass DNS/IP SANs as a JSON list
./cert.yml -e cn=myservice \
  -e '{"san":["DNS:myservice.local","DNS:myservice","IP:10.10.10.50"]}'

# Issue a one-year certificate
./cert.yml -e cn=myservice \
  -e '{"san":["DNS:myservice.local","DNS:myservice","IP:10.10.10.50"]}' \
  -e expire=365d

# When customizing key/crt, provide both paths together
./cert.yml -e cn=custom \
  -e key=/secure/path/custom.key \
  -e crt=/secure/path/custom.crt

Verify the certificate without displaying or copying private-key content:

openssl x509 -in files/pki/misc/myservice.crt -noout -subject -issuer -dates -ext subjectAltName
openssl verify -CAfile files/pki/ca/ca.crt files/pki/misc/myservice.crt

For PostgreSQL client certificates, cn must match the database role expected by HBA/cert authentication. Install the certificate, key, and root certificate on the client. Set the private key permissions to 0600. With sslmode=verify-full, the connection hostname must appear in the server certificate SAN.

Trust the CA Certificate

Distribute only the public ca.crt. Never distribute ca.key. Verify its SHA-256 fingerprint through a separate trusted channel before installation.

Debian / Ubuntu

sudo cp ca.crt /usr/local/share/ca-certificates/pigsty-ca.crt
sudo update-ca-certificates

RHEL / Rocky / AlmaLinux

sudo cp ca.crt /etc/pki/ca-trust/source/anchors/pigsty-ca.crt
sudo update-ca-trust

macOS

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ca.crt

Windows (Administrator PowerShell)

Import-Certificate -FilePath .\ca.crt -CertStoreLocation Cert:\LocalMachine\Root

Infra Nginx normally exposes the public CA certificate at http://<infra_ip>/ca.crt. Verify the fingerprint after download. HTTP transport alone does not prove certificate authenticity.

Nginx and Let’s Encrypt

Each infra_portal entry can name a certbot certificate. Pigsty’s /etc/nginx/sign-cert uses Certbot webroot mode. It groups domain and domains entries that share a certificate name. Then /etc/nginx/link-cert links the resulting certificate into Nginx.

Prerequisites:

• Public DNS A/AAAA records resolve to the intended Infra node.
• Port 80 is publicly reachable for HTTP-01, and Nginx serves the ACME webroot.
• certbot_email is valid and the Certbot package is installed.
• Portal domains, additional domains, and certificate names are exact.

certbot_email: [email protected]
infra_portal:
  home:
    domain: example.com
    domains: [www.example.com]
    certbot: example.com
  grafana:
    domain: grafana.example.com
    endpoint: "${admin_ip}:3000"
    websocket: true
    certbot: grafana.example.com

Update the Nginx configuration and issue certificates:

dig +short example.com
./infra.yml -l infra -t nginx_config,nginx_launch

./infra.yml -l infra -t nginx_certbot,nginx_reload -e certbot_sign=true

certbot certificates
test -r /etc/letsencrypt/live/example.com/fullchain.pem
nginx -t
openssl s_client -connect example.com:443 -servername example.com </dev/null

Renewal scheduling depends on the Certbot package for the operating system. Do not add a duplicate cron job before checking existing timers and cron configuration:

systemctl list-timers --all | grep -i certbot
certbot renew --dry-run

After Certbot replaces certificates on disk, Nginx still needs a reload to use them. Configure a renewal deploy hook, such as systemctl reload nginx, or an equivalent managed process. Verify that it works. Treat automatic renewal as proven only after a real or staging renewal exercise.

Troubleshooting and Acceptance

For final acceptance, verify each result separately:

• The certificate content and SANs are correct.
• Chain validation succeeds.
• The service has loaded the new certificate.
• Target clients trust the certificate.
• Renewal is configured and passes a dry run.

A generated file or successful playbook alone does not prove these results.