Monitoring

MySQL metric collection, Grafana dashboards, alert rules, and log queries.

MYSQL sends mysqld_exporter metrics to VictoriaMetrics and Journald/Vector error logs to VictoriaLogs. Grafana supplies five dashboards. vmalert loads 68 recording rules and 27 alerts.

Collection Architecture

Each node runs mysqld_exporter on 9104 with least-privilege dbuser_monitor@'127.0.0.1'. Deployment creates a discovery file on the INFRA node:

/infra/targets/mysql/<instance>.yml     # e.g. my-test-1.yml

The VictoriaMetrics mysql job reads this directory. mysql_exporter_enabled: false empties the target list. Only mysql-rm.yml removes the file.

Collectors cover global status/variables, binlog size, InnoDB, process lists, top-50 statement digests, table/index I/O waits, MGR membership, and replication statistics.

Label Model

All MySQL metrics carry a consistent label set:

Label	Meaning	Example
`job`	Scrape job	`mysql`
`cls`	Cluster name	`my-test`
`ins`	Instance name	`my-test-1`
`ip`	Member address	`10.10.10.11`
`topology`	Topology type	`innodb_cluster` / `standalone`

Derived names use mysql:ins:* for instances and mysql:cls:* for clusters. See Metrics.

Grafana Dashboards

Dashboard	Purpose
MySQL Overview	Fleet view: cluster inventory, health, QPS/TPS, active alerts, instance list
MySQL Cluster	One cluster: member states, workload, node resources, cluster logs
MySQL Instance	One instance: connections, statements, InnoDB, temp tables, locks, logs
MySQL Group Replication	MGR deep dive: roles, certification/applier queues, flow control, read-only safety, GR logs
MySQL Alert	Alert summary and key platform logs

mysql:cls:health reports 2 healthy, 1 degraded but writable, or 0 critical or unwritable. Overview’s Healthy Clusters stat and Cluster Health timeline use it.

Group Replication panels apply only to innodb_cluster. No data is expected for standalone clusters.

Alert Rules

The 27 rules use severity CRIT, WARN, or INFO. Use these alerts for paging:

Availability and Cluster State

Alert	Severity	Condition
`MySQLInstanceDown`	CRIT	Connection probe failing for 1m
`MySQLClusterNoPrimary`	CRIT	No ONLINE primary for 1m
`MySQLClusterQuorumLost`	CRIT	ONLINE members below majority for 1m
`MySQLClusterMultiplePrimary`	CRIT	More than one primary for 30s (split-brain signal)
`MySQLSecondaryWritable`	CRIT	A secondary writable for 2m (divergence risk)
`MySQLClusterMemberOffline`	WARN	A declared member out of the group for 5m
`MySQLPrimaryReadOnly`	WARN	Primary read-only for 5m
`MySQLExporterDown`	WARN	Scrape failing for 2m

Capacity and Performance

Other alerts cover:

Connections: MySQLConnectionsHigh WARN at 80%, MySQLConnectionsCritical CRIT at 95%.
Replication queues: MySQLGRQueueHigh WARN and MySQLGRQueueCritical CRIT.
Flow control: MySQLGRFlowControlHigh.
InnoDB: MySQLBufferPoolWaits, MySQLInnoDBLogWaits, MySQLRedoCapacityHigh, MySQLDeadlocksHigh, and MySQLHistoryListLarge.
INFO hints: slow queries, disk temporary tables, full joins, buffer-pool hit ratio, and recent restarts.

Tests showed approximately 20-second primary crash failovers produced pending alerts without false pages. Complete outages fired ClusterNoPrimary and QuorumLost within two minutes.

Log Queries

MySQL writes errors to /var/log/mysql/error.log and through syslog → Journald → Vector → VictoriaLogs. app=mysqld-<instance> labels support dashboard panels and LogsQL queries:

# Recent errors from one instance
curl -s http://<infra>:9428/select/logsql/query \
  -d 'query=app:mysqld-my-test-1 level:err _time:1h'

# All MySQL-related logs for a cluster, including backup runs
curl -s http://<infra>:9428/select/logsql/query \
  -d 'query=job:syslog cls:my-test (app:~"mysqld-" OR unit:mysql-backup) _time:1h | limit 100'

Log cls uses the node cluster, node_cluster. Match it to mysql_cluster, as examples do, to align metric and log labels.

Known boundaries:

Slow queries: slow.log, with a 1s threshold, remains local. Inspect it on the instance or use statement digests such as mysql:ins:statement_latency.
Router runtime logs: /var/log/mysqlrouter/ also remains local.

Verify the Pipeline

Verify every pipeline stage after deployment:

# Exporter itself
curl -s http://<member>:9104/metrics | grep -E '^mysql_up '

# VictoriaMetrics scrape and recording rules
curl -s 'http://<infra>:8428/api/v1/query?query=mysql_up'
curl -s 'http://<infra>:8428/api/v1/query?query=mysql:cls:health'

# vmalert rule groups (expect mysql-rules and mysql-alerts)
curl -s 'http://<infra>:8880/api/v1/rules' | grep -o '"name":"mysql-[a-z]*"'

# Log ingestion
curl -s 'http://<infra>:9428/select/logsql/query' -d 'query=app:~"mysqld-" _time:1h | stats by (app) count()'