Add bounded direct ProxySQL capacity probes

doc/mysql.md

@@ -203,7 +203,8 @@ primary-load and ProxySQL-capacity checks without enforcing them:
   "prod4": {
     "ShadowRequiredClusters": [
       "prod4-primary",
-      "prod4-proxysql"
+      "prod4-proxysql-frontend",
+      "prod4-proxysql-writer-backend"
     ],
     "HAProxySettings": {
       "Host": "my.haproxy.mydomain.com",
@@ -224,25 +225,54 @@ primary-load and ProxySQL-capacity checks without enforcing them:
       ]
     }
   },
-  "prod4-proxysql": {
+  "prod4-proxysql-frontend": {
     "User": "${proxysql_stats_user}",
     "Password": "${proxysql_stats_password}",
-    "MetricQuery": "select connection_pressure_percent from operational_metrics.proxysql_capacity limit 1",
-    "CacheMillis": 500,
+    "Port": 6032,
     "ThrottleThreshold": 80,
     "RecoveryThreshold": 60,
-    "RecoveryDurationMillis": 5000,
+    "RecoveryDurationMillis": 30000,
     "FailOnNoHosts": true,
+    "ProxySQLCapacitySettings": {
+      "Metric": "frontend",
+      "FrontendMaxConnections": 200000,
+      "WriterHostgroupID": 30,
+      "BackendLimits": [
+        {"Host": "/var/run/mysqld/mysqld.sock", "Port": 0, "MaxConnections": 1000}
+      ]
+    },
     "StaticHostsSettings": {
-      "Hosts": [
-        "my.prod4.proxysql.example.com:6032"
+      "Hosts": ["my.prod4.proxysql.example.com"]
+    }
+  },
+  "prod4-proxysql-writer-backend": {
+    "User": "${proxysql_stats_user}",
+    "Password": "${proxysql_stats_password}",
+    "Port": 6032,
+    "ThrottleThreshold": 80,
+    "RecoveryThreshold": 60,
+    "RecoveryDurationMillis": 30000,
+    "FailOnNoHosts": true,
+    "ProxySQLCapacitySettings": {
+      "Metric": "writer-backend",
+      "FrontendMaxConnections": 200000,
+      "WriterHostgroupID": 30,
+      "BackendLimits": [
+        {"Host": "/var/run/mysqld/mysqld.sock", "Port": 0, "MaxConnections": 1000}
       ]
+    },
+    "StaticHostsSettings": {
+      "Hosts": ["my.prod4.proxysql.example.com"]
     }
   }
 }
 ```
 
-The ProxySQL query above is illustrative: deployments must provide a scalar query or operational view whose value increases as usable connection capacity is consumed.
+The capacity limits and thresholds above are examples, not production-approved
+values. Capacity probes execute built-in queries against ProxySQL's `stats`
+database, not an application view or a MySQL backend. They require explicit
+stats credentials and do not inherit MySQL credentials, queries, host ignores,
+or the legacy fail-open no-host behavior.
 
 - `RequiredClusters` lists additional configured metrics that must return `HTTP 200`. Missing required runtime metrics fail closed.
 - `ShadowRequiredClusters` lists prospective dependencies, evaluated only after
@@ -308,6 +338,63 @@ zero on its first observation; dependency/status counters appear only on
 failures. Counters reset on restart and do not replace probe health, discovery,
 or coverage validation.
 
+### Direct ProxySQL capacity sampling
+
+`ProxySQLCapacitySettings.Metric` selects `frontend` or `writer-backend`.
+Frontend utilization is `100 * Client_Connections_connected /
+FrontendMaxConnections`. Writer utilization is the maximum of `100 * ConnUsed /
+MaxConnections` across the expected backend endpoints in `WriterHostgroupID`.
+`ConnFree` is not the configured capacity; `ConnUsed / (ConnUsed + ConnFree)` is
+not used. Unknown ONLINE backends, duplicate endpoints, missing expected
+backends, non-ONLINE expected backends, invalid counters, and query failures
+reject writer capacity checks. Frontend and backend checks must both be attached
+when enforcing full connection-capacity protection.
+
+The stats-only account cannot read runtime configuration tables. Operators must
+verify every configured limit against runtime before enabling collection, and
+revalidate limits whenever backend membership or ProxySQL configuration changes.
+All proxies discovered by one configured metric must have matching verified
+limits and backend endpoint mappings; otherwise split them into explicit probes.
+There is no automatic runtime-limit drift detection.
+
+Only the leader samples. Both metrics share one reusable admin connection and
+one five-second snapshot per proxy, credential set, and writer hostgroup.
+A complete snapshot uses two small stats queries, at most 0.4 queries/second in
+steady state. Failed polls replace the previous snapshot and are cached for the
+same five seconds; application requests cannot trigger a hot retry loop.
+The entire two-query batch has a one-second deadline, with one-second dial,
+read, and write limits. Inactive endpoint pools are retired after ten minutes.
+
+An admin port must be explicit or default to 6032; common application ports
+3306 and 6033 are rejected. For capacity probes, discovered port numbers are
+replaced by the configured admin port, including Vitess tablet ports. HAProxy
+and Vitess discovery are only appropriate after confirming that ProxySQL is
+colocated with those database hosts. Detached deployments require an explicit
+proxy roster. `ProxySQLSettings` is not suitable: it discovers MySQL backends.
+Capacity probes do not inherit the replica HTTP check or honor host-skipping
+settings that could hide a saturated proxy.
+
+These queries add no SQL or connections to the MySQL backend. They still consume
+ProxySQL CPU/memory, including shared resources on a colocated primary host.
+Measure that overhead during shadow rollout; do not claim zero host overhead.
+No listener, permissions, deployment, or enforcement is enabled by adding code.
+
+The opt-in integration test uses an isolated ProxySQL fixture with monitoring
+disabled and a listening backend sentinel. It verifies successful stats-only
+sampling while the sentinel receives zero connections:
+
+```bash
+docker run --rm --name freno-capacity-test \
+  --mount type=bind,src="$PWD/pkg/proxysql/testdata/capacity.cnf",dst=/etc/proxysql.cnf,readonly \
+  proxysql/proxysql:2.7.3
+# In another shell; no container ports need to be published:
+docker run --rm --network container:freno-capacity-test \
+  --mount type=bind,src="$PWD",dst=/source,readonly -w /source \
+  -e FRENO_PROXYSQL_CAPACITY_INTEGRATION=1 golang:1.20 \
+  go test -mod=vendor ./pkg/proxysql -run TestCapacityIntegrationNoBackendConnections -count=1
+docker stop freno-capacity-test
+```
+
 ### Rolling out primary safety checks
 
 Deploy a binary containing these settings before deploying configuration that

pkg/config/mysql_config.go

@@ -30,10 +30,11 @@ type MySQLClusterConfigurationSettings struct {
 	HttpCheckPath          string   // Specify if different than specified by MySQLConfigurationSettings
 	IgnoreHosts            []string // override MySQLConfigurationSettings's, or leave empty to inherit those settings
 
-	HAProxySettings     HAProxyConfigurationSettings  // If list of servers is to be acquired via HAProxy, provide this field
-	ProxySQLSettings    ProxySQLConfigurationSettings // If list of servers is to be acquired via ProxySQL, provide this field
-	VitessSettings      VitessConfigurationSettings   // If list of servers is to be acquired via Vitess, provide this field
-	StaticHostsSettings StaticHostsConfigurationSettings
+	HAProxySettings          HAProxyConfigurationSettings  // If list of servers is to be acquired via HAProxy, provide this field
+	ProxySQLSettings         ProxySQLConfigurationSettings // If list of servers is to be acquired via ProxySQL, provide this field
+	VitessSettings           VitessConfigurationSettings   // If list of servers is to be acquired via Vitess, provide this field
+	StaticHostsSettings      StaticHostsConfigurationSettings
+	ProxySQLCapacitySettings *ProxySQLCapacityConfigurationSettings
 }
 
 // Hook to implement adjustments after reading each configuration file.
@@ -118,6 +119,15 @@ func (settings *MySQLConfigurationSettings) postReadAdjustments() error {
 				return fmt.Errorf("Stores.MySQL.Clusters.%s.VitessSettings: %w", clusterName, err)
 			}
 		}
+		if clusterSettings.ProxySQLCapacitySettings != nil {
+			if !clusterSettings.VitessSettings.IsEmpty() && len(clusterSettings.VitessSettings.Cells) == 0 {
+				clusterSettings.VitessSettings.Cells = settings.VitessCells
+			}
+			if err := clusterSettings.adjustProxySQLCapacity(); err != nil {
+				return fmt.Errorf("Stores.MySQL.Clusters.%s.ProxySQLCapacitySettings: %w", clusterName, err)
+			}
+			continue
+		}
 		if clusterSettings.User == "" {
 			clusterSettings.User = settings.User
 		}

pkg/config/proxysql_capacity_config.go

@@ -0,0 +1,101 @@
+package config
+
+import (
+	"fmt"
+	"math"
+)
+
+const ProxySQLDefaultAdminPort = 6032
+
+type ProxySQLBackendLimit struct {
+	Host           string
+	Port           int
+	MaxConnections int64
+}
+
+// ProxySQLCapacityConfigurationSettings uses explicitly verified limits because
+// ProxySQL's stats-only account cannot read the runtime configuration tables.
+type ProxySQLCapacityConfigurationSettings struct {
+	Metric                 string
+	FrontendMaxConnections int64
+	WriterHostgroupID      int
+	BackendLimits          []ProxySQLBackendLimit
+}
+
+func (settings *ProxySQLCapacityConfigurationSettings) Validate() error {
+	if settings.Metric != "frontend" && settings.Metric != "writer-backend" {
+		return fmt.Errorf("Metric must be frontend or writer-backend")
+	}
+	if settings.FrontendMaxConnections <= 0 || settings.WriterHostgroupID <= 0 {
+		return fmt.Errorf("FrontendMaxConnections and WriterHostgroupID must be positive")
+	}
+	if len(settings.BackendLimits) == 0 {
+		return fmt.Errorf("BackendLimits must identify the expected writer backends")
+	}
+	seen := make(map[string]bool)
+	for _, backend := range settings.BackendLimits {
+		if backend.Host == "" || backend.Port < 0 || backend.Port > 65535 || backend.MaxConnections <= 0 {
+			return fmt.Errorf("BackendLimits must contain a host, valid port (0 for a Unix socket), and positive MaxConnections")
+		}
+		key := fmt.Sprintf("%s:%d", backend.Host, backend.Port)
+		if seen[key] {
+			return fmt.Errorf("duplicate BackendLimits endpoint %q", key)
+		}
+		seen[key] = true
+	}
+	return nil
+}
+
+func (settings *MySQLClusterConfigurationSettings) adjustProxySQLCapacity() error {
+	if err := settings.ProxySQLCapacitySettings.Validate(); err != nil {
+		return err
+	}
+	if settings.User == "" || settings.Password == "" {
+		return fmt.Errorf("explicit stats-only User and Password are required; MySQL credentials are not inherited")
+	}
+	if settings.MetricQuery != "" {
+		return fmt.Errorf("MetricQuery must be empty; capacity probes only execute built-in ProxySQL stats queries")
+	}
+	if settings.Port == 0 {
+		settings.Port = ProxySQLDefaultAdminPort
+	}
+	if settings.Port < 1 || settings.Port > 65535 || settings.Port == 3306 || settings.Port == 6033 {
+		return fmt.Errorf("Port must identify a ProxySQL admin listener, not a MySQL or ProxySQL application port")
+	}
+	if len(settings.ProxySQLSettings.Addresses) > 0 || settings.ProxySQLSettings.HostgroupID != 0 {
+		return fmt.Errorf("ProxySQLSettings discovers MySQL backends, not ProxySQL capacity endpoints")
+	}
+	discoverySources := 0
+	for _, present := range []bool{
+		!settings.HAProxySettings.IsEmpty(),
+		!settings.VitessSettings.IsEmpty(),
+		!settings.StaticHostsSettings.IsEmpty(),
+	} {
+		if present {
+			discoverySources++
+		}
+	}
+	if discoverySources != 1 {
+		return fmt.Errorf("exactly one HAProxy, Vitess, or static proxy-host discovery source is required")
+	}
+	if math.IsNaN(settings.ThrottleThreshold) || math.IsInf(settings.ThrottleThreshold, 0) ||
+		settings.ThrottleThreshold <= 0 || settings.ThrottleThreshold > 100 {
+		return fmt.Errorf("ThrottleThreshold must be a finite percentage in (0, 100]")
+	}
+	if settings.RecoveryThreshold == nil || math.IsNaN(*settings.RecoveryThreshold) ||
+		math.IsInf(*settings.RecoveryThreshold, 0) || *settings.RecoveryThreshold < 0 ||
+		*settings.RecoveryThreshold >= settings.ThrottleThreshold {
+		return fmt.Errorf("RecoveryThreshold must be a finite non-negative percentage below ThrottleThreshold")
+	}
+	if settings.RecoveryDurationMillis < 10000 {
+		return fmt.Errorf("RecoveryDurationMillis must be at least 10000 (two sampling intervals)")
+	}
+	if settings.IgnoreHostsCount != 0 || settings.IgnoreHostsThreshold != 0 || len(settings.IgnoreHosts) != 0 {
+		return fmt.Errorf("capacity probes must not ignore hosts or errors")
+	}
+	settings.FailOnNoHosts = true
+	settings.HttpCheckPort = -1
+	settings.HttpCheckPath = ""
+	settings.CacheMillis = 0 // The sampler shares a bounded success/error cache across both capacity metrics.
+	return nil
+}

pkg/config/proxysql_capacity_config_test.go

@@ -0,0 +1,75 @@
+package config
+
+import (
+	"strings"
+	"testing"
+)
+
+func capacityClusterSettings() *MySQLClusterConfigurationSettings {
+	recovery := 70.0
+	return &MySQLClusterConfigurationSettings{
+		User: "stats", Password: "test-only", ThrottleThreshold: 90,
+		RecoveryThreshold: &recovery, RecoveryDurationMillis: 30000,
+		StaticHostsSettings: StaticHostsConfigurationSettings{Hosts: []string{"proxy.example"}},
+		ProxySQLCapacitySettings: &ProxySQLCapacityConfigurationSettings{
+			Metric: "frontend", FrontendMaxConnections: 200000, WriterHostgroupID: 30,
+			BackendLimits: []ProxySQLBackendLimit{{Host: "/var/run/mysqld/mysqld.sock", Port: 0, MaxConnections: 1000}},
+		},
+	}
+}
+
+func TestCapacityDoesNotInheritMySQLProbeSettings(t *testing.T) {
+	cluster := capacityClusterSettings()
+	settings := MySQLConfigurationSettings{
+		User: "mysql-user", Password: "mysql-password", Port: 3306,
+		MetricQuery: "SELECT mysql_only", IgnoreHostsCount: 1, IgnoreHosts: []string{"proxy"},
+		HttpCheckPort: 9210, CacheMillis: 20000,
+		Clusters: map[string]*MySQLClusterConfigurationSettings{"capacity": cluster},
+	}
+	if err := settings.postReadAdjustments(); err != nil {
+		t.Fatal(err)
+	}
+	if cluster.Port != 6032 || cluster.MetricQuery != "" || cluster.IgnoreHostsCount != 0 ||
+		len(cluster.IgnoreHosts) != 0 || cluster.HttpCheckPort != -1 || cluster.CacheMillis != 0 || !cluster.FailOnNoHosts {
+		t.Fatalf("capacity inherited unsafe MySQL settings: %+v", cluster)
+	}
+}
+
+func TestCapacityConfigurationRejectsUnsafeInputs(t *testing.T) {
+	tests := []struct {
+		name   string
+		change func(*MySQLClusterConfigurationSettings)
+	}{
+		{"missing user", func(c *MySQLClusterConfigurationSettings) { c.User = "" }},
+		{"missing password environment", func(c *MySQLClusterConfigurationSettings) { c.Password = "${FRENO_TEST_MISSING_STATS_PASSWORD}" }},
+		{"MySQL port", func(c *MySQLClusterConfigurationSettings) { c.Port = 3306 }},
+		{"application port", func(c *MySQLClusterConfigurationSettings) { c.Port = 6033 }},
+		{"custom query", func(c *MySQLClusterConfigurationSettings) { c.MetricQuery = "SELECT mysql_only" }},
+		{"missing discovery", func(c *MySQLClusterConfigurationSettings) { c.StaticHostsSettings.Hosts = nil }},
+		{"backend discovery", func(c *MySQLClusterConfigurationSettings) { c.ProxySQLSettings.Addresses = []string{"proxy:6032"} }},
+		{"ignored hosts", func(c *MySQLClusterConfigurationSettings) { c.IgnoreHostsCount = 1 }},
+		{"no recovery", func(c *MySQLClusterConfigurationSettings) { c.RecoveryDurationMillis = 0 }},
+		{"invalid throttle", func(c *MySQLClusterConfigurationSettings) { c.ThrottleThreshold = 101 }},
+		{"missing recovery threshold", func(c *MySQLClusterConfigurationSettings) { c.RecoveryThreshold = nil }},
+		{"invalid metric", func(c *MySQLClusterConfigurationSettings) { c.ProxySQLCapacitySettings.Metric = "unknown" }},
+		{"zero frontend capacity", func(c *MySQLClusterConfigurationSettings) { c.ProxySQLCapacitySettings.FrontendMaxConnections = 0 }},
+		{"zero backend capacity", func(c *MySQLClusterConfigurationSettings) {
+			c.ProxySQLCapacitySettings.BackendLimits[0].MaxConnections = 0
+		}},
+		{"missing backend limits", func(c *MySQLClusterConfigurationSettings) { c.ProxySQLCapacitySettings.BackendLimits = nil }},
+	}
+	t.Setenv("FRENO_TEST_MISSING_STATS_PASSWORD", "")
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			cluster := capacityClusterSettings()
+			test.change(cluster)
+			settings := MySQLConfigurationSettings{
+				User: "must-not-inherit", Password: "must-not-inherit",
+				Clusters: map[string]*MySQLClusterConfigurationSettings{"capacity": cluster},
+			}
+			if err := settings.postReadAdjustments(); err == nil || !strings.Contains(err.Error(), "ProxySQLCapacitySettings") {
+				t.Fatalf("unsafe configuration accepted: %v", err)
+			}
+		})
+	}
+}

pkg/mysql/mysql_throttle_metric.go

@@ -10,12 +10,14 @@ import (
 	"strings"
 	"time"
 
+	"github.com/github/freno/pkg/proxysql"
 	"github.com/outbrain/golib/sqlutils"
 	"github.com/patrickmn/go-cache"
 	metrics "github.com/rcrowley/go-metrics"
 )
 
 var mysqlMetricCache = cache.New(cache.NoExpiration, 10*time.Millisecond)
+var proxySQLCapacitySampler = proxysql.NewCapacitySampler()
 
 func getMySQLMetricCacheKey(probe *Probe) string {
 	return fmt.Sprintf("%s:%s", probe.Key, probe.MetricQuery)
@@ -68,9 +70,11 @@ func (metric *MySQLThrottleMetric) Get() (float64, error) {
 // ReadThrottleMetric returns replication lag for a given connection config; either by explicit query
 // or via SHOW SLAVE STATUS
 func ReadThrottleMetric(probe *Probe, clusterName string) (mySQLThrottleMetric *MySQLThrottleMetric) {
-	if mySQLThrottleMetric := getCachedMySQLThrottleMetric(probe); mySQLThrottleMetric != nil {
-		return mySQLThrottleMetric
-		// On cached results we avoid taking latency metrics
+	if probe.ProxySQLCapacitySettings == nil {
+		if mySQLThrottleMetric := getCachedMySQLThrottleMetric(probe); mySQLThrottleMetric != nil {
+			return mySQLThrottleMetric
+			// On cached results we avoid taking latency metrics
+		}
 	}
 
 	started := time.Now()
@@ -88,6 +92,13 @@ func ReadThrottleMetric(probe *Probe, clusterName string) (mySQLThrottleMetric *
 		}()
 	}(mySQLThrottleMetric, started)
 
+	if probe.ProxySQLCapacitySettings != nil {
+		mySQLThrottleMetric.Value, mySQLThrottleMetric.Err = proxySQLCapacitySampler.Read(
+			probe.Key.Hostname, probe.Key.Port, probe.User, probe.Password, *probe.ProxySQLCapacitySettings,
+		)
+		return mySQLThrottleMetric
+	}
+
 	dbUri := probe.GetDBUri("information_schema")
 	db, fromCache, err := sqlutils.GetDB(dbUri)