Luthfi Farabi Senior Database Engineer & DBA, 10+ years keeping production cloud fleets alive —
and building the tools so they mostly keep themselves alive.

The page said "high memory," the cause was three things stacked. No connection pooler in front of a Puma app meant 498 idle connections each holding memory hostage. Long-running statements created lock contention. And a couple of hot queries were doing 23-second sequential scans on tables missing the right index.

The fix went in as a layered stack: PgBouncer in transaction-pooling mode to collapse the connection count, server-side idle timeouts to reap the stragglers, work_mem dialed back to shrink the per-connection memory footprint, and a targeted index plan to kill the seq-scans (so the heavy sorts disappeared instead of needing more memory). Then I wrote the 5-Whys RCA in Confluence so the root cause — not just the symptom — is on record.

A raw ALTER TABLE on a 1TB / 543M-row MySQL table means a full table rewrite — hours of lock, application errors, or a maintenance window nobody wanted. Instead, I used Alibaba Cloud DMS Lock-Free Schema Change: DMS creates a shadow table, streams all ongoing writes from the live table in the background using CDC, then performs an atomic cutover with no locks held on the original table at any point.

Production traffic ran normally throughout the entire operation. Zero downtime, zero application disruption, and the service integration shipped on schedule.

The instinct on a CPU-pinned cluster is to scale up. I scaled down — after finding why it was hot. 843,000 slow queries in 27 hours and 94–98% CPU weren't a capacity problem, they were a query problem. The worst offender was a LATERAL JOIN doing a filesort on every execution; the right composite index removed the sort entirely.

On top of that, over 99% of connections were idle — bloat, not load. With the queries fixed and the connection story cleaned up, the 3× 32vCPU/128GB footprint was wild overkill. Right-sizing the boxes took 40–55% off the bill.

You can't alert on what you've never looked at. I read all 33 panels across the 19-host fleet and wrote down what each one was actually telling me. That surfaced the host quietly running 13,000 QPS with 21 slow queries/sec, another near connection saturation, and 17 blind spots — signals we were graphing but not alerting on.

The output wasn't a complaint, it was a spec: the exact alerts, thresholds and owners needed to turn those blind spots into pages before they turn into incidents.

The thesis: a DBA's judgment can be encoded, and the cloud's control plane can act on it. The roadmap pairs Claude as the brain — anomaly detection, RCA reasoning, index recommendations — with Alibaba RDS as the body that executes scaling, backups and schema changes.

Ten projects, sequenced from safest to boldest: AI anomaly detection, index tuning with shadow validation (test the index against real traffic before it goes live), predictive scaling, automated backup/restore drills, schema-change CI gates, and auto-RCA. Doc first — now turning into running code (see $ now).

Auto-RCA agent — pulls the signals from Grafana, walks a 5-Whys analysis, and writes the draft RCA straight into Confluence via MCP, so the post-incident doc exists before the adrenaline wears off.

Slow-query monitor — a Bash watcher that catches slow queries, hands them to Claude for an explanation + fix suggestion, and drops the whole analysis into Google Chat. Daily standups — auto-generated from Jira via MCP, so the status update writes itself.

A lock-free schema change on a 100GB+ table tripped a known-nasty MySQL virtual generated-column index bug, and the instance started looping through failovers. The safe path: drop the offending index to stop the bleeding, rebuild the table cleanly via DTS, then swap it in with an atomic rename — all inside the agreed maintenance window so no one downstream felt it.

Same week, on the other engine: a MongoDB COLLSCAN on ApsaraDB tracked down and indexed. Different database, same instinct — find the missing index, stop the scan.