After two years of running a custom message broker we finally pulled the plug and migrated everything to NATS JetStream. The migration took six weeks of careful planning, three weekends of late-night cutovers and one production incident that we are still apologising for.
Every team eventually discovers the joy of shipping behind a flag. Fewer teams discover the misery of cleaning up flags two years later when the original author has long since left the company. A short rant and a longer guide.
Internal tools tend to rot the moment they leave the founding team. We rebuilt our admin from scratch around three principles: every action should be auditable, nothing should require a tribal-knowledge runbook, and there should be exactly one button that resolves the on-call page.
Sometimes the right answer is not Hugo or Astro but a file you can read in one sitting. We walked through every line in a guild lunch and somehow shipped a new docs site by the end of the day.
Five years ago this would have been a controversial post. Today we have battle-tested operators, decent storage classes and a community that has finally agreed on backup tooling. Here is the stack we landed on for a fleet of around forty clusters.
Continuous profiling sounds great until you turn it on in a fleet of three hundred pods and your p99 latency doubles. Here is how we instrumented pprof endpoints behind a feature flag, sampled at one percent and shipped flame graphs straight to Grafana.
Most database migrations fail not because the SQL is wrong but because nobody tested them against a realistic dataset. We share the rollout checklist we use for every schema change, including the awkward question of when to lock a table and when to live with the inconsistency.