CI/CD Pipelines Explained: From Push to Production

Part 16 of 17 · ~2 min

Monitoring and Post-Deploy Verification

A green pipeline confirms one thing only: the code that exists got assembled and ran its checks without incident. Whether it actually behaves once real traffic hits it is a separate question the pipeline was never asked, and the gap between those two shows up for real about a month in: a deploy adds a new required configuration value — the OCR vendor's updated API endpoint — to .env.production, but nobody adds the equivalent entry to the pipeline's own secrets and variables, because it's easy to forget a config file that lives outside the repository entirely. The pipeline builds, tests, and deploys without a single complaint, because none of those steps ever touch that configuration value. web's /health endpoint keeps returning 200 the whole time, because the process is genuinely up and genuinely listening — it just can't parse a single receipt, because every attempt to reach the OCR vendor fails immediately on a malformed URL.

Two checks close that gap, and snapledger adds both after this incident rather than before it:

• A smoke test right after deploy, hitting not just /health but an actual receipt-upload endpoint with a known test image, specifically because "the process started" and "the thing it's for actually works" are different claims and only one of them was being checked.
• A few minutes of metrics watched automatically after every deploy — error rate on the upload endpoint specifically, not just overall request count — so a spike shows up as an automatic alert instead of as a support email from Berrywell's bookkeeper, the same way the very first incident in this whole story got noticed.

The broader habit this pushes the team toward is shipping in smaller pieces, more often, instead of saving up a week's worth of changes for one release. When a smoke test does flag something, a deploy that only moved a handful of lines gives you a short list of suspects instead of five days of tangled changes to sort through — and there's correspondingly little sitting on top of the previous good version that a rollback would actually have to throw away.