Why the pain of wake up calls leads to gain


I recently wrote an article for Zignal’s engineering blog explaining how we own everything we build: from early development of an idea right through to being on call for the feature in production.

This contrasts with the more traditional, siloed approach where dev/test/release/ops are separate functional teams; each using their own tools and processes.

Initially I found being part of the oncall schedule challenging (yes, EVERYONE on the team including engineering leads and managers feel the pain of 2am calls from PagerDuty).

However, the huge benefits I’ve seen from this practice made me wake up:-

  • Happier customers:- Problems that have the potential to impact our customers (and teammates) are addressed far more quickly because the engineers responsible for building the app can efficiently get to the root cause of the problem and fix it. And leaders/managers quickly prioritize postmortems and fixes.
  • Higher quality:- We build better tests, monitoring hooks and alerting mechanisms at the same time as building features; this helps us both prevent and troubleshoot problems in production.
  • Less alert fatigue:- The overall level of disruption to the team has reduced due to a sharp focus on consolidating duplicate metrics/monitors; this results in far fewer alerts being triggered.
  • Empowered engineers:- We are closer to the customer and have broader understanding of our product, technology, tools and code. This helps us make better choices that keep stability, resiliency and customer top of mind.
  • Faster velocity:- The improvement in quality and efficiency allows us to spend more time working on what we enjoy doing most: building new product features that add value to the product and our customers.

Learn more here.

Leave a Reply

Your email address will not be published. Required fields are marked *