What does it cost to declare an incident?

Why engineers hesitate to call an incident, and how to make it easier

This article was first published on Uptime Labs.

You’re on call. A couple of days ago you made some updates to a dashboard, and you’ve just gone back to it, mostly to check whether it’s showing you what you want to see. Something odd catches your eye. It could be nothing, but it’s worth a closer look. This service is generally very reliable, and you haven’t received any alerts. But if it is in fact malfunctioning, it could be a big issue. How much time do you allow yourself to check before calling an incident?

If you’ve been in this situation before, you probably know how you would react. Would you immediately declare an incident, better safe than sorry? Or would you hesitate, hoping it’s not that serious, or that giving it some time will make it go away? Or would you just want to do a little more digging to find out what’s going on first? If you would hesitate, it’s probably not because you’re lazy. More than likely, you would hesitate because you know what comes attached to declaring an incident.

The small things add up

Pick a severity. Fill in the affected products and an impact estimate. Lead the incident. Schedule a postmortem and agree on action items. Depending on your industry, you might even have regulatory reporting deadlines that require certain information.

We declare incidents so we can respond well and learn from them, and every one of these fields and steps serves that purpose. On its own, each one seems small, but they add up.

So by the time you’re looking at that odd graph, declaring doesn’t just mean “something might be wrong”. It means signing up for all of it. In the back of your mind, you’re weighing the consequences. Will this lead to a lot of admin? What if I’m wrong, and I’ve pulled people in for nothing? And if I declare, am I the one who has to lead it? I’ve never done that before.

What it costs

If it turns out the odd graph you saw indicated a real issue, delaying calling an incident means losing time. While one person is digging, nobody else knows there’s anything to look at. The people who could help aren’t looking, support doesn’t know what to tell customers who start asking, and whoever eventually joins the response starts from behind.

On top of that, when declaring is expensive, people will find ways around it. Issues get fixed in a DM or a side channel, never become incidents, and never get learned from. Your incident numbers might look great, but you’re not learning from the things that actually went wrong.

Make declaring cheap again

To get back to a place where declaring feels light, you might need to take a step back and think about what you’re trying to achieve through the whole cycle of declaring, mitigating, fixing, reviewing and improving. It’s tempting to add a requirement here, maybe another useful field there. And oh, by the way, wouldn’t it be so helpful to also capture the estimated revenue impact, before anyone’s even confirmed something is broken?

But wait, that’s not actually the point. Incidents shouldn’t be an exercise in collecting as much detail as possible. You should be trying to find out what happened, what contributed to it happening, and what you can do to prevent it, or something similar, from happening again. Very little of that needs to happen in the first five minutes.

So keep it light:

• Make declaring one step. One command or one button, and one sentence describing what you see. Severity, impact and everything else can be filled in later. If declaring takes longer than describing the problem, it’s too much.

• Not every incident needs a postmortem. Decide on the appropriate follow-up once the immediate response is over. Declaring an incident shouldn’t automatically sign someone up for a full review.

• Separate declaring from leading. The person who notices something isn’t automatically the right person to run the response. Make it normal to declare and hand over.

Think back to the engineer looking at that dashboard. The only thing they should have to ask themselves is: might something be wrong, and could I use some help checking? If so, declare an incident.

So here’s a question for your own team: what does it cost the person who declares an incident? The answer will tell you where to start.