How Many Bugs is Too Many?
In some companies an Engineering Director may also be responsible for related technical departments - so let's talk a little about Support.
Just as with the inherent (and necessary) conflict between product owners and project managers - the latter wants every new feature now, and the former has resource constraints - there is a conflict between Engineering and the customer-facing support team.
The support team wants to keep every customer happy, yet real-world constraints mean that only so many bugs can get fixed.
Hopefully the product owners will be in-between Support and Engineering, applying their business knowledge to prioritize which bugs get fixed first. Support has an opinion of course, but that's just one factor in the equation.
Product owners also take into account the size, value and relationship of the customer(s) involved, how often it occurs, the LOE in fixing the bug, etc. They turn a dollar ROI into a priority.
After that, Engineering just pulls from the top of the prioritized list.
Makes sense, right? The most important bugs get fixed first, so we maximize the available "client happiness".
All is good..
..But then a client calls the CEO
Their case was rare or very hard to fix, or they were a small client, but for whatever reason their case is at #400 in the queue, and has been there for 2 months now.
It's never going to be fixed, because it's never more important than the cases above it.
But wait! There's an answer.. the CEO escalation increases the priority immediately, putting it at top of the queue, and the best and brightest fix it that day and it gets deployed overnight.
Of course, this trick can be attempted by every support guy by making their case high or urgent, which is why it's critical to have a triage queue that product owners review and prioritize.
I haven't said this yet, but Engineers should never be allowed to prioritize or pick and choose cases. They usually don't have the business acumen or background information to make those decisions.
But the queue keeps growing!
In a small company startup, the support queue is small, and engineers may even keep up at the beginning. No bugs in the queue == happiness and flowery meadows.
There, everyone gets used to having a goal of an empty queue. Priority is less important, because "everything should be fixed".
In contrast, an older company may have 15,000 bugs in the system, starting 10 years ago.
There, everyone gets used to fixing the most important bugs, and ignores the size of the queue.
The trouble comes when you're transitioning between the two types of company. When everyone expects the queue to be emptied, but instead it's growing steadily.
Sales are booming, customer numbers are growing, your app is getting bigger and more complex... bug reports accelerate.
Let's do a Bug Bash!
One strategy I've seen (in young companies) is once a quarter all the engineers attack the bug queue for a week. Nothing gets re-prioritized, but we simply apply more power until we get to the bottom. I’d argue diminishing returns here, but as I said, leadership can give us that instruction if they want. It’s their money.
My argument is that you're deliberately doing less important tasks.
A bug that has sat for two months without an update may never get fixed - and that's ok!
If age of ticket is truly a factor in your industry (it isn't), you could use it in prioritization. As a case ages, it naturally gets bumped to medium, then high. That may even work initially - but be careful what you wish for.
You’re still being tempted to think you can “finish” because the queue is small. Imagine 1,000 cases next year. Or 2,000. No bug bash will fix that. And with 1,000 cases using age in prioritization won’t work either, because the old cases will quickly overrun all the new important cases.
We’d always be fixing the wrong thing!
Ultimately, a change of mindset is required, and not much else. Your system will work just fine, once you:
- Prioritize effectively, using dollars and cents
- Accept that lower priority bugs won't ever get fixed
- Train your support staff to be realistic with clients
Incidentally, I wonder how closely #3 is related to how often you deploy - if you boast to me about deploying your SAAS updates daily then I expect the bug I just found to be fixed, and quickly.
Conversely, if I find a bug in Word, I don't even bother to report it..