Making a great Bug
This post originally appeared January 25, 2013 on BetterProjects.net.
I’ve been meaning to post this for quite some time, but I’ll be honest, my desire to write has been a bit, well, lacking as of late. Which is really ironic as this post is all about how to not be lazy in writing up a good bug report.
To explain why this has been on my mind, you have to step back in time about 25 months to when I was promoted to manage a QA team in addition to BAs. At that point, the team was undersized and undertrained. I immediately hired 2 new analysts, but they both came from our call center. While they had a deep understanding of our company’s business processes and a good understanding of our technology stack, neither of them had ever worked in a full time testing role. To add to those two hires, I also brought on a few new experienced contract analysts in order to help keep a big project going.
It was this large spate of hiring that made me realize something sad… most people who claim to know how to test really suck at the job. This was a strange thing to me because while I’m generally regarded as someone who can break a system just by looking at it funny, I really hate to test. I was interviewing a bunch of people who claimed to love testing, but it was really obvious that most of them were really bad at it. Telling who would be a great tester was not easy, but there was one question that would immediately unveil who was a really bad tester, “Tell me what information you put in a quality defect report.”
The really amazing thing was how many potential analysts really could not answer this question. I interviewed people who had a couple years of experience up through people with over twenty years of experience, yet claimed to not really know what went in a good defect. I remember my first project, where I was brought in to do user acceptance testing and taught day 1 how to write up a proper defect report. I learned everything from the data I needed to capture, how to put it in the tracking system (sadly, a spreadsheet) and why it was so important to get everything right the first time. It wasn’t that I did get it right the first time (or the first half dozen, really), but within a week or so into the job, it was so boring that I could have logged a bug in my sleep.
Getting all the paperwork right isn’t really the most important thing in the job, but getting this one piece of paperwork right makes such an incredible difference. Less time spent by developers trying to recreate the problem, less churn as the defect gets sent back and forth for clarification and a faster time to get the fix in a build and retested. Not knowing how to log a bug, especially from someone expected to be a leader in the testing process, was just a non-starter, especially when I had two true newbies to train at the same time.
To explain why this has been on my mind, you have to step back in time about 25 months to when I was promoted to manage a QA team in addition to BAs. At that point, the team was undersized and undertrained. I immediately hired 2 new analysts, but they both came from our call center. While they had a deep understanding of our company’s business processes and a good understanding of our technology stack, neither of them had ever worked in a full time testing role. To add to those two hires, I also brought on a few new experienced contract analysts in order to help keep a big project going.
It was this large spate of hiring that made me realize something sad… most people who claim to know how to test really suck at the job. This was a strange thing to me because while I’m generally regarded as someone who can break a system just by looking at it funny, I really hate to test. I was interviewing a bunch of people who claimed to love testing, but it was really obvious that most of them were really bad at it. Telling who would be a great tester was not easy, but there was one question that would immediately unveil who was a really bad tester, “Tell me what information you put in a quality defect report.”
The really amazing thing was how many potential analysts really could not answer this question. I interviewed people who had a couple years of experience up through people with over twenty years of experience, yet claimed to not really know what went in a good defect. I remember my first project, where I was brought in to do user acceptance testing and taught day 1 how to write up a proper defect report. I learned everything from the data I needed to capture, how to put it in the tracking system (sadly, a spreadsheet) and why it was so important to get everything right the first time. It wasn’t that I did get it right the first time (or the first half dozen, really), but within a week or so into the job, it was so boring that I could have logged a bug in my sleep.
Getting all the paperwork right isn’t really the most important thing in the job, but getting this one piece of paperwork right makes such an incredible difference. Less time spent by developers trying to recreate the problem, less churn as the defect gets sent back and forth for clarification and a faster time to get the fix in a build and retested. Not knowing how to log a bug, especially from someone expected to be a leader in the testing process, was just a non-starter, especially when I had two true newbies to train at the same time.
Primary Ingredients
So what makes a great bug? Truthfully, the list is somewhat subjective, but lets get to a few items first off which there would be very little controversy in including.- Short Description / Title. This is the short, usually 15 words or less, way that the team will know the bug. Make it memorable. Make it describe what you’re trying to do. Don’t say “Something crashes” but “Application has an error #71 when clicking New in Contacts screen”.
- Who found it. This should be pretty self-explanitory, but its not always you. There are some occasions where you may be logging the defect on behalf of someone else.
- An ID. This is usually assigned by a bug tracking system (you should have one, but if not, there are free ones and cheap ones available with a quick Googling) but if not, you should assign it yourself and make sure that it is unique to your project, if not to the entire tracking system.
- Test Case Reference. What, you don’t have test cases? You have worse problems than I can fix in this post. If you don’t have a test case, what exactly are you testing? Sure, you could be performing exploratory testing without any specific test case, but explaining that is in itself defining your test case.
- Steps to Reproduce. If you don’t know how you did it, you probably shouldn’t be logging it.
- Area/Sub Area/Product/Sub Product. Lots of defect processes use a business process identifier and a screen name for these. There are lots of ways to do it. The reason this one is so important is because you want to know where in the system to go look for a defect. Note that there are often many ways to reach the same screen, or at least screens with the same name, in a single system, so you need to be careful with this one and make sure you’ve got the correct one.
Stuff That’s Good to Know
Depending upon your situation, there are a few things that, while not required in every situation, might be good to note. Some of these might be controversial, mostly because they just don’t apply to your situation.
- Environment / Data Configuration. Especially if you have multiple test environments, this is really critical. If you’ve ever wasted a lot of time failing to reproduce a defect, only to find out you’ve been using the wrong test system because someone didn’t specify, you already know why this one is so important.
- Severity. Lots of people have difficulty distinguishing this from its cousin, Priority. Severity, at least as I define it, is a technical understanding of how bad the defect is to the system. While a tester’s evaluation of the Severity could be incorrect, it isn’t bad information to include. This is often where you denote the difference between a blocker (something so bad you can’t test scenarios other than the one you were testing when you found the issue) and a minor irritant.
- Screenshots & video. I can’t say enough good things about being able capture in high fidelity exactly how you caused the problem to occur. This detail has been priceless to me in the past, especially in situations where your testers, developers and/or business teams are not collocated.
- Your thoughts. Testing is both Science and Art. Sometimes, especially with senior testers who are deeply knowledgeable about the system they are testing, its really good to know their thoughts about where they feel the problem to be. They’re not always right, but getting their logical understanding of the situation can be helpful.
- Related Bugs. If this bug looks like something you’ve seen in the past, make a reference to that bug.
- Database IDs / URLs. Do you have an exact URL in a web based application where the error occurred? Does the issue only happen with specific records in the database? Include these examples in the defect. Its not always easy to find these, but in a system that is well architected, you should be able to locate these without much trouble.
Stuff You Probably Shouldn’t Include/Decide
This post has been all about things you should include when logging a bug, but there are a few things you should steer away from when logging a bug. Its not that you can’t include these, but its usually better not to do so. Most often this isn’t because its objectively wrong to include these, just that its often political suicide to do so as defect managers and development managers often feel you are overstepping your bounds by doing so.
- Assigned to developer. Unless you’ve already had approval to assign it to a developer or the developer asked for it, you probably should let someone else handle this. Lots of development managers or defect managers take a dim view of anyone encroaching on their ‘territory’. Thankfully many are not that territorial, but even if they are not, there are often good reasons to not do this assignment yourself. Mostly this has to do with priority as assigning a low priority defect to a developer who is assigned to critical only defects is not the best use of their time.
- Priority. This one is best defined by your business partners or their appointed representative. This is their estimation of how bad the defect would be to business operations if it were to make it into production. Think of this as how valuable it is to fix for the business areas.
- Status. Most defect tracking systems have a default status and you should probably just stick with that, again especially if there is an assigned defect manager or product owner. Let them handle this one.
- Estimated time to fix. Don’t even touch it. Its not worth making people mad about a non-developer estimating developer tasks.
What am I missing?
As I said, this post has been in my mind to write for quite some time now and I just haven’t gotten around to it, so I am positive that you’ve just read my list and thought, “Why didn’t he include XXXXX?” Well, the easy answer is, I just missed it. The more nuanced answer is, I may not agree. :) Lets hash that out in the comments!