How I Report Bugs
As a fervent exploratory tester, a single test session may produce upwards of 40 issues I’d like to investigate further or report in some way. Turning a 90 minute QA flâneurship into 40 stellar bug reports doesn’t scale so well. So how does one move from a bug dump to flawless compelling bug reports? Well that is a tough nut to crack. Mostly we just want to give our paired developer a rough list, and let them cherry pick the ones they don’t need further info on, as they tend to be living and breathing the project. Let them skim the crème of the crap, as it were. Then let QA file the rest, as god and project managers intended. The pairing bit, well, that doesn’t always get to happen because every day can’t be your birthday. Bug reports are QA’s greatest deliverable so, time to file whatever is left of what we shook out.
My aim is to write such an interesting and compelling bug report that no follow-up with the reporter is necessary, the developer is persuaded to want to fix it and perhaps a product owner is impelled to triage as we, the lowly QA bug report submitters, suggest.
The majority of bugs a tester will find can thankfully be observed directly in the UI of an application and that ends up being the predominant way in which to find them when black box testing. Lucky for us on OS X we have a bunch of cool tools to capture them where they lay. A bit of extra effort on our part can ease the managing of the case for the bug triager and provide a greater depth of clarity for the bug fixer.
Let me walk you through what that looks like on the ground. Typically I’m running an iOS app from the simulator or on device, looking for observable issues (bugs found via logging or other methods is a subject for another time). And then all of the sudden, I see you, Bug!
Having observed—and hopefully reproduced—the anomaly, it’s time to capture it. For static UI issues, it’s really easy to just snap a screenshot from the iOS Simulator with ⌘S or by using the Xcode Organizer. I have set up some Hazel rules to watch the folders where those files will land and move new images into Skitch automatically. Skitch then opens with my screenshot and I can notate and highlight the image to better point out the defect or its effects. If the defect is something that a picture can’t capture, I will record a video with Reflector and turn that into a gif with Gif Brewery. Adding videos to a bug, with our current tracker, means the developer has to download and open the whole file. If the bug shows itself for only a second or two after some actions, how long will the video be and will things be clear? Maybe I’ll have to edit and notate it to highlight the issue – well that’s just absurd. A few quick strokes in Skitch is one thing, video editing is quite another and ain’t no one got time for that. Just drag the file into GifBrewery, constrain the video to the relevant section and export to .gif for embedding in the case. The app has some controls to assist here but 9 out of 10 times no further editing is needed and a few seconds in and out of the app will get me most of what I need.
From there, it’s a quick hop over to my favorite Markdown editor, Mou. Create a new document and in it goes my per project TextExpander snippet that I’ve carefully preconfigured. Setting up a per project snippet at the beginning of the project saves an enormous amount of time. It also has the advantage of uniformity which will help the team hone in on the content they are most concerned with. Crafting these with your standard bug report fare in the Markdown format is trivial. I like to add a few secret sauce ingredients to save me time. One is adding a short shell script that will go to my
HEAD and run a
git log --pretty=format:"%H%x20%an%x20%ad%x20" -n 1 or perhaps a
git log -n 1 --decorate to notate which commit I built with. Another handy trick is to populate a dropdown menu with a pre-input device list of my specific test devices. I also employ TextExpander fill-ins to format my bug summary in the “If, Then, When” acceptance story format like so:
If %filltext:name=SetupCriteria% when %filltext:name=ActionCriteria% then %filltext:name=Result%.
When I’ve got the report all gussied up and ready to go, I create a new case in the bugtracker. Then it’s a paste of my nicely formatted report into the new case and drag my Skitch or GifBrewery file into the drop target. Fiddle with the title, set some fields, and voila, I’m done. The composition of the report in Mou also has the advantage of not risking the browser feeding your hard work to the ether if you accidentally misbehave.
Once you’ve got your set up dialed in, you can really crank the reports out. This happens to be so streamlined that often, it can get rather dull. I like to reward my reader—usually the developer whose favor I want to curry—for having read it. Sneaking funny things in whenever I can reasonably do so one way I do that. I do recall an autolayout anomaly where some text in an article was stretched beyond all recognition and to me looked rather full of Eldritch horror. In lieu of a bug summary, I put in Ph’nglui mglw’nafh Cthulhu R’lyeh wgah’nagl fhtagn which is the sort of thing you can get away with if you have a lovely moving .gif of the bug in question. On another report, I listed “if your eyes are not open, open them now” as the first item in the steps to reproduce; you’re damn right I want to know if the reader is paying attention. Putting in some Easter eggs will show you when they do. While that is beneficial to me, I think bringing some levity to the process helps builds a camaraderie that you just can’t get by showing up and doing a good enough job.