Kontur Mobile Test Session: 446 bugs in 5 hours

In December, Kontur hosted the annual city test session in Yekaterinburg. This time, 38 testers for 5 hours searched for bugs in the new mobile application. Igor Borisikhin, a testing specialist and organizer of the event, shared his experience, said that the Contour came up with a new test session and how to get to the event this year.




What is a test session


A test session is a competition for beginners and advanced testers and people who are not indifferent to testing, an analog of a master class or workshop for developers. At the test session, you can meet colleagues from other companies, check for “strength” a new product for yourself, find out who is better at finding bugs. Testing sessions are a traditional event for Yekaterinburg. What was the last fourth citywide test session you can find in the community of testers in Yekaterinburg - UTC .


About format


The classic test session format involves testing a web service. Member has product analytics. There are N hours to find bugs. At the end, the jury considers who found how many bugs. The best are prizes. Contour added to the classic test session format:


Mobile app.
The product for testing was the Kontur mobile application. Conference for conducting external and internal conferences. Read more about the Contour app. You can read the conference in the markets: Google Play , App Store . Here's how it works:



MindMap instead of analytics.
We did not load participants with tons of boring analytics for the application; instead, each team had a cap with the features of its work and a free field for research. For mapping used the program XMind .


Training.
According to the format of the test session, in the process of searching for bugs, participants studied new methods and techniques for testing mobile applications. More on points 2 and 3 a little further.


About the participants


38 testers from different companies of Yekaterinburg gathered for the test session: iRidium mobile, Ridero, Motorsport.com, Ural Airlines, Point, Advanta, Mercata, Extreme Pro, Flag Studio, BD Cube, ITM Holding, Digital Spectr, SkyDNS, Naumen and Kontur .




Experience Based Testing


Instead of loading the testers with analytics, we proposed another way to get acquainted with the application - testing based on experience.


Testing based on experience - heuristic exploring in an open multidimensional space. If it’s simpler, research the application and draw conclusions whether it works correctly or not. Each participant in the test session has already been to conferences, so the subject area of ​​the application is familiar to him.


Experience-based testing is based on three methods:


  1. Guessing mistakes - suppose a mistake and check that it is not in the application.
  2. Testing based on a checklist - come up with top-level checks for the application, and generate and test specific test cases during testing.
  3. Research testing is a comparison of the expected and received results of the application’s behavior based on their own experience.

But not all participants in the test session had experience testing mobile applications.
In order to suggest new directions for research, to share experience and specific cases with experienced testers, during the test session we told participants about heuristics of mobile testing.


Mobile testing heuristics are a collection of research methods that contribute to the discovery of previously unknown.




About pair testing


At the session, participants tested in pairs. 19 teams were composed. We combined teams and balanced them according to the following criteria:


Experience in testing.
Beginners are more comfortable with beginners. Experienced with experienced. If a beginner works in tandem with a very experienced and experienced tester, in most cases, the experienced one dominates and does not allow the ideas of a novice colleague to be revealed. If the experience of participants is more than 4 years, for example, 5 and 10 years, then you can safely combine them into 1 team.


Experience in testing. мобильных приложений.
The test session is an open event for testers with any specialization, so it’s good if testers involved in more than mobile testing get on the same team.


Devices.
People came to the test session with their gadgets. We tried to ensure that the devices in the team were both iOS and Android.


A variety of communication.
We are for communication! Colleagues from the same company cannot be on the same team.


Pair testing helps to focus on the task, helps the tester to keep moving while the other takes a break. The couple encourages each tester to explain and implement ideas. When the tester explains his thoughts to another, the process of formulating itself gives rise to new ideas and cases. Pair testing is a great way to upgrade communication skills and learn how to effectively interact with a colleague. For some, this was the first test in pairs, for some it was not. We hope that participants will want to apply this method in their work.


About the bugtracker


The bug tracking system was traditional for the Youtrack Contour. Many testers worked with Youtrack for the first time, so we prepared a short video tutorial on how to work with the system and how to get bugs.




About the bug rating system


According to the plan, 5 teams won with the most points. We give an example of a rating system.


Two entities could be created in the bug tracker.
Task - suggestions for improving the system. Tusky was rated at 1 point.
Bug - defect in the product. Bugs had a different priority: crash, major, minor.


Bugs with crash priority included endless loading, freezing, data loss (entered or edited data are not saved), blocking of the device’s main functions. Such bugs were rated at 20 points.


By major treated outside buttons or links are not editable fields, the unexpected result of, moved down layout (interfering operation), data security violation. Such bugs were rated at 10 points.


For minor include misspellings, moved down layout (do not interfere with work), incorrect animations, not informative tips, unstable it is reproduced the problem. Such bugs were rated at 5 points.




About slang


Participants received slang features for competent bug reports in mobile applications. A small example:


Tap - a brief touch on the touch screen, followed by removal.
Double-tap - two short touches with your finger, akin to a double-click.
A touch is a touch longer than a tap.
Touch-and-hold - touch and hold. The touch is longer than the touch.
Svayp (Slide) - prolonged sliding your finger across the screen.
Toast - a pop-up message on the surface of the application window.
Togle is a status switch.
Title - the title of the screen name.
State - state, orientation of the device (portrait or landscape).


Now you also remembered how tap differs from swipe, toggle from toast, and title from state.


About prizes


Based on the results of the test session, we selected 5 winning teams that scored the most points. The circuit often holds competitions in sports programming, so we chose a proven scheme and handing prizes like on ACM ICPC. On the common table with prizes, the team with the most points, the first selects prizes, the second team scores the points - the second, etc.




Who is the best to test


While the jury was summing up the results, the participants could chat with colleagues, have a pizza snack, take a tour of the office or spend time in the game zone: play a telephone dart, 100 to 1 for testers, consoles, kickers.




In total, participants made 446 reports, of which 349 were of type bug and 97 were task.




The jury accepted 278 and rejected 140 reports. Rejected incomprehensibly described, irreproducible or repeated within the same team bugs.




Of the accepted reports, 215 were bugs. Of these, 118 bugs have priority minor, 80 - major and 17 - crash.




I emphasize that these are statistics for all teams. In 215 reports there are many repeating between teams or device-dependent bugs. Therefore, the number of unique tasks that migrated to the bugtracker of the team developing the application was 23.


All photos from the test session can be viewed here .


TestHackathonChallenge


We tried a new topic for ourselves. And they did a test session on mobile applications. The guys liked it, so we will continue in 2018, but we will add novelties. The next test session will be held at the end of the year, the announcement will appear on our blog - do not miss!