Testing is not a bug search!

Many people think that software testing is a search for bugs. Sometimes I tell testers: “don’t try to find as many mistakes as possible, try to miss as few as possible!” And they don’t understand me: what's the difference?

And the difference is huge! In this article I want to tell you what it consists of and what tools you need to use for real useful testing.

What is error search?

I am testing a product. My task is to get as many bugs as possible. It is logical! Testers are always pleased to start bugs, this is a visible measurable result of work, and the more they are, the more I am appreciated as a tester.

What areas will I test in this case? First of all, the most unstable. Often they are unstable because they are less priority, but it doesn’t matter, the number of bugs is much more important.

What happens if I encounter a difficult to reproduce bug? The ROI for his research is considered in the head very quickly. Why should I bother with him, if during the same time I can get 3 less critical, but easy to establish?

What tests will I conduct in the first place? Koneno, the most non-standard. Enter in the login field “War and Peace”, divide by zero, insert a profile photo in .exe format into the profile.

I will tell you a secret - sometimes at interviews testers in response to a request to “test a calculator” list interesting and practical tests, but among the first thirty there is no test “check addition” and other basic operations.

This is exactly what error searching looks like - having nothing to do with testing.

What is testing?

I am testing a product. My task is to skip as few priority bugs as possible for the user. The fewer bugs missed, the less customer dissatisfaction expressed - the higher I evaluate the effectiveness of my work.

What areas will I test in this case? Naturally, I will start with the highest priority for the user. Even if they work stably and successfully, I will still check the main user scenarios so that in no case will I miss serious problems.

What happens if I run into difficulties? For example, with a difficult to reproduce defect, or a misunderstanding of the user's business process, or lack of requirements? If this is an important functionality, then I will find out "what is wrong", "how to." As a result, it can take a lot of time to establish a defect, and from the point of view of bug / time, the result of testing effectiveness will not be very high, but I will have more in-depth knowledge about the product, architecture, users.

What tests will I conduct in the first place? Of course, the most standard. Execution of the most basic scenario in the most basic conditions to make sure that the most important functionality works. And only after that I will pass to less standard scenarios.

Testing and Error Finding Results

In the case of the search for errors, in the short term, the results are higher: more bugs start up immediately.

But in the long run, things are not so rosy:
  • due to the lack of in-depth knowledge about the product, the% of missed defects gradually begins to grow
  • the development team is busy fixing scary-terrible-inconceivable bugs obtained by clicking on the same button 144 times under IE in the full moon
  • the release gets some terribly unpleasant and obvious to the user bugs
  • the number of errors found in the LONG TERM is falling

How to go from finding bugs to testing?

For testing to be effective and useful in the long run, you need to follow simple rules and use key testing tools:

1. Product analysis and test documentation

By clicking on the buttons, you can get a lot of bugs - but you can not say what has been checked. The only solution is to document the tests. Detailed test cases, depressing testers and consuming a lot of time, are rarely needed. But checklists with a list of “what needs to be checked” are necessary.

What do they give:
  • You analyze the product, write out the main features, actions, their parameters. Thus, the risk of forgetting something is significantly reduced.
  • Checklists are a great reminder "you need to delve deeper here." There is some slurred feature with insufficient description. How to test it? In testing without tests, the easiest way is to say “I will return to this later” and never return. And with the tests - you will have a test hanging in which it is unclear how and what to check, you will see such tests and do not forget the need to find out.
  • Checklists can and MUST be agreed upon. With developers, analysts. The whole team is included in the testing process, testers learn a lot about the product, the collective mind improves the quality of testing. And in addition to a one-time increase in the quality of a single checklist, the quality of testing in general increases: testers begin to take into account more in testing, develop, this knowledge pays off in the form of more effective testing over time.

The key to success in conducting tests is creating a map that you will follow. The goal is to cover the entire product. Just please, no excuses about the terrible resource consumption - I covered projects with millions of lines of code in less than a month and a half. And in the process of writing such tests, unexpected questions were raised and critical errors surfaced, which, despite the presence of unfortunate testers, have been hanging around in the product for years.

2. Testing Evaluation

In order not to be blind kittens, it is necessary to evaluate the effectiveness of testing. Analyze missed errors and reasons for missing them. Functionality and code coverage by tests. The level of user satisfaction through questionnaires and feedback collection. The quality of making mistakes by questioning developers.

There is ALWAYS something to improve, and the lack of a continuous improvement process is an inevitable swamp.

3. Discussion of testing objectives with the team

Many people think that testing has some mythical goals. And that they are always the same.

No matter how!

In each project, company, team, goals are their own. Do everyone understand them the same way? Have you spoken them out loud?

To bring maximum benefit, you need to understand well what this very benefit is. And do not be surprised if the opinion of the RMs and the developers does not correspond to yours. It is not necessary to convince them, but to adapt to the current design goals!

4. Understanding users and their business processes

For me, it’s a mystery how this is possible, but nevertheless it is a fact: often testers test the product without knowing anything about the user.
  • How is this product used?
  • Why is it needed at all, what problems does it solve?
  • What is the average qualification of users?
  • Under what conditions do users work? On what environments, equipment?

No need for guesswork or thought about the "average industry"! Testers should PERFECTLY know YOUR users. Often they do not provide this information to analysts. Think it over! Without knowing the user, it is impossible to test the product in the normal way.

5. Technical skills and understanding of architecture

To illustrate, I’ll give you a bug that was recently brought to me in a bug tracker:
Go to the website of the tested product http: //****.ru in the Firefox browser
Enter your username and password
Login from the same computer in the Opera browser
Asks to re-enter the login and password does not automatically login.

Such bugs are not only useless, they disgrace testers and discredit the industry as a whole! To start defects correctly, you need to understand the platform on which the product under test is written. If we are talking about web testing, then you can at least indicate in the bug report the error code returned by the server, see the details of the firebug, provide detailed information and save a lot of development time!


Many developers do not like testers. And they’re doing it right!

But good testers are loved and appreciated by everyone. But testers, not clickers and bagoders!

Learn to find out what is wrong and what other members of the development team don't like. Be sure to investigate the missed errors and do everything in order to no longer miss them. Do not chase after bugs — your mantra should be “user happiness”, “quality product” and “successful project”, and not “get as many bugs as possible” - VERY often these 2 goals are too far apart.

And may the force be with you!