I was recently a judge at ISTC, the Israeli Testing Cup finals. So how does one judge testing? What makes some testers better than others?

A few things to note about the competition, which differentiates the testing situation from regular work: The team is presented with a product to test that they know nothing about, apart from a 15 minute introduction by the developers.

Well, maybe not so different than regular testing.

Then the testers have a couple of hours to devise a plan, test and report. The judges check the logged bugs, reporting quality and accuracy.

Obviously, this is not an optimal situation, but that’s the challenge. The team needs to decide very quickly what is important and how to test, without knowing anything but the intro. Then they need to familiarize themseles with the app as quickly as possible, and judge what is proper and what is not. The time constraint and the new context means all tests are exploratory.

Through this process you can see how people are thinking, and you clearly see who works and produces better results.

In fact, you can find the same traits of the great testers in great developers, product people and humans in general.

Step 1: Identify patterns and outliers

As the testers start to test, they write down bugs. looking at those logged bugs, you see that the testers are trained in identifying invalid inputs, missing stuff, and eye-poking ugly controls.

Whenever we try something new, we do most of our learning by touching and feeling the product. We are much better in identifying the things that annoy us, because we recognize patterns, and the things that don’t fit ours, pop up immediately.

Step 2: Adapt

As time goes by, the way the testers define severity of the logged bugs changes. Things that looked like high severity issues at the beginning, now don’t look that aweful (compared to others, of course). The understanding of what is important and what is less, becomes more apparent.

As we learn, we start to understand the importance of the differnt features. We prioritize important tasks or features over those in lower priorities. The patterns we now recognize in the product make more sense to us, and we can make better decisions.

Step 3: Be creative and curious

Given the time limit, and depending on the quality of the app, when you go over the logged bugs you see a spread of differnt issues. However, you can see that some testers are more creative.  You can see bugs that were discovered in more creative ways than the superficial “go over the regular flows”. You can see they were asking and re-asking the question “What if I do this…“.

It takes time to understand the context of what we’re building and using.  Creativity and past experience give us an edge there, and some of us become “super users”, while others just “go with the flow”. Curiousity gets us away from the “happy path”, and helps us explore the new, the risky, and sometimes, the impossible.

Step 4: Communicate

And here comes the coup de grace. As they presented their report, some testers just summarized the bugs. Others gave more information and recommnedations, based on their opinions.

The better ones gave those in the context of the product vision, market and users. They not only understood the product better, but also reported the information in the context of the user – how the user would be affected, what they might feel and think.

Communication is the crucial thing here. Any collaboration requires communication. The more we use common language, more shared understanding, more empathy to the listener – communication is much more effective.

That means that someone else can pick up the ball and run with it quicker and in the right direction.

Not just great testers

What makes better testers, is what makes better workers – in any field. It’s the ability to identify patterns, separate the wheat from the chaff, be creative and curious and communicate effectively.  All these are qualities that make us effective in our work.