Manual Testing Does Not Scale

By 2 commentsMetrics, Quality, Testing

During the management review, we discussed the fact that although there’s extensive testing, critical bugs slip through. One of the PMs, who is a big proponent of exploratory testing, said we’re concentrating too much on writing test procedures, and less on testing.

I did actually produce the numbers, thanks to the newly implemented MS Project server. Although there’s no pattern, since it’s very new, during the last months, we see there’s some basis to his argument.

So what happens here? are we writing too much? Currently, I think we’ve bumped into a scaling issue. The systems are complex, and in order to cover many (but not even most) test cases, we have to invest a lot of time in writing the procedures. This doesn’t scale well, as the complexity grows exponentially. Even with enough time, there’s no way to cover all possibilities, and this is where bugs slip by.

Is it possible to do some planning ahead to catch those bugs? The scaling issue rears up its ugly head again: Even with enough time, how much stuff that can go wrong is it possible to foresee? Not a lot, so the procedures do not find the bugs.

We are currently doing only manual testing, that takes a lot of time on one hand, and is not repeated, due to lack of time. I believe, automation of tests can improve quality by freeing the time for the manual exploratory testing. While it does take an effort to place the automation, it’s worth it in terms of quality.

2 comments on “Manual Testing Does Not Scale”

  1. James Bach Reply

    I’d like to see you develop this idea more.

    What is it specifically that doesn’t scale about exploratory testing? What do you mean by “scale”?

    One could say that driving a car does not scale, because one driver can’t drive every car in the world. Except, of course, it does scale in some ways: more drivers, or drive bigger things like planes and trains.

    One way that exploratory testing scales is that higher skilled testers have the ability to focus their testing on the things that matter.

  2. Gil Reply

    Maybe to say it does not “scale” is not a great choice of words.

    In our case, the project is limited in time, and team has more developers than testers. Let’s assume that we know everything in advance in terms of requirements, so we don’t need to waste some of this period in re-writing test procedures, and let’s be optimistic that in the end, there were no bugs found that required no re-testing.

    The amount of complexity put into the system by the developers is gets higher exponentially with each additional developer, as the code integrates and impacts other developer’s code.

    However, the testers cannot cover each change put in by the developers. The addition of writing a tester is “linear” as it stands alone.

    The rate of additional testing will not exceed the rate of complexity put into the system.
    In our case, what happens is there’s little time for exploratory testing, since we fill up the allotted time with writing more test procedures and running them.

    If our target is to produce quality code, “scaling” this process by adding testers at some point will get us to some balance point, but will be too costly. (Obviously, if the customer was willing to pay, everything is fine).

    So, if we know this is not possible, the next best thing is to optimize the testing to get feature coverage and depth. It’s a balance between written and exploratory testing, and given resources, skills, and so on. I can see where optimization and scaling can be interpreted the same.

Leave A Reply

Your email address will not be published. Required fields are marked *