Software Testing and the hammer and nail approach

If all you have is a hammer, every problem looks like a nail.



It is sometimes a similar situation with software testing and
specifically with test automation. This may not be much of an issue for
single-product companies but when your organization produces a range of
products and has multiple teams of testers handling these different
products then the likelihood of hitting the hammer and nail problem
arises.

One might say that the problem is prevalent more in a centralized
testing structure than in a decentralized structure. The issue I am
alluding to is the mandate to use a common test tool, mostly for
automation since that is an area that everyone would like to see
standardized but poses most challenges to standardization.

I have come across many instances where there is the attempt to identify
a common tool that can satisfy all the requirements for automating
various products being tested by different test teams. In some cases,
the products would be as different as chalk and cheese. For example,
there was this situation where we had a suite of products addressing
different customer needs. This suite has a thick client application
written in Java, a Win 32 application, a complex Ajax based web
application, a few server products that you interact with using CLI and
APIs, some middle ware products and few mobile applications.

Trying to find a common tool to handle all of the varied requirements
for automating the different products could lead us to three possible
ways of solving them.

1. One, you talk to a few tool vendors who would naturally promise the
moon and claim that whatever tool they are selling can automate every
kind of application that ever existed and will come into existence in
the undetermined future. You could take this option like many folks do
and purchase expensive tools and licenses and then mandate your testing
teams to go use these (and only these) to automate in a standardized
manner. Simple solution using a brute-force approach.

2. The second approach may be to take the compromise route. Here, you
realize that a single tool may not be able to handle all the unique
requirements of automating each product and go ahead to procure a tool
that probably results in being the lowest common denominator solution.
The tool ends up being a good enough solution which essentially means
that it is neither good nor enough from a testing perspective. However,
the organization gets a standardized jack-of-all-trades kind of tool
that all groups could use with varying degrees of effectiveness.

3. A third approach could be to take the time to understand the specific
requirements and needs for automating each product, perform due
diligence in identifying & then evaluating the tools that best fit
the specific automation requirements before deciding on what tools to
procure. This may result in having to procure more than one tool. If
your products have similar automation requirements you may end up with
needing just one tool but in cases where you have products with
differing needs similar to what I stated as an example earlier, you
might realize that more than one tool is needed to perform effective
test automation.

Effectiveness
, is the key here. Ultimately you automate not for
the sake of automating but to support your testing campaign and deliver
value. Trying to force teams to use a specific tool that does not fully
support their needs is akin to the hammer and nail solution. When all
you have is a hammer, then every problem is dealt with as you would a
nail. Sometimes it is the right thing to do and in many cases it may not
be the optimal approach to follow. In testing, such an approach could
lead to teams bending their testing and automation practices around what
the tool is capable of while ignoring other possibly important areas of
the product which are harder to automate using the tool. The automation
tool should not dictate how & what you test. It should truly be a
tool (amongst other tools and techniques) in your arsenal enabling you
to tackle the challenges of testing your software.