What's your test automation strategy?

I overheard
some test managers discussing problems with their test automation
effort, so I couldn't refrain from asking the redundant question, "What
is your test automation strategy?" They looked at me as if I had just
beamed down from another planet and said, "c'mon, you know our strategy
is to automate everything!"

It is
unfortunately true that some managers drink the proverbial kool-aid and
blindly regurgitate the 100% automation mantra or similar incantations
such as "no manual testing" popular among agile pundits like Lisa
Crispin.

Let me be clear. A goal of 100% automation is not a test strategy; it is a fantasy!
Similar to the Disney fairytales where fairy dust causes magical
transformations, evil is defeated, the prince marries the maiden, and
everyone lives happily ever after forever automating everything is not
practical or realistic.

Perhaps
the single biggest problem with most test automation efforts is lack of
a practical strategy. A practical test automation strategy is one that
provides a pragmatic solution to address specific business needs with
well-defined, measurable goals based upon realistic expectations.

Business
needs drive a lot of the change in any organization, and usually
involve cost saving measures, quality improvement, or increased customer
satisfaction. A business need for test automation includes reduced
testing time. (This doesn’t mean reduced ship cycles; it simply means
the time it takes to perform certain tests during the product life cycle
can be shortened.)
For example, the
Build Verification Test (BVT) is a necessary test suite to verify the
stability of each new build. Depending on the size and complexity of the
product a manual BVT suite can be very time consuming. An automated BVT
suite (which should be 100% automated including results validation
because it establishes a baseline measure on build stability and the
tests remain relatively static over the duration of the development life
cycle) can substantially reduce the time spent in this phase of testing
especially in iterative build environments where the team is getting
daily or even weekly builds. It doesn’t take long to realize the cost
savings over the product life cycle.

Test
automation strategies must also have realistic expectations. For
example, I have never been convinced that finding “new’ bugs is a
realistic expectation for test automation. (Yes, it will occasionally
find some new bugs, but let’s face it…the majority of the 5 -15% of the
bugs exposed by test automation in production environments are
regressions.) I have never seen data that suggests increased automation
reduces the overall development cycle. Nor will test automation
eliminate testers. (This is a false hope imagined only by prima donna
developers and bean counting managers scheming of ways to find value in
their Masters in Business Mismanagement degrees.) So, what are realistic
expectations for test automation? Well, I can reasonably expect test
automation to identify stress issues such as mean time to failure (MTTF)
and mean time between failures (MTBF). I can reasonably require test
automation to establish baseline measures such as BVT suites or
regression suites. Test automation is a pragmatic solution for load any
type of load testing or other forms of concurrency testing.

Finally,
a good test automation strategy must have measurable goals so we
clearly understand what success looks like (or identifies where we need
to improve). Without goals we are developing automated tests just to say
we are automating. Unfortunately, I occasionally see teams with goals
of automating n
% of existing tests. This really doesn’t make
much sense because it doesn’t take into account logical decisions of
what tests should be automated (remember, not all tests need to be or
should be automated), so some redundant tests or run-once type tests are
automated (which may not be the best use of your limited testing
resources). Also, the ‘existing’ set of tests is usually a moving
target, so that means the goal is a moving target, which means we can
never achieve the goal. Goals for test automation should be specific,
measurable, achievable, realistic, and timely (SMART). Set short term
and long range SMART goals for your test automation effort. For example,
a short term goals might be 100% automation of the BVT suite within 1
week after the first build drop. Long term goals might include design
elements and processes to transfer automation to sustained engineering
or maintenance teams, or 100% language neutral automation that will
execute on any localized (or pseudo localized) language version.

Test
automation is expensive. Testers have a lot of work to do in a very
limited timeframe, so it is important that we use our testing resources
effectively. A well defined automation strategy will establish clear
goals, set expectations, and provide practical, automated solutions.