The article was published in June-2015 edition of Tea time With Testers. You can read the same along with other testing related articles by downloading the magazine here.

Given the web context, I decided to include the full text of the article on my website as well. This is the first article in my series – “Notes on Test Automation“.

1 Introduction

Test Automation Frameworks (TAFs) do not get the treatment in terms of design that they deserve. Spending time on understanding the precise requirements and then designing/choosing the framework becomes much more important when a general-purpose framework is required. Such a situation is unavoidable when your organization is looking at the possibility of a single underlying TAF that could be employed by multiple teams involved in strikingly different products; their testers are conversant with different programming languages, and they choose or need to employ different tools to support testing.

This article is focused on the technical aspects of designing a scalable, general-purpose, object-oriented TAF. This in turn means that the contents would mainly comprise concepts that directly translate into an implementation in an object-oriented language. One could choose to write the same thing in the traditional programming way, but there would be further work required on part of the reader to translate the design.

The article will not attempt to answer the questions of why to automate, when to automate, which tests to automate, automated tests versus manual tests, ROI of automation, etc. There are quite a few books available on the subject of test automation, and these topics are discussed at length in them. So, if the reader chooses to follow this article, it is assumed that s/he already understands the importance of test automation. This article series focuses on how to automate, whereby the “how part” is not about using an existing tool or framework, rather about designing one.

2 The Problem of Dumb Tests

In the design of TAFs, most of the important information that a test requires is kept outside the TAF, as many a times such frameworks are designed using traditional programming rather than employing object-oriented programming. A simple example is grouping tests into platform-based folders, after which the scheduling logic copies in only the tests meant for a test execution machine based on a specific platform. There are multiple problems with this approach. One is redundancy, multiple copies of tests across folders. If you think about it, we could have a common test platform and then specific platform folders, but my question is how many such “commons” would you have? Even if you have the patience, what happens when a test gets modified and is not meant to be run on Platform A? How do you know to which folder you should go to change it? So, there should be an external means to map tests to folders! By not making the information available within the test about which platforms it should (or should not) run on, you have the following situation:

• You need a complex folder structure, which is virtually impossible to maintain as the number of supported platforms grows
• You need an external means to map tests to the above folder structure.
• If the test writer is not the test framework administrator, it means the person who requires this is different from the one who implements it.
• You have a “dumb” test, which does not know on which platform it can run. So, if anything goes wrong, you have an incorrect test case that could crash the system or, at worst, lead to an incorrect result.

So far, we have only talked about the platforms. We have not talked about categorization of tests, priority values associated with tests, known bugs, API version checks and so on. Can you see the complex and impossible folder structures that these options would lead to if kept outside the test?

This article focuses on the concept that a test is meant to be the most complex part of the framework in terms of its power and flexibility. Test encapsulation is at the heart of building a general-purpose framework, which can be used by testing and development teams as a common test automation platform for white-box and black-box tests. To know about what exactly is meant by test encapsulation, its approach and benefits, keep reading!

3 Introduction to Test Automation Frameworks

As usual in the software testing world, even the term “test automation framework” is interpreted differently by different individuals and in different contexts. Here are some general points which arise when one discusses terms:

• Data-driven test automation approach: This separates test data from the test execution code. This is helpful when the same scenario is to be driven with different sets of data. This is also helpful in enabling changes being made to the test data for an existing scenario without modifying the test execution code.
• Keyword-driven test automation approach: This is a command-based model. Commands exist outside the test execution code in the form of plain text files with corresponding parameters. The execution logic reads from the files, interprets the commands and calls corresponding functions in the test automation framework. This is mostly used in combination with the data-driven approach.
• Generation and mutation based test automation approach: Instead of relying on hard-coded data provided, these frameworks rely on data generated at run-time. Another difference from traditional data-driven frameworks is that these contain commands for mutations directly within the input configuration files, thereby incorporating a flavour of key-word driven frameworks as well. These are more complex than both the above categories and their implementation is usually found in the world of “fuzzing” frameworks.
• Multiple test nodes on a single test runner machine: (Test node is the thread or process running tests on a test machine). This is mostly seen in performance testing frameworks, which use a single test runner machine and multiple test nodes connected via multi-threading (mostly) or multi-processing. This is made possible only for non-GUI testing and testing in which no system-wide changes are made by a test. In recent developments of headless browers as well as automation frameworks like WebDriver, it is possible for functional test automation for the web applications as well.
• Central execution control: The framework should be able to trigger, control and report the results of multiple tests from a single point of control. For example, CPPUnitLite, a unit testing framework for C++, can execute all tests via make command.
• Distributed test execution: Testing frameworks can also be thought of as providing the functionality of central execution control from one or more “controller” machines, which can trigger, control and report the results from multiple test node machines.
• Enhanced reporting via GUI: As an extension, frameworks can support GUIs (stand-alone executables or web-based) to support customized reporting. In performance testing frameworks, for example, features for plotting and analysis of performance statistics is essential. Also, different stakeholders would have different needs in terms of the test report they want to see with respect to details and contents.
• Hardware/operating system infrastructure: In some contexts, where hardware and operating system requirements are precise (or even otherwise), some refer to the complete set-up (i.e. test automation code + hardware and operating system set-up) as the test automation framework. This can be seen, for example, where the functioning of the TAF depends on external tools that are pre-installed on the test runner machines, which cannot be done as part of the test set-up.

Note: Some texts enumerate the first two bulleted points above as the “Types of test automation frameworks”. I beg to differ. I do not consider these as types of test automation frameworks, because that would suggest they cannot co-exist in a single framework. The first three are approaches towards a TAF design in terms of enabling the TAF to execute tests via data and commands present in clear text in various forms, such as plain text files, XML files, databases etc. All three can be built on top of a general-purpose framework. In simple words, a test framework can provide any and all of the features mentioned above, by choice.

Following is the list of other general features, on which decisions have to be made when choosing/building a test automation framework:

• Which language should be chosen to build the base frameworks? For which languages would extensions be supported?
• Should the test automation framework be generic or specific to a product under test, testing tool or a language, in which tests can be written?
• How much complexity do you want to hide from the end user (the one writing/executing/reporting a test)? Many times, hiding the complexity might result in providing less flexibility as well. Some performance testing tools, for example, provide for the recording of the test scenario as a GUI-based tree representation with a limited amount of controls for customization – ease of use with less flexibility. Others might choose to provide both.
• Should it be cross-platform? What browsers it will support? Which versions of the product can it test?
• How do you manage framework files (e.g. core library, configuration files) and the user contributed files (scripts)? Are you going to opt for version management? What would be the protocol? Should it exist along with the development repository for the product under test, or be separate?
• What kind of regression strategies would it provide? Does it provide for bug regression, priority based regression, author based regression, creation date based regression, API version based regression, etc.?
• Should it support physical and/or virtual test runner machines?

The above is an incomplete and a very high-level list of features that one might need to consider when building a test framework. So, it shouldn’t be a surprise if one sees testers dissatisfied with the testing tool/framework they currently employ, because there would be one or the other feature missing.

4 Did We Miss Discussing Something?

In the discussion of this complex and long list of requirements, we missed the most important thing. What’s the final purpose of a TAF? With all the “noise-features”, which help in scheduling, executing and reporting with an icing of user-friendliness, at the core of it, a TAF is about running one or more tests. Where in our discussion did we talk about what the test should look like? How often does this aspect get discussed when we talk about TAFs?

TAFs are like onions. Their tastiest part is their core covered with lots of wrappers/layers. For a TAF the tastiest (the most important) part is the code written by a tester which is going to execute the actual test on the system. In our TAF discussions, we are often so caught up in the wrappers and layers, that by the time we reach the heart of the problem, we have made it powerless and small, putting most of the things in the hands of the surrounding wrappers/layers.

If the “test” part of a TAF is not designed well, all other features will be useless. All user-defined configurations, default framework configurations, platform conditions, tool decisions, pass/fail decisions etc. have to be translated into how the test gets executed and how it should report the results back.

Because of this faulty approach, most of the information, which should enable the test to take decisions at runtime, is made available outside of the test, so that some outside component and not the test itself makes these decisions. As mentioned in the introduction, this can become highly problematic when you try to add more and more features to the framework.

5 Encapsulation

As per Grady Booch, encapsulation is – “The process of compartmentalizing the elements of an abstraction that constitute its structure and behavior; encapsulation serves to separate the contractual interface of an abstraction and its implementation.”

If that sounds complex, let’s settle for this:

When we encapsulate X, we provide inside the X:

• All data that defines the state of X as well as the data it requires for doing its job.
• All methods that X needs to use and manipulate the data available. Then we expose a subset of these methods as the external/contractual interface which the calling components would use to exercise its behavior.

6 Test Encapsulation

With test encapsulation you provide inside the test all the data it needs for test execution and all the methods related to setting-up, running, reporting and clean-up.

6.1 What does a test need to KNOW?

Let’s personify test and see what it asks:

6.1.1 Meta data

Every test is unique in that it would have at least one thing that’s different from the other tests in that framework. (If it is not, please check your review process!). It means, every test should be identifiable as a separate entity.

A test could ask the following questions in order to find out what it is:

• What is my name?
• What is my identification number? (a single test could have multiple IDs in different contexts)
• What is my purpose?
• Who is my author?
• What is my birth (creation) date? On which date, did I take my present shape? (Modification date)
• What is my version number?
• Who are my parents and grandparents? (parent group/sub-groups)
• What type of test am I?
  • At which level do I work? (unit/component/system)
  • What kind of software attribute do I test? (functional/performance/security)
  • When should I get executed? (BVT/acceptance/main)
  • Am I based on custom extensions? (e.g. CPPUnitLite based test)

6.1.2 Logging options

Logging of test results, progress of execution and dumping key information at precise locations is one of the key aspects of test automation.

The following are questions that a test might ask:

• Do I use the centralized logging mechanism or my own?
• Am I running in debug mode, thereby increasing what I log and changing where I log?
• Should I log performance statistics and local resource utilization statistics?

6.1.3 For runtime

Prior to test execution, a test must know some important things about itself to take decisions at runtime. Compare this to a game show, in which the person asking the questions would request that only those that satisfy a particular condition would be eligible for that round of the game. So, when he shouts “All those in black shirts”, the ones in black shirts would rush towards the podium. But before getting up, the person must know that he is wearing a black shirt!

On the same lines, at runtime, the test would come to know about configuration settings chosen and the test environment around itself. At that time, to make important runtime decisions, a test would ask:

• How important am I? What is my priority?
• What is my version? Yes! I will utilize this when version based regression is on.
• Did I uncover some bugs so far? If so, what are their IDs?
• For which API versions of the product can I and can’t I run?
• On which platforms can I or can’t I run?
• Do I need stubs to run? Can I run at all if stub mode is on?

(There could be a requirement that you want to use stubs instead of actual components, which is usually the case with unit and component testing. There could be tests, which are meant to be run only with stubs, others which can run with and without stubs and still others which would not run with stubs.)

6.1.4 From runtime configuration

There are various runtime configurations, which can be configured by the user or are set as defaults by the framework. As discussed in the previous section, these should get passed on by the framework to the test so that it can match them against its properties to take runtime decisions. One important example of such a decision is whether the test should get executed or not.

Let’s see what kind of questions the test might ask:

• What is the API version under test?
• What is the platform on which I am being asked to run?
• What are the regression options? Are you looking for priority based regression, bug regression, version regression, author based regression or any other form?
• I have been asked to always log performance data. Do you want to override this?
• Do you want me to run in debug mode?
• What is the base directory reference path from which I am getting executed?
• What is the current mode of execution? Are you using stubs?

6.1.5 Execution properties

Once the test takes the decision that it is meant to execute (i.e. the criteria received from the runtime framework configuration settings match its own filter properties), there are execution related questions, which the test would need to ask:

• Where is the build under test located?
• Where are the tools that I need?
• How many threads/processes should I launch?
• The previous test has just completed. Should I delay execution by going into sleep mode?
• What configuration should I use for the tools that I plan to use?
• Where are my input files?
• To whom should I hand over the test results? Can I talk to the database directly or is there a middleman?

6.1.6 During and post execution

There are properties of the test which are set during and after execution of the test. These make up the basis of the following questions:

• How much time did I take to execute?
• Where am I in terms of execution? What is the current step I am executing?
• Where should I maintain my state information if I am asking for a reboot now?
• Did I ask for a system reboot/hibernate at a previous step?
• How many assertions have I made? How many of them passed or failed?
• Did any of the assertions relate to errors in execution?
• What is my final say about the test I executed? Did it pass or fail?
• Should I send notification to the administrator that I potentially succeeded in finding a bug?
• Should I send notification that I couldn’t execute due to error in execution?

6.2 What does a test need to DO?

In section 5.1, we talked about all that a test needs to KNOW at runtime. Let’s now look at all it needs to DO at runtime:

6.2.1 Prepare

A test needs to prepare itself. This means that at runtime the test needs to load all properties that were set for it statically by the writer of the test. It should also get hold of the runtime configuration and test environment settings passed to it by the test framework.

6.2.2 Set-up

It needs to do the initial set-up for the test that needs to be executed. For example, if the test needs something to be installed so that it can run, the set-up step should, as a pre-cursor to the test, complete the installation.

6.2.3 Run

This step in the test would include the code that executes the actual test. This could comprise of multiple assertions (sub-tests).

6.2.4 Report

The test would report the results of the test as per the configuration, e.g. to a chosen file or database, or publish the results to an object, or return them as part of the function call.

6.2.5 Clean-up

The test would clean up anything it created specifically for itself, so that the system is returned to its previous state ready for the next test.

7 How to Achieve Test Encapsulation?

What we discussed in Section 6 might seem like a lot to be put in a test, might it not? Relax! The fact that a test needs to know and do a lot does not mean that the solution has to be complex. You can still hide most of the complexity from the end user, thereby making the test scripts simple, yet powerful. The following are some quick tips:

• If you take a careful note of what a test should know, you will observe that most of it is optional and would be used only for certain test cases.
• You can set the meta data and other properties in a container to which the tests are registered. This means that this would not appear for every test case written, but still can be overridden at the test level, if needed. For example, a test group can have the author name as the net, to which 30 test cases are subscribed and not even one of them needs to have the author property set by the script writer. Later, if another tester adds a test case, the author property can be overridden for the 31^st test case.
• From all that the test should do, you’ll find that, apart from prepare() and run(), all other commands are optional and can be set for the container if the setup() and tear-down() are common. In fact, grouping the tests like this into containers would make it easy to manage.

Similarly, the runtime checking for whether a test is “runnable” or not, is hidden from the end user, and is placed within the core library.

8 Benefits of Test Encapsulation

As a first benefit to reap once you have taken a decision to proceed with test encapsulation, you have already crossed the first hurdle by making the tests the most complex and powerful part of your framework.

In the introduction section, I mentioned the problem of running a test on multiple platforms and difficulties with the approach of folder wise grouping. Let’s try to solve that with test encapsulation.

• Imagine that you created a test that knows on which platforms it is meant to run. Let’s call it the MEANT-FOR-PLATFORMS property of the test, which is list or array of platform strings. For this example, let’s say it is: [“WINXP_X86”,”WIN7_AMD64”], wherein the values signify that this test is meant to be executed on 32-bit Windows XP and 64-bit Windows 7 platforms.
• All tests reside together in a single folder structure; no redundant copies are created for the above-mentioned platforms.
• In the test cycle, when the TAF is running tests on a given test runner machine, it already knows the platform of that machine. While it is looping over the tests, it would pass this information to the test, for example, by setting the test’s RUNNER_PLATFORM property.
• Now when the TAF asks the test to execute, the test can search for RUNNER_PLATFORM in its MEANT-FOR-PLATFORMS array, and only if it is found in the list, will the test agree to execute. Accordingly, the test would execute or the TAF would skip to the next test in queue.

In a similar way, you can add support for any runtime decisions. The approach is pretty much the same:

• Encapsulate the properties that form the basis of runtime decision in the test
• Make the base value available in advance
• Make the value against which comparison should be done at runtime available at runtime!

The second benefit that you get with test encapsulation is that your framework does not dictate to a test which product it is testing, which tools it should use with what parameters/configuration, how it should analyze the results or how it should report back etc. You can still provide these features which the test writer is free to use or ignore at will. This puts a lot of flexibility into the hands of the test writer, and you are free to enforce rules if the test writer chooses to use a given framework feature. An example is that a test writer might choose to write the report in a format and place of choice, but if he wants to use the web reporting platform provided by your framework (if it’s there), then he has to provide the results in a pre-defined format.

The third benefit is that whatever your test does can be kept independent of other tests. It can set up and clean up any changes to the system. Also, you can put exception handling in the caller covering all individual test methods being called. This will enable the framework to continue execution, even if there’s an error in a given test which has been called. These exceptions can be reported in the final test report generated by the framework and would help in providing accurate results as well as point testers to the areas which need investigation.

9 Execution Time Overhead of Test Encapsulation

As you have probably observed by now, to skip a test with test encapsulation, involves a runtime decision. This means any skipping of the test would have to be done at runtime. Before this happens, the TAF must have already called the test constructor.

Being from the performance testing background, I can relate to any concerns relating to overhead, but this is not as much as it sounds. It doesn’t take minutes or even seconds to create a test object. The time taken is a tiny fraction of a second. I designed a framework with this approach, and my requirement was to measure the time taken by the test to execute. For some tests, the values were not captured even with a precision of 6 decimal places when the unit was seconds.

Having said that, you can reduce this overhead still further:

• As discussed in section 6, you can group tests in containers. You could put the filter at the container level rather than at individual test level. For example, you can set the MEANT-FOR-PLATFORMS property for ABCTestGroup to which all tests of ABC category subscribe.
• When using this approach you have unlimited test filters. Look into the order in which filtering is done. Place the most commonly used filters first. For example, if the probability of an author based regression or a creation date based regression is much lower than bug regression in your context, filtering based on empty bug lists should come first.
• Test encapsulation has given you the capability to take runtime decisions, but it does not stop you from making careful exceptions. If you feel that most of your tests can be easily split up into platform-wise categories and there’s minimal overlap, you could still group them into folders. Then resort to scheduling only specific tests for specific platforms. If over a period of time you observe that the scope of overlapping tests is increasing considerably, all you have to do is set the optimum platform property for tests and start relying on runtime checks.

So, the overall benefit of test encapsulation is that once you have carefully built it into your framework and created the underlying skeletal support, you can still go ahead with your traditional approach to test automation.

Image Source: http://p1.pichost.me/i/28/1512141.jpg

One Response to “Test Encapsulation – Enabling Self-Aware Automated Test Cases”

1. Testing Bits – 7/5/15 – 7/11/15 | Testing Curator Blog

   July 12, 2015 at 5:28 pm

   […] Test Encapsulation – Enabling Self-Aware Automated Test Cases – Rahul Verma – https://rahulverma.net/test-encapsulation-enabling-self-aware-automated-test-cases/ […]

   Loading...

   Reply