In the first article – Test Encapsulation – published in the Q3/2010 edition of Testing Experience, I explained how the design of a test automation framework (from now on referred to as TAF) should start by thinking of tests first and then the surrounding features. In this article I’m going to address another challenging area in the design of TAFs – Distributed Testing.
Distributed testing is about enabling the TAF to distribute and run the same or different tests across multiple test machines at the same or different time(s), monitor their status and consolidate test results from all test machines.
In distributed testing, the test framework is logically split into two types of machines:
- The Test Controller (from now on referred to as Controller) – The machine which distributes the tests, monitors their overall status and is responsible for results consolidation.
- The Test Runner Machines (from now on referred to as Runners) – The machines which run the tests.
In certain situations, a machine could very well be the Controller as well as a Runner. Accordingly, it can have logic corresponding to either one or both functional areas.
Some typical uses of distributed testing are:
- OS Platform Coverage – Running tests on various platforms
- Hardware Coverage – Running tests on machines with different hardware or resources
- Performance Testing – Generating simultaneous requests to a target application because a single machine would not be able to generate the target load
- Security Testing – Separating the Runner from the Controller for the basic reason that such tests can result in operating system crashes
- Reducing Test Cycle Time – Even though the tests can be run on a single machine, multiple similar machines are used to allow parallel execution
- Common Test Framework – Running various types of test on different machines because they need different configurations.
Through the course of the next sections, in this article, we will try to understand the challenges faced in the distributed testing mechanism and various ways of developing an automated solution for the same.
The article is available as a PDF file here (~ 0.5 MB): Notes on Test Automation – Distributed Testing Models
You can also download the complete magazine from the Testing Experience website after a small registration process.
Rahul Verma
One of the biggest beneficiaries of “Distributed testing” that I find missing in your list of benefits is the Globalization Testing. I think the ability of being able to do the tests in an automated fashion together for all the locales your product supports, “would be” a fantastic achievement in itself. I use future tense (“would be”) here because in my experience, I have seen Software (test) Organization lose out on some serious benefits from Test automation when they fail to encorporate what it takes to make the scripts Internationalized i.e. the failure to include the Internationalization specific details into the base Test Automation framework. The main reasons for such a state, in my opinion, are far from being just “technical” understanding of what it takes to make the Automated scripts Internationalized. There are several “political” factors that play a part such as “if there is Globalization testing is being done by say European office, then why should I worry about it sitting in say Asia”. But whatever the reasons be, when we talk of ROI, if the tests are fully internationalized then the true return is generally “time saved by running scripts in one language * no. of language supported”. Somehow, this basic fact escapes everyone’s attention.
Having said this much, I think your article is relevant in the current context. I have played around with some of the Approaches you have mentioned to figure out what works best for Globalization testing (Specifically cross language scenarios when Server is in different language than client and the whole test matrix becomes very complex depending upon the no. of languages supported). What we end up using is really context based.
Regards,
Anuj Magazine
http://anujmagazine.blogspot.com
@Anuj,
Thanks for bringing in the point of globalization testing. I’d make sure that when I am writing/presenting on this next time, I do add this point.
As mentioned earlier, the hybrid model would be the ideal one so that at the time of deployment or later, any kind of models and a mix of them – pull/push/peer – can be chosen. This sort of framework can be deployed as per the context as pointed out by you. Its development, though, would be tricky and I’m still experimenting with ways to achieve it 🙂
Yes, hybrid model may be more applicable but even before that the Test Automation Framework should have encorporated Internationalization considerations. These days with more awareness Globalization is not considered as an After-thought in more mature organizations. In reference to Automation, what i have seen in my experience is that lot of things including which tool is to be used, changes when the thought of Globalization is brought in. I had written a little piece on reusing English language Test automation while doing Globalization Test Automation.
http://anujmagazine.blogspot.com/2009/09/uncovering-myths-about-globalization.html
Though, reusing is not an ideal scenario as it causes lot of rework. I do see a lot of Technical debt that Test automation focus groups pay just because Globalization is considered as an After-thought.
Thanks for sharing the link.
I’ll go through the mentioned link and get back if I’ve any questions.
Regards,
Rahul