- The Test Controller (from now on referred to as Controller) – The machine which distributes the tests, monitors their overall status and is responsible for results consolidation.
- The Test Runner Machines (from now on referred to as Runners) – The machines which run the tests.
- OS Platform Coverage – Running tests on various platforms
- Hardware Coverage – Running tests on machines with different hardware or resources
- Performance Testing – Generating simultaneous requests to a target application because a single machine would not be able to generate the target load
- Security Testing – Separating the Runner from the Controller for the basic reason that such tests can result in operating system crashes
- Reducing Test Cycle Time – Even though the tests can be run on a single machine, multiple similar machines are used to allow parallel execution
- Common Test Framework – Running various types of test on different machines because they need different configurations.
The article is available as a PDF file here (~ 0.5 MB): Notes on Test Automation – Distributed Testing Models
You can also download the complete magazine from the Testing Experience website after a small registration process.
One of the biggest beneficiaries of “Distributed testing” that I find missing in your list of benefits is the Globalization Testing. I think the ability of being able to do the tests in an automated fashion together for all the locales your product supports, “would be” a fantastic achievement in itself. I use future tense (“would be”) here because in my experience, I have seen Software (test) Organization lose out on some serious benefits from Test automation when they fail to encorporate what it takes to make the scripts Internationalized i.e. the failure to include the Internationalization specific details into the base Test Automation framework. The main reasons for such a state, in my opinion, are far from being just “technical” understanding of what it takes to make the Automated scripts Internationalized. There are several “political” factors that play a part such as “if there is Globalization testing is being done by say European office, then why should I worry about it sitting in say Asia”. But whatever the reasons be, when we talk of ROI, if the tests are fully internationalized then the true return is generally “time saved by running scripts in one language * no. of language supported”. Somehow, this basic fact escapes everyone’s attention.
Having said this much, I think your article is relevant in the current context. I have played around with some of the Approaches you have mentioned to figure out what works best for Globalization testing (Specifically cross language scenarios when Server is in different language than client and the whole test matrix becomes very complex depending upon the no. of languages supported). What we end up using is really context based.
Thanks for bringing in the point of globalization testing. I’d make sure that when I am writing/presenting on this next time, I do add this point.
As mentioned earlier, the hybrid model would be the ideal one so that at the time of deployment or later, any kind of models and a mix of them – pull/push/peer – can be chosen. This sort of framework can be deployed as per the context as pointed out by you. Its development, though, would be tricky and I’m still experimenting with ways to achieve it 🙂