Rahul Verma: My Petty Bits

“What are the types of testing?”

A popular interview question. A sure shot way for Linkedin post impressions. And like most things which fall into either of these categories, this is a cheap and useless question in itself if it looks for a simple list of named types.

I would not have chosen to write on this topic if the damage is confined to cheapness and uselessness. The damage is much bigger.

Here’s my take on the this subject. I am writing this article while sitting in a hotel room with absolute horrendous internet connectivity, so pardon my tone :-).

Types of Testing

Humans love to categorise. Testers are super-humans. So of course, we love to over-categorise.

Tip: If it’s not clear by now, this is NOT a post on types of testing. So, please read it accordingly.

Based on complete ignorance of anything beyond functionality:

• Functional Testing
• Non-functional Testing

Non-functional testing does not mean that it is not functioning any more :-). It is supposed to mean that the test object is supposed not just to function, it should function well. This “well” part is the target of non-functional testing. Some testers tried to solve this problem by creating some other terms to solve this – “Para-functional testing”, “Beyond Functional testing” and so on. You can try to create some term as well. Because that’s what we do – create new terms – that’s what we as testers have been inventing for the last 2 decades or so.

Based on Quality Dimension:

• Functionality Testing
• Performance Testing
• Security Testing
• Usability Testing
• Accessibility Testing
• and some other 100 and growing types.

Tip on how to create a new Term: Find a suitable dictionary word, add “-ility testing” and voila! that’s another one. E.g. “Describe” + “-ility testing” = “Describability Testing”. Now force it to mean something: A type of testing which is done to evaluate whether the test object is able to describe its output and behaviour.

No, we don’t stop there. Within a dimension, there are other dimensions too. For example, within performance testing:

• Load Testing
• Stress Testing
• Robustness Testing
• Soak Testing
• Endurance Testing
• Volume Testing
• Sizing
• Fail-Over Testing
• and so on

Based on how the test object is exercised:

• Static Testing – Testing by looking at a test object.
• Dynamic Testing – Testing by executing a test object

Testing by looking at a test object after executing it is not static testing. This difference is more important to know about rather than the skill to do what is called static testing.

Based on Exploration and Level of Preparation:

• Ad-Hoc Testing: Means typically “some testing”, “random testing”, rather “for-a-defined-purpose-testing” as that will make it sound like exploratory testing based on charters.
• Exploratory testing: Means anything apart from what James Bach possibly envisioned and has been fighting for.

In practice, if you are NOT writing it down, or you are using a mind map instead of a spreadsheet and if you can blindly talk against automation or formal test design – you can safely mark your testing as exploratory testing, without ever getting caught.

• Scripted Testing (a term coined by Exploratory testers to tell what is not exploratory testing and later to mostly tell that it is not even testing.)
• Checking – Is that a type of testing, if checking is not testing? Not a debate I would like to get into.
• Testing !!! – Because without exploration it’s not testing, just checking. (I like this one, but don’t like the reasoning.)

Based on how the test object exercised and whether a tool/automation is involved:

• Review
• Static Analysis
• Dynamic Analysis
• Manual Testing
• Automated Testing
• Scanning
• Audit
• Assessment

Based on depth and coverage:

• Smoke Testing
• Sanity Testing
• Package Testing
• And many custom terms there

Based on how the test idea is derived:

• Black box testing
• White box testing – although white is as opaque as black.

Some testers tried out the term – Glass box testing instead of white box testing. The developers responded – “Jinke ghar khud sheeshe ke hon wo doosron ke gharon par pathar nahi phenkte.”. So, the term never caught up.

• Gray box testing (come on now, already)
• Light gray testing – It’s same as gray box testing but prefers a lighter T-shirt.

Based on level of test object:

• Unit Testing – although the term unit is itself vague. There is a scope of 10 other types of testing waiting to be coined as a term: statement testing, for-loop testing, while-loop testing, function testing, class testing, module testing … – anyone, take the lead! Although unit testing is also white box testing, it is a separate type because guess what, we love separation.
• Component Testing – It’s like the bat. Hasn’t found its place in the animal kingdom. Sometimes same as unit testing. Sometime a different vague term.
• Integration testing – No, it’s not integration of two small functions which is still unit testing. It’s again driven by the mood of the user.
• System testing
• System of Systems testing
• System of system of system of systems testing – Anyone? That will be a great term too if you care.

Based on who does it

• Developer testing – Many organisations also call any testing done by developers as unit testing.
• Full Stack Testing – Testing done by the magical full stack tester, whatever that means.
• User Acceptance Testing – A type of testing done by everyone else except the user.

Based on relationship to change

• Re-testing/confirmation testing – typically for a fix
• (Just) Testing? – when the change is not related to a fix. Vague, anyone?
• Regression testing – based on impact of a change. The most confusing term of all. And guess what, when it is confusing, we over-simplify. It deserves an article of its own and I’m going to write it soon.

Based on Human Evolution (joking of course), based on randomness and God knows what:

• Monkey Testing
• Gorilla Testing

Based on Mutations, Transformations, Faults etc

• Fuzzing – Blind Fuzzing, Protocol Aware Fuzzing
• Data Corruption Testing
• Fault Injection Testing

Based on Release Type and Management

• Alpha Testing
• Beta Testing
• A/B Testing

Based on character sets and culture:

• Localisation Testing
• Internationlisation Testing

Based on -based or -driven :

• Risk based testing: So, is it okay to not to consider risk and still call it testing?
• Context-Driven testing: So, is it okay to not to consider context and still call it testing?

The Damage

One of the key damage this style of categorisation is that it makes the discussions cheaper. This is just the case of types of testing. We as testers are doing it to the subject at all levels. How many types of reviews? Which roles? What do they do?

We as testers have toned down the knowledge of our profession to just being able to talk about it with fancy vocabulary.

Look at the interviews. Look at the certifications.

We don’t just stop there. Even if we are talking of tech knowledge, the kind of knowledge which we end up talking about is:

• What’s the name of the command which does X in this tool?
• What are the options that you can provide?

This also results in following kind of content:

• Learn testing in 3 days
• Learn design of test automation framework in 6 hours

The damage is that we are very concerned about what should we call X. Who should do X. What should we call this Who. When we should do X. What do we call that layer of When.

We don’t want to discuss how to do X.

Many “types” of testing have become “Types of testers”.

Some Food for Thought

Consider the following example and answer what type(s) of testing does this situation represent:

While looking through the code of a module with an interface internally as well as externally used, referring to and building on written-down test ideas, I look for areas which can result in performance bottlenecks, for which I use regular expression based search features in the editor. During the activity, I also execute small parts of the code and I observe resource utilization along with latencies. Who Am I? I am a developer who is going to accept this module developed by another company to be used in my own software. Once I am done with this activity, I will write a test which will exercise this module’s public interface layer. I am doing this activity because this area is related to another feature where a functional change has gone in.

Now think again. Were this activity and needed skills more important and worth spending time on, or thinking about which all types of testing this situation represents?

I am often told that types of testing help us understand what we are doing. No, with due respect, they don’t. This over-categorisation is stopping us many a times from doing good testing because the testing which is needed does not fit a particular container created by types of testing.

That’s all for now.