To automate or not to automate?
All too often, I’ve witnessed development organisations believe that a good automation framework replaces all other types of testing, that you won’t see any issues in production if you just implement the framework. In reality, test automation provides a way to rapidly repeat a set of test steps, comparing results to a known baseline. Automation complements other test activities but doesn’t replace them.
Starting Out: When is a Test a Regression Test?
If you are starting to move to automation, start with your regression pack, which may be a collection of manual test cases that you run each time you do a release. If you start by automating just that, over time, you’ll save a lot of expended effort.
I’ve heard people ask: “What is regression testing and at what point does a new test case become a regression test?” There are lots of different views on this, but the rule I follow is: “Regression tests are a collection of scenarios that ensure, regardless of the functional change, that previous code implementations are not broken by new changes”.
For example, let’s say I write a test case which checks a button has the text “Hello world” and then, a few weeks later, someone adds a new button that says “Hello world 2” next to the original button. My first test is run as a regression test to make sure the second button didn’t break the first button.
In short, all tests become regression tests as soon as they are written! All tests guard against regression of the system against a known good baseline.
Finding the Right Tool for the Job
If you’ve already been looking into how best to approach test automation, you may have stumbled across a huge number of frameworks, all purporting to be the “world’s best automation tool”. In my opinion, there’s no such thing. It’s more that you need to find the most appropriate tool for the task at hand. You can pick the best libraries, frameworks or tools out there and it won’t suddenly mean you have a good test pack. In fact, if you’re just checking against a list of things, are you really testing?
There’s good news here – computers are really, really good at doing boring checking, whereas humans are great at testing. Let’s make the computer do the boring checks, and we can get on with the interesting testing!
To get a good set of tests, you need to be using good principles in your framework that make it:
Basically, your test code is still just code. Like writing any other code, test code should go through the same rigorous processes development code goes through. This also means test cases are code, should therefore live inside code repositories, and to update them you should follow branching strategies and pull requests to ensure they are protected in the same way dev code is.
Test code is software. You should use the same principles in designing and building it as you would any other software.
Do The BDDs!
At the beginning of your test automation journey, you will come across BDD and TDD, among many other acronyms, and have to decide which one to use. It’s a decision made harder by the fact there are numerous articles about BDD split evenly between “BDD saves the world” and “BDD is ruining test frameworks and doesn’t work”.
The distinction between TDD and BDD is:
- [T]est [D]riven [D]evelopment is where the developers write the unit tests before they write the development code and then, as they write the code, the tests slowly start passing.
- [B]ehaviour [D]riven [D]evelopment is an extension to this where you write the expected behaviour in Gherkin format as a team and then developers go and write code.
BDD and TDD are not mutually exclusive, meaning they can both be used within a project. For example, developers can use TDD to write their unit tests before they start writing the development code, whilst you still follow BDD as a team to ensure all the desired behaviour is met. My view is that BDD is good if you use it the way it was designed, which means that the actual test framework is only part of the process. BDD is a good way for organisations with more of a Waterfall model approach to testing (by which I mean development and test teams as stand-alone teams that pass work ‘over the fence’ to each other) to move to an approach that breaks down the walls between development and testing teams. One of the best ways BDD achieves this is by getting Developers, Testers, BAs, Product Owners, Designers, and anyone else involved, into a Scrum team, to sit in sessions called “Refinement” or “Three-Amigos”, break down pieces of work into Stories and define the Acceptance Criteria, commonly written in Gherkin format. In short, this lets you explain the expected behaviour of a piece of work in simple English so that anybody, regardless of how technical they are, can understand what you’re aiming to achieve. The Acceptance Criteria are then put into a BDD-style automation framework, which is broken down into actual code under the hood but still produces automation test reports that show each of those tests in simple English. This process helps teams design effective test cases and come up with good scenarios to be used as a starting ground for tests.
Priority/Severity – Yes or No?
Some of the common questions I see coming from manual testers moving to automation is “do you still define high priority/severity test cases and have certain ones that are more important than others in the pack?”
The short answer is: yes and no.
The reason it’s a yes and a no is that this amazing automation framework you are writing will become your gateway to saying, “Yes! That bit of code is good enough to make it to the next stage of the route to live”. Here, the entire pack is run in a CI/CD fashion, where every single time a developer pushes their code, the automated regression pack is run, ensuring that nothing is broken (or regressed). This includes testing of the code changes as well as the deployment of that code into environments (regression testing). That same test pack is then used when deploying into each subsequent environment but can normally be tailored (for example, by applying a tag to specific tests, identifying them as something like “Smoke Tests”). A lot of the time, this is a subset of the test pack that the team has defined as tests that cover high “priority/severity” or risky integrations to speed up the execution time of the test pack.
The justification for no is: if you’ve taken the time to write automation tests, then they were important enough to be written. Therefore, they are important enough to be run and run as often as possible. However, the caveat to this is you need to ensure you keep refining/refactoring and ensuring the tests inside your framework stay up to date. Tests need to be regularly reviewed to make sure they are still useful for the evolving nature of the system under build.
It’s a journey – so take the wins when you get them
You will encounter many more challenges relating to test automation. This article alone won’t get you from a manual test pack to comprehensive CI/CD pipelines allowing you to release code to production daily. When starting your automation journey, you can’t expect that you will get there in a few days, weeks or months. So take the wins and recognise the benefits when you achieve them. If an automation pack that used to take you weeks to run manually now runs overnight, that’s still a huge time saving and a great step forward.