Provocative question

When you try to shake the very foundation of how things have worked for decades, you’re in for imminent repercussions. Changes are hard, and changes to allegedly essential parts are hard, with an asterisk. And yet, this question finds its way to the air from time to time. For some, this topic won’t be groundbreaking at all — those are lucky people working in flexible environments. For the rest of us, let me shoot this question once again, plain and loud:

Do we still need to manually create test documentation and keep it up to date? Some brave tester

For me, the answer is an obvious “no,” but… Let’s start by defining terms. By “test documentation” here, I mean a number of test cases, usually organized in meaningful test suites or other forms of test repository structure. Of course, test documentation is not only that, but in most cases, it is exactly that.

The ISTQB tradition tends to distinguish between low-level and high-level test cases. Suffice it to say that high-level test cases are somewhat close to what’s called “test checklists” in less academic environments, that low-level test cases are, well, test cases, and that I include both of these types in my question.

Arguments for

Now that we have clarified what we are challenging, let’s go through the strong arguments and reasons in favor of maintaining test documentation.

Only source of truth

Test documentation is the only synchronized source of truth. It happens, and happens often, that the SRS (Software Requirements Specification) is not up to date or even exhaustive. Or even present, he-he. Client requirements cover only basic stuff. After clarification sessions take place, information is stored in fragments in tickets, threads, or private chats. Test documentation thus remains the only real place where effort is paid to keep knowledge about the application complete and updated.

Thinking while writing

Writing test documentation is a process that facilitates a deeper dive into features. When your task is to cover the application with test cases to ensure that basic flows are bug-free, you will start to envision these flows as well as potential ways to break something. You start investigating, remembering things; connecting nodes in your brain brings more and more context into something that started as “test this simple form”.

Articulating efforts

Test documentation and its derivatives are the only understandable format for describing testing efforts. When the client asks what was done to make sure the application is shipped with sufficient quality, you either ask them to have faith in you or show protocols for making sure the application meets the requirements. This is especially true when a serious bug is found in production, as it creates fertile soil for blame games.

Bureaucratic obligations

Test documentation can be required as part of iteration sign-off. Test cases and suites might be required not because of their precious content but simply as checkboxes in the list.

Traceability network

Test cases can be a core part of different metrics and procedures in the project that interconnect different artifacts. Impact analysis is a simple illustration of that. First, you decide what parts of the application the feature in question might influence. Second, you check how many test cases are connected to those parts. That’s your maximum required regression scope, and now you can picture time requirements and risks more clearly.

Problems

Yeah, test documentation seems very important! Of course, it depends on the type of project, agreements, regulations, client gimmicks, etc., but usually, at least one of the points above remains true. But what about the cons of maintaining test documentation? Oh, there are plenty as well!

Narrowing bottleneck

Testing is a common bottleneck in the software development life cycle (which is a separate topic on its own). Having a tedious procedure of creating, updating, and deleting test documentation in response to every iteration makes this bottleneck even more troublesome.

Cumbersome process

We want to shift left testing activities, right? So, in an ideal world, we want to create test cases as soon as requirements show up. In the real world, though, test cases cannot be considered finished until their author has seen the final implementation of the described feature. Why? Because while they can capture abstract business logic expected right off the bat, actual page behavior, text pieces, UI, and myriads of changes made on-the-fly will become clear only after the author sits down and goes through the actual page.

This requires working on test documentation at least twice: while working with requirements and after code implementation is finished.

It’s boring!

You get paid for your work, so bringing up boredom as a problem here might sound strange. But I really would like not to disregard this point. I never saw a living person enjoying their routine of test documentation updates for another version. When a regular human being does something he or she is not involved in, it becomes mechanical, less context-oriented, and thus more error-prone.

Duplicating the work

It can feel like particularly boring nonsense when you have a test automation solution with system, system integration, acceptance, and API tests (i.e., your tests). Furthermore, with such an automation suite, you now have two places that describe application behavior that both have to be updated and kept synchronized. If you update test cases, be sure to ping test automation engineers about what exactly you updated. If test automation engineers find some discrepancies, bugs, or ambiguities in test documentation, they’ll be coming for you.

Keeping the big picture

Writing new test documentation is relatively easy. It is updating existing documentation that makes life hard. The tester responsible for keeping test documentation up to date is bound to have an exhaustive understanding of the whole documentation, which might be a lot, and constantly keep it in mind. Otherwise, they won’t be able to remember that your change to the ‘Sign in’ button colors affects two test steps in three different test cases from several test suites.

DRYing seems impossible

The DRY principle (Don’t Repeat Yourself), for some reason that remains a mystery to me, is pretty hard to apply to test cases with modern test management tools. They let you reuse test steps, yes, but when it comes to constants — sorry, no. It’s such an obvious feature! Maybe I overlooked some gems, but it’s still yet to be invented. So whenever you want to use names, text lines, colors, positions, resolution, and other properties — you introduce brittle spots in different places that will require your attention simultaneously the very moment those properties are changed.

Solution: generated living documentation

Living documentation is basically any documentation that lives and evolves with the project it describes. Ideally, at least in the past, we wished for specifications and/or designs to be “living”, but modern reality disillusions us in a pretty persuasive manner. Test documentation is currently kept “living”, but at what cost?

Remember when I said “no” to the question stated in the beginning? Well, to be more precise, I said “no” to the “manually” part of the question. I believe that test documentation, living documentation, and test automation can be combined in a single powerful core piece. These elements are intertwined and destined to merge:

1. A test automation solution has to derive from your manual test cases because its coverage should start from basic things, and covering basic things is the first priority of test documentation.
2. A test automation solution has to keep its internal description of the application up to date, otherwise it means red tests, which nobody tolerates.

The only reason I see why the whole world hasn’t adopted this automation-as-documentation approach yet is that it’s not clear how to convert code into human-readable documentation, accessible to anyone at any time. Luckily for us, there are major solutions for this already. I’m going to dive into them for my next topic, so… stay tuned?

But wait, what if there is no test automation?

If the project is not complex or long-term enough for system test automation to be feasible, then you are not burdened that much with test documentation anyway.

If the project is, in fact, complex or long-term enough for system test automation to be feasible, but you haven’t implemented it yet, you’re doing something wrong.

Adopting generated documentation

If your team has been creating test documentation manually for years, your processes and mindset obviously revolve around this fact. You think about terms, risks, and estimations in this paradigm. These are the key things to address when introducing any changes.

Terms. Test documentation can be broken down into two parts: test idea and test implementation. We really care only about the idea because implementation is just details about where, when, and in what format you will write down the idea. In the manual world, the test idea was the monopoly of manual testers. They came up with scenarios, wrote them down, and called it a day. In the automation world, however, it is bound to change from monopoly to cooperation.

The automation world cannot wait for manual testers to come up with test ideas and write them down because that would simply recreate the manual world. Instead, automation engineers, who certainly can include transitioned manual or general testers, come up with automated test cases. They can ask for additional help and expertise from testers during code implementation. Additionally, testers should be involved in the code review process to verify proper coverage.

In other words, test automation should move from being reactive to being proactive. Not react to changes in test documentation and software but proactively create test code based on requirements for future tests and adjust red tests along the way. If it sounds familiar to you, that’s because it sounds like ATDD (Acceptance Test-Driven Development), at least like its weird asynchronized and distributed version.

Risks. The main risk here is insufficient coverage. Automation engineers are usually slightly or completely detached from the pulse of application development. They might not be up to date with the latest news and details, and management might not invite them to all the discussions, because the testing squad is already represented there with manual testers, right?

That must change immediately. Automation engineers should get all the information firsthand. That is a part of the reactive-to-proactive transformation. There should be no such thing as “knowledge exclusive to manual testers”. Whatever manual testers know about an application, whatever experience and solutions they might have accumulated — it should be on the table for everyone.

And still, we would need testers to review test automation MRs for test coverage because some experience is not easily shared from person to person.

Estimations. Time requirements do not change that much. After all, you still create tests and update them as features get their final polish. What changes is that manual testers get more free time on their hands, which can be spent more wisely: finally attend those parts with high human decision involvement, extend the coverage further from basic functional testing, investigate edge cases, spend more time with developers and the test automation team, etc.