Should Test Cases Include Steps? QA Owes Information, Not Acts

Should a test case spell out the exact steps? The short answer is that writing them or not writing them is not the real question. Steps are unnecessary as instructions given in advance, and indispensable as a record kept afterwards. The real problem is not the format at all — it is what QA hands to the organization as its deliverable.

One line per step, or just the key points, or some template in between. It looks like a question about formatting, but follow it far enough and you arrive somewhere else: not at the granularity of steps, but at what that procedure document has been made responsible for inside the organization.

The argument of this article is that most of what goes wrong around test procedures does not come from immature test technique. It comes from choosing the wrong unit of accountability.

I start from how to write a test case and end at a single claim: what QA takes responsibility for is not activity, but information. I include the counterarguments along the way, and I state explicitly where this claim would break — its falsification condition.

1. Who this article is arguing with (not the people writing procedures)

Before the argument itself, let me place it. Skip this part and everything after it reads as an attack on working QA engineers, which is not what it is.

1-1. The question is about where accountability sits, not about people

This is not a criticism of QA engineers who write step-by-step procedures, or of teams that have accumulated large test case libraries. I have written enormous numbers of test cases with every step spelled out, and I know there are situations that require exactly that.

What I am questioning is not the quality of anyone’s individual work. It is the structure: where QA’s deliverable sits inside the organization. The same person doing the same careful work will pass a completely different amount of information to the organization depending on where that deliverable is placed. That is a design question, not a skill question.

1-2. This is a way of thinking that protects QA itself

One more thing, because it is easy to misread. This argument is not about loading new obligations onto QA. It is closer to the opposite.

When you report “all test cases passed” and an incident happens in production afterwards, the person put in the worst position is QA. Nothing in the report was false, and yet the question comes back: what exactly was testing doing? That asymmetry starts the moment QA defines its own job as “the work we performed.”

Where should QA’s value sit so that this asymmetry disappears? This article is one answer to that question. It is not a claim that anyone’s method is wrong — it is a question about which shape actually protects QA.

2. The starting question: should test cases contain concrete steps?

Let me work through the answer I gave at the top. Steps themselves are not the problem. The problem is where the document containing them has been placed.

2-1. Real users do not follow the path you wrote down

Start from the obvious premise: real users do not travel only the path we wrote into the procedure. To reach the same goal they enter from a different screen, operate in a different order, back out halfway through, and continue on a different device.

If that is true, then what a test should be confirming is that the expected result holds even when the steps differ somewhat. Most people, I think, agree this far.

2-2. But “therefore stop writing steps” does not follow

The conclusion “so don’t write steps” does not follow from that, and this is the first place people trip.

If there are paths A, B, and C leading to expected result X, and you want to claim X holds on all of them, then you are committing to writing three sets of steps. The volume of writing goes up, not down. Push toward “write no steps” and you lose the path information itself.

2-3. The document is carrying two jobs at once

The problem is not the existence of steps. It is that the procedure document carries “what to verify” and “how to operate” at the same time.

These two belong to different artifacts:

• “What to verify” belongs to the expected result
• “How it was operated” belongs to the execution record

Split them that way and steps-as-instruction become unnecessary while steps-as-record become mandatory. Put differently: steps are not something you write in advance, they are something that remains afterwards.

Delete the steps without making that split first and things get worse. When something fails you no longer know which path it failed on, and when everything passes you cannot guarantee that the risky paths were actually traversed. Steps should not be removed — they should be moved. That is the first step.

3. What is structurally broken about running fixed steps

Fixing the steps and running them repeatedly has a defect that is different in kind from inefficiency: the fact that something was not verified never appears in the report.

3-1. Unverified paths do not show up as unverified

Take a process that verifies three paths, A, B, and C. A path D also exists, but nobody thought of it at design time. What happens?

D is not merely unverified. The fact that it is unverified never surfaces in the report. The result reads “all passed.” Whoever reads that report learns neither that D exists nor that it was never checked.

3-2. Design-time imagination becomes a ceiling, and the ceiling never moves

The one property fixed test cases have is that they can re-verify the same paths repeatedly. And that is only re-verification of whatever someone happened to imagine at some point in the past.

The imagination available at design time becomes a ceiling, and that ceiling does not move over time. Run the regression suite as many times as you like — D will never come out of it.

This is where not fixing the paths shows its real meaning. It is not that coverage goes up. It is that the ceiling moves. If each execution travels different paths, then paths nobody knew existed at design time get picked up as time passes. What you can pick up is limited to the paths actually travelled, and whether you notice an anomaly there depends on how far your expected results are written out.

• Fixed cases = a mechanism for defending known paths
• Varying the paths = a mechanism for turning unknown paths into known ones

The former is a by-product of the latter. Find an important path and you can pin it. The reverse does not work: no amount of accumulated fixed cases produces an unknown path.

3-3. A fixed case is not a net — it is a clamp

That changes what a fixed case is for. It is not a net prepared in advance; it is a clamp that holds on to something already found. It is not an output of the design phase but a by-product of execution that accumulates over time.

This is not only my reading. A Japanese engineering blog post on defect analysis makes the same observation:

I have seen it several times: the countermeasure for a production incident becomes “add the test case we missed to the regression suite and check it every time.” But that countermeasure does not address the root cause of the defect leak (the missed test), so the same cause is highly likely to produce another missed test when a different feature is modified.
(Kawachiibaa’s blog, “How I do defect analysis (analyzing missed tests and defect-leak causes),” August 10, 2021, in Japanese)

What is being described there is exactly the state of “just adding more clamps.” Adding the D you found to the fixed suite changes nothing about the E, F, and G you have not found. Used as a clamp it is perfectly correct; called a preventive measure, it leaves the ceiling exactly where it was.

4. Regression testing verifies expected results, not paths

“If we stop fixing the paths, won’t we lose comparability with last time?” is a fair worry. But it mistakes what regression testing is for.

4-1. The comparability objection

This week A, B, C all pass. Next week A, B, D all pass. Doesn’t that mean C could be broken without anyone noticing? A reasonable point.

But what regression testing verifies is sameness of expected results, not sameness of paths. Does the expected result that held last time still hold? That is the question being asked.

If you can confirm that, the paths do not have to be identical. In fact, holding on a different path widens the range of what you confirmed. “X holds on path A” is a weaker claim than “X holds on A, B, and D alike.”

Fixing the paths was only ever an implementation convenience. If you run by hand, fixed steps are cheaper — that is the whole reason. Remove the effort constraint and the reason for fixing them disappears with it.

4-2. Triage is unaffected by whether paths are fixed

The same goes for triage. “D failed — is this an environment problem or a bug?” is a question that has nothing to do with whether paths are fixed.

If it is environmental, A, B, and C would show the same thing. If it is not environmental, it is the same phenomenon as “A had been passing and suddenly failed.” In how you handle a failure, there is no reason to treat a newly travelled path as a special case.

5. Making the procedure the deliverable was a choice, not a constraint

I just wrote that fixing paths was an implementation convenience forced by effort constraints. But the constraint only explains so much.

5-1. “We didn’t have the hours” does not finish the explanation

Expected results were always primary and paths always secondary; manual execution simply obscured that. It is a tidy explanation.

It does not hold. The constraint explains why fixing the steps was an available option. It does not explain why that document was made the thing handed outside the team.

Even under the constraint, steps could have stayed as internal working material while the deliverable handed to the organization was “what holds, and what we do not know.” Not having enough hours and deciding what to publish are separate judgments. And what actually got published was the procedure document.

5-2. Choosing a deliverable is declaring a scope of accountability

Choosing what to deliver is the same act as declaring what you are accountable for. The moment you hand over a procedure document, you have said: this is what I take on.

That is why the metrics became case counts and completion rates. I read the causality in the opposite direction from the usual one: it was not that metrics existed first and procedure documents followed, but that procedure documents existed first and the metrics conformed to them. Once the procedure is the deliverable, the only things measurable from it are how many were written and how many were executed.

5-3. “All passed” survives as a quality report

The decisive symptom is that “all test cases passed” functions as a report about quality.

I have watched incidents hit production after testing was reported as fully passing. Nothing in those reports was a lie — every written case had passed. The incident happened anyway, for a simple reason: the path where it happened was never in the test cases at all.

That report was not wrong about anything, and it was useless. The person reading it could not make a single decision from it.

And this only works when steps are the unit of accountability. If the deliverable had been “the scope we could not verify” from the start, you could not write that report even with fixed steps. The absurdity of “all passed” comes directly from how accountability was defined, not as a side effect of an effort constraint.

5-4. The procedure document also works as an indemnity device

There is more. A procedure document also functions as an indemnity device. Once you have executed and reported the steps you wrote, an incident on a path you did not write is not a failure to perform on QA’s part.

I have seen this used in practice. In an incident retrospective, “it wasn’t in the test cases” ends the discussion right there. Nobody is doing this in bad faith. Given that the deliverable is a procedure document, it is the logically correct response. The moment you decide to hand over that document, everything you did not write sits outside your accountability.

So moving the deliverable to expected results plus unverified scope is not merely shifting an axis. It is a move from “accountable for what I did” to “accountable for what I decided not to look at.” You put your name on what you chose not to verify and why. The burden increases. It increases — and in exchange, when someone asks what testing was doing, you can answer.

6. What QA takes on is information, not activity

Compressed into one sentence: what QA takes on is not activity but information.

6-1. Defining “activity” and “information”

The terms need precision, because writing a report is itself an activity, and used loosely the distinction says nothing.

• Activity = the fact of having performed. “We executed 200 cases.” “We covered every path.”
• Information = content that can change a decision. Something that could alter what the reader chooses to do

So the contrast is: do you make the fact of having performed your deliverable, or do you make decision-changing content your deliverable?

6-2. The three things a report should carry

From that, QA has three things to deliver.

• What you were trying to protect (the expected results)
• The paths on which you confirmed those results hold
• The scope you could not verify, and why

As an actual report, that looks something like this:

What we were protecting: when a payment completes, the order and the inventory always match.
Paths confirmed: card payment (web and app), coupon combined with card, cancellation after payment.
Not verified: convenience-store deferred payment — the payment notification is not wired up in the test environment; unverified for the same reason last cycle. Simultaneous purchases from one account — no way to reproduce it; newly unverified this cycle.

Note that execution counts are not among the three. “200 cases executed, all passed” fails to mean anything not because the number is small, but because the unverified scope never appears in it. It says nothing about the risk that remains.

6-3. Counts can be information, though

The format of a count is not itself forbidden. If 180 passed last time and only 150 pass now, that carries the information “something got worse.” “Fact of having performed” versus “content that can change a decision” is not a binary — it is a matter of degree.

6-4. The test: can a reader tell what changed since last time?

So where is the line? There is one test.

Reading this report, can you tell what changed compared with last time?

If you can, then even a count is functioning as information. If you cannot, then no matter how detailed the document is, it is a proof of execution and nothing more.

This shift is not something only I am arguing. NTT DATA, one of Japan’s largest systems integrators, has written that with today’s diversity of development technologies, “using source lines of code (SLOC) as the size measure when calculating bug density and test density is no longer considered appropriate” (in Japanese), and proposes “viewpoint coverage” and “DDP monitoring” as replacements. DDP (Defect Detection Percentage) measures how many of the defects that should have been caught in a phase were actually caught in it, so the value drops as more defects slip through.

That is the same movement seen from the metrics side: measuring not “how much did we do” but “how much did we miss.” Going out to measure what you cannot see points in the same direction as this article.

Practitioners describe the same instinct. In a Qiita post on aggregating test progress, a QA engineer at MIXI writes that when the defect count falls too far below the historical average, “an excessive dip starts to make me worry whether we are actually detecting defects properly” (in Japanese). Not the count itself, but the delta from last time. In practice, this is already how it gets read.

7. Why “state the unverified scope” resists hollowing out

“‘We stated the unverified scope’ will turn into empty ritual just like ‘all passed’ did” is a fair suspicion. Ritual is possible — but the conditions under which hollowing out happens are different.

7-1. Different conditions for going hollow

“All passed” is hollow because receiving it changes no decision. The only content is a pass, so the reader has nothing to choose between.

If the unverified scope is written down, the reader is presented with a choice: have it verified, or ship as is.

But a choice appearing and a choice being made are different things. An operation where the reader skims past the unverified section and approves anyway is entirely plausible. I have not confirmed this part.

Ritual can certainly set in. But that would be degradation of the implementation, not a defect in the format — and if the same wording appears every cycle, the fact that nobody re-evaluated is visible in the artifact. “All passed” degrades without the degradation ever becoming visible. That is the decisive difference.

7-2. You cannot write the whole unverified scope either

There is a stronger objection. The unverified scope is not just D; it is E, F, G, endlessly. Write everything and the report grows without limit; narrow it and the narrowing criterion creates a new invisible region. Either way you are selecting.

That objection holds. Whoever writes the unverified scope is also selecting.

7-3. The difference is what happens when you select badly

The difference is the behavior on a bad selection.

A procedure document reports “pass” about the paths it did not select. A bad selection turns into a positive claim.

An unverified-scope report says nothing about what it did not select. It reads “A verified, B unverified, nothing said about the rest,” and the reader structurally knows there is a region nobody spoke about. A bad selection stays as silence.

7-4. Silence is not misinformation

That is the fork. Silence is not misinformation. A pass is.

Being unable to write everything and telling an untruth are different things. The constraint of incompleteness applies equally to both formats; what differs entirely is how the unwritten part gets treated.

8. What if nobody reads it?

Even with the unverified scope written down, the person making the release decision can approve without reading it. Worse: the moment it is written, QA is indemnified and the decision-maker absorbs the downside, so the political response should be pressure to “stop writing so much.”

Or perhaps what actually comes back is not pressure but indifference. Resistance costs something; ignoring costs nothing.

This is the most practical and most painful objection to the argument. I have two responses.

8-1. If pressure comes back, accountability has moved

First: pressure coming back means stating the unverified scope is in fact moving accountability onto the decision-maker. If nothing had moved, nobody would complain. Resistance appearing is evidence that accountability which had been floating unattached has landed in someone’s hands.

8-2. Even ignored, the ignoring is on the record

Second: even if it is ignored, the fact that it was ignored is itself on the record. The state becomes “it was handed over and not read.” “I didn’t know” stops being available.

But both of these are predictions I have not confirmed either. It remains possible that no pressure comes back and the record is never consulted, and the practice settles in that state.

8-3. The problem with the standard “worthless if unread”

Beyond that, I think there is a problem with the standard itself, because it measures QA’s accountability by the reader’s behavior.

QA’s job is not to make people decide. It is to fix the conditions under which the decision was made. Whether it gets read falls inside the reader’s scope of responsibility and does not determine whether QA discharged its own.

And if you trim accuracy to make it more readable, you are deliberately degrading the material the reader decides on. That is the road back to “all passed.”

8-4. Total risk is unchanged — only visibility differs

Suppose 200 unverified items pile up and nobody reads them. That still means the fact that 200 risks exist has been made visible.

Under fixed-case operation the same 200 risks exist. They have simply been absorbed into “all passed” and disappeared. The total quantity of risk is identical; only whether it is visible differs.

So the trade-off is not between accuracy and reach. It is between carrying risk without seeing it and seeing risk you cannot finish processing.

There is a line not to cross here. Saying “there are 200 unverified items and no trace of them being consulted in the last three decisions” is different from saying “this organization is beyond its capacity.” The first is observation, the second is evaluation. QA delivers the first; the decision-maker judges the rest.

9. Does clearer attribution actually improve quality?

If what this format does is fix, after the fact, whose judgment it was, then the incident rate is unchanged and only the attribution moves. Users do not want to know whose fault it was; they want a product that is not broken. This is also a strong challenge.

9-1. Clear attribution is not the goal — it is the route

My answer is that clarified attribution is not the goal but the route.

When the unverified scope becomes visible, the decision-maker chooses: add verification, or accept the risk. The moment an act of choosing occurs, the risk enters management.

Risk under management becomes material for the next decision. Accumulate a history of “we waived D last time and an incident followed” and D enters the verification scope from then on. This is not the same as adding a missed case to the fixed suite: what remains is not the path itself but the record of having decided to waive it.

If the unverified scope is invisible, D is treated as nonexistent, the incident is filed as “unforeseen,” and the same thing happens again. No learning occurs.

9-2. Post-incident analysis stops stopping

This shows up most clearly in post-incident analysis.

If the report said “all passed,” the answer to “why couldn’t we prevent it?” is “we didn’t anticipate it.” The analysis stops there. There is nothing further to say about something unanticipated.

If the report said “D unverified,” the answer becomes “we anticipated it and shipped without verifying.” Now the questions continue. Why wasn’t it verified? Who made that call? Was the criterion sound?

This effect — that post-incident analysis does not stop — is the center of the argument. And it is also unconfirmed.

Relatedly, defect-leak analysis has been discussed in Japanese software quality circles for a long time. The Software Quality Management Research Group at JUSE’s SQiP (the Union of Japanese Scientists and Engineers’ software quality initiative) took up the theme “Preventing bug leakage — what tests would have found the bug?” back in 2011 (SQiP software quality library, in Japanese). The title itself is shaped like a post-incident question, which is what I find notable. Whether that question can be answered depends heavily on what was written in the pre-release report.

9-3. The “records don’t get read” objection

In real incident response what gets read is logs and code, not the pre-release report. This objection holds. Records existing but never being consulted is entirely plausible.

There is another way to put it, though: perhaps they were not read because there was nothing worth reading. A report containing only “all passed” has no reading value. A loop may have been running — not read so not written, not written so not read.

10. What this definition rules out, and where it is weak

“If anything can be called information, this definition rules out nothing” is a legitimate suspicion. Building checks into CI, raising points in review, writing a report — all of them can be described as delivering information.

10-1. It rules out exactly one thing

One thing is ruled out: making the fact of having performed your only deliverable.

“We executed 200 cases,” “we covered every path” — unless they carry what changed since last time, they change no decision. That one thing, and only that, is clearly placed outside by this definition.

And ruling out exactly one thing is the weakness of this definition. As a definition of a job function, the outline is loose. I acknowledge that and keep it anyway, because being loose and being wrong are different. And failing to rule out that one thing is at the center of what goes wrong in practice.

10-2. A green CI can have the same structure

Building checks into CI sits inside this frame too. Someone decided what to check, and that decision is itself a definition of what to protect.

Nothing is known about what was not built in, and a green CI is only the information that “the checks we built in passed.” Structurally identical to “all passed.”

So stating what is and is not in CI remains necessary. Whether the execution is automated is independent of this problem.

11. The falsification condition

If I am going to make a claim, I should also write where it breaks. This definition has a testable condition.

Reports that state the unverified scope and reports that do not produce no difference in how far post-incident analysis gets. Observe that, and the center of this argument is empty.

There are two ways to check. Compare one organization before and after switching practices, or compare across organizations.

A strictly controlled experiment is impossible — you cannot observe both states of the same incident. But that is the same constraint that applies to management and to engineering, and it does not mean nothing can be checked. Apply that standard consistently and every competing proposal becomes unfalsifiable too.

12. What changes when AI does the execution

Everything so far was tool-independent. Now: what changes, and what does not, when AI performs the execution.

Let me be precise about the premise. This is not “hand it to AI and it will go explore your service for you.” It is an operation where you fix the expected results and let AI vary the paths that reach them.

Humans supply the expected results to protect (order and inventory match after payment, say) and the scope those results define — target screens, permitted input values, preconditions on the data. AI decides only which paths to travel, in which order, within that scope.

Humans draw the outer boundary of the search space; AI varies the route inside it. That route is the only thing AI varies.

If something you could only run on one path by hand can now be run on three, what you can report changes. Saying the same expected result holds on three paths is qualitatively different information from saying it holds on one.

This is a means of making unfixed-path operation feasible. It removes the blocker that used to stop the discussion at “we should write multiple paths, but the hours.”

12-1. Where AI scales things

Anomaly detection, classification, deduplication, blast-radius estimation, and raw throughput. This is where it pays off in practice.

Varying paths makes execution cheap, so run counts go up and the total volume of anomalies goes up with them. If the processing side stays manual, the practice converges on “run a lot, let anomalies pile up, leave the expected results as originally written.” AI clears that blockage.

Note that anomalies piling up and clogging does not happen with fixed cases. Not because that is healthy, but because nothing is being picked up. Fixed cases only look at phenomena named in the expected results, so the unanticipated is never observed in the first place. A clog forming is evidence that observation is occurring.

12-2. What humans keep

Judging “is this phenomenon a spec violation or the spec working as intended?” demands information from outside the spec. What should happen from a business standpoint, what the contract says, what this feature was meant to achieve. AI has only what it was given.

This is not an open problem but a division of labor. AI’s detection ability is bounded by the granularity of the expected results you hand it. A side effect that only occurs on path B passes straight through unless it appears in an expected result. So the work of adding path-fragile behavior to the expected-results side remains.

Anomalies that cannot be judged go into the unverified scope. Not knowing whether something is a bug or the spec is the same as not having verified it. It lands on the stated side, not the silent side.

The same applies when AI generates the expected results themselves. Record that “this item’s expected result is inferred and has not been confirmed with the business side,” and it becomes a form of unverified scope. The danger of AI filling in blanks arises only when the filling-in goes unrecorded.

There is also the question of whether AI execution records can be trusted. That is not AI-specific. A tester writing “executed” without having executed was the biggest hollowing-out pattern in procedure-based operation. Human execution depended on self-reporting; AI execution leaves machine logs. Verifiability goes up, not down. A suspicious record can be checked by fixing that path and re-running it.

12-3. What tools cannot move

Whether stating the unverified scope actually changes organizational behavior. Tools do not move this.

AI can make reports accurate, enumerate unverified scope exhaustively, and cross-reference past reports during post-incident analysis. The material side is fully solvable. But material existing and an organization acting on it are different things.

If anything, the better AI gets at producing precise unverified scope, the more a reader takes on by reading it — so the reasons not to read get stronger.

And I do not think this should be solved with technology. Make being read the objective and you start writing reports that are easy to read, which is the way back to “all passed.” This is not an unsolved problem; it is the boundary of QA’s accountability.

13. Where this sits relative to existing work

Let me place all of this against the existing literature. To say it up front: this is not a new claim.

13-1. Structurally identical: Session-Based Test Management

Session-Based Test Management (SBTM), formalized by James Bach and Jonathan Bach in 2000 out of practice at Hewlett-Packard, has the same structure as what I have described.

Charters are written open-endedly, not as exhaustive descriptions of what to do. After execution, a session sheet is produced so a third party can see what happened. The report records missing information and unresolved questions, and the debrief asks what got in the way and what is still left.

Do not fix steps in advance; keep them as a record afterwards; include what could not be verified in the report. The structure matches.

Further, the title of a 2014 paper by Bach and Aaron Hodder is literally “Test Cases Are Not Testing” (Testing Trapeze, inaugural issue, April 2014). Its argument: creating and running test cases is not the same as testing, test cases are an unsuitable foundation for organizing a testing process, and they are no basis whatsoever for measuring testing progress.

So this is a way of thinking formalized in 2000 that has remained a minority position ever since. What AI-driven execution changed is the cost of this structure, not the structure.

The reason it never became mainstream is also inside this article. Given a structure where readers are better off not reading, organizations do not actively choose accurate reports. A shape with “no test cases, no expected pass/fail metric” is hard to adopt for a manager who wants progress as a line chart. Not a defect in the structure — a matter of organizational preference, as I read it.

13-2. Same direction: ISTQB and Google

The testing principles in the ISTQB Foundation Level syllabus v4.0 state that testing can show the presence of defects but cannot prove their absence, and that exhaustive testing is impractical so risk analysis and prioritization must narrow it down. That a test case is only one example is the official position too.

A caveat, though. In ISTQB terminology QA is process-oriented and preventive, while testing is a form of quality control (QC). By that definition, what I have discussed here is QC.

But ISTQB also states that QA is everyone’s responsibility — the agent of prevention is not the QA department. If so, what a QA practitioner can produce for prevention is only the information “this process is not preventing this”. ISTQB writes prevention as the objective without writing the means. What I have described can be positioned as filling in that means.

In How Google Tests Software, Google states that quality is not equal to testing, that having few dedicated testers works because developers own quality, and that engineers with “test” in their title exist to enable other engineers to test well. Same direction on “performing activity does not guarantee quality”, but that is a description of organizational structure, not of where the deliverable sits.

These two face the same direction without saying the same thing. Only SBTM matches structurally.

13-3. The alternative the mainstream actually chose: CI/CD and production monitoring

Here is the strongest counterargument. What the mainstream actually chose was a different answer: CI/CD and production monitoring.

Deploy more often, roll out gradually, detect anomalies in production. Rather than supplying information in advance for someone to decide on, make the decision granularity small enough that mistakes can be undone. This works, and it spread.

It has two limits.

What can be undone is limited. You can roll back a deployment, not the processing that already ran. Incorrect charges, lost data, notifications already sent do not come back. Cap the blast radius at 1% with a canary release and that 1% is still real users.

Production monitoring does not know the expected results. It catches error rates and latency changes; “this amount is wrong under the business rules” returns as a normal response and is not caught.

And a green dashboard is the information “the monitored items are normal.” What to monitor was decided in advance, so nothing is known about what was not decided on. Structurally the same as “all passed”: you cannot see outside the scope you fixed beforehand. The alternative looks to me like the same problem rebuilt in a different location.

13-4. A genuine counterexample: regulated industries

There is one domain where this argument does not hold. In medical devices, aviation, and automotive functional safety, procedure documents are legal deliverables. Prior approval and reproducibility are required, and “we varied the paths and recorded afterwards” does not stand as evidence.

The purpose of what is demanded there does sit on the information side. What must be protected is defined as requirements, and what was verified against them must be traceable. Documenting residual risk is mandatory, and “we checked everything” is not accepted as a report. Fixing the steps is specified because verification-by-reproduction was adopted as the evidentiary form.

But the requirement to fix the steps still stands. The purpose sitting on the information side does not mean “don’t fix the paths” is adoptable in this domain. I treat it as a counterexample.

14. The same direction, from practitioners in Japan

Is any of this far from what practitioners actually feel? Not especially. Publicly available Japanese writing contains observations pointing the same way.

14-1. The movement happening on the metrics side

NTT DATA’s piece on quality assurance that does not depend on bug density and test density, cited earlier, is trying to abandon quantitative metrics and move to a metric that measures the proportion of defects that slipped through (DDP). A move from “how much did we do” to “what did we miss.”

From defect analysis practice comes criticism of adding missed cases to regression as a preventive measure (the Kawachiibaa blog cited above). That is the same point as section 3 here: piling up fixed cases does not move the ceiling.

14-2. The listed drawback of exploratory testing is precisely the issue

What I find interesting is the drawback usually listed for exploratory testing. An explainer from SHIFT, a large Japanese software testing company, states that exploratory testing “has the drawback that the full volume of tests performed cannot be grasped” (in Japanese).

That is exactly the issue this article has been circling. Not being able to grasp the full volume is not a flaw of exploratory testing — it is the actual condition of testing as an activity. Fixed-case operation does not grasp the full volume; it calls the graspable range “the full volume.” Which is how “all passed” becomes sayable.

The same article notes that trying to account for every exception means “having to factor in patterns that cannot occur in operation, which you can imagine is far too much effort in both design and data preparation.” The recognition that up-front design has practical limits is shared.

If so, one question remains. Do you write the range you could not finish designing into the report, or not?

15. Conclusion: is what should be delivered actually being delivered?

The conclusion: what QA takes on is information, not activity.

QA’s accountability is to define what must be protected and to state accurately whether it holds — including the parts that do not hold and the parts that are unknown. Steps are not part of that.

15-1. The answer to the opening question

What was wrong with procedure-based operation was not inefficiency, and not low coverage. It was placing the act of execution as the unit of accountability.

An activity is only a means of producing material for someone’s decision. Make the means the deliverable and the decision material is lost — which is how a contentless report like “all passed” becomes possible.

Not fixing the paths is only part of correcting that error. At the center is a single point: what should be taken on is information, not activity. Whatever the tools, whether or not the paths are fixed, if the report does not contain the unverified scope, QA has not discharged its accountability. If it does, then how paths are handled drops to a technical choice.

Back to the opening question. Should test cases contain concrete steps? The answer is that writing them or not is not the question. As long as the debate is about the presence of steps, execution is still sitting in the seat of accountability.

The question to ask is whether that document is material for someone’s decision, or proof that work was performed. If it is the latter, then steps or no steps, what should have been delivered has not been.

15-2. Three limits of this argument

End without this section and the article itself becomes an “all passed.”

The outline of the definition is loose. It rules out one thing: making the fact of having performed your only deliverable. As a definition of a job function that is weak. But failing to rule out that one thing is at the center of what goes wrong in practice.

It is not a new claim. SBTM reached the same structure in 2000. Arriving independently at the same place supports the validity of the reasoning but is not a novel contribution to the field. What changed is the cost, not the structure, and AI-driven execution is what removed it.

The central part is unconfirmed. Whether stating the unverified scope actually changes how far post-incident analysis gets — only operation answers that. If it does not change, then either this claim is wrong or the organization has a different problem. Either would be worth knowing.

15-3. This is for QA’s own protection

To repeat what I wrote at the top.

This article does not reject the people writing procedure documents. The procedure document was a rational invention for securing a verification scope within limited hours. The problem is not the document itself but that it became the only deliverable reaching the organization.

And in that structure, I think the party losing the most is QA. Do the work accurately and what you are able to report is limited to “we performed it.” When an incident happens you are asked what testing was doing. You are indemnified, and your value is never conveyed.

Writing the unverified scope looks like added burden and actually strengthens QA’s position. A function that can state, under its own name, what it looked at and what it did not, is far harder to replace than a function that accumulates proof of execution. People holding decision material get invited into the room where decisions happen.

Whether to write steps in a test case comes after that. The first question is: can the person reading your report say what changed since last time? That is the starting point.