A flaky test produces inconsistent outcomes, often passing and failing when you expect the same behavior. Vansah’s Flakiness indicator highlights repeated Pass/Fail changes so you can investigate the underlying Test Runs. It is a signal to review, not a diagnosis of the cause.
Read the indicator
Label | What it means |
Suspected Flaky | The assessed runs show repeated switches between Pass and Fail without a sufficient recent streak of the same result. |
Not Flaky | The assessed history does not currently show the suspected pattern. This does not mean the case passed or can be ignored. |
Insufficient data | There is not enough eligible completed Pass/Fail history in the assessed scope. |
A consistently failing case can show Not Flaky. Review its Failed result as a problem even when its outcomes are consistent.
Find cases to investigate
Open the Test Case list in Vansah and check the Jira space and testing context.
Show the Flakiness column using the column selector if it is hidden.
To narrow the list, add Flakiness from More and select Suspected Flaky.
Select the case’s Suspected Flaky indicator, then View Test Runs.
Open the relevant runs and compare their results and evidence.
Compare the execution conditions
Check | Question to answer |
Test Case version and script | Were the instructions or expected results changed between runs? |
Environment, release, and build | Were you testing the same application and configuration? |
Test data and starting state | Was the account, data, or dependency state different? |
Actual results and evidence | Which step changed, and what do the screenshots, comments, or logs show? |
Automation and dependencies | Could timing, unavailable services, or test setup explain the change? |
For example, a checkout case might alternate between Passed and Failed because a test account sometimes has an empty cart. Confirm that with the recorded data and evidence before changing the test or raising a product defect.
Understand the assessment scope
The calculation uses recent history with a 30-day window and a limit of 50 run records. Only completed Pass/Fail outcomes contribute to the assessment. Untested, incomplete, or other outcomes may leave too little eligible history.
The testing context can change the history being assessed. A repository view and a work item or Test Plan requirement view may show different evidence. Compare the run conditions yourself; the indicator does not prove that every run used the same environment, version, or data.
A recent streak of consistent outcomes can clear a suspected pattern. Those consistent outcomes can be failures as well as passes.
Record the follow-up
If the evidence points to a product problem, create or link a defect from the relevant execution.
If the procedure or expected result is unclear, update the Test Case and document the reason.
If setup or automation is unreliable, record the condition and fix that cause before relying on reruns.
Record a new execution when retesting so you can compare it with the earlier result.
If the indicator and filtered list disagree, record the case key, current context, and visible labels for Vansah support. Do not change results just to clear the indicator.
See understand test results and view Test Runs.