Flaky tests

Available onBasicProEnterprise

A flaky test is one that fails and passes on the same code. It trains your team to click Re-run instead of reading the failure, burns CI minutes, and hides real regressions. Gitera finds flaky jobs and tests automatically from the runs it already receives — you do not change your workflows or upload test reports.

What you getCopy Anchor Link

• A Flaky tests page that lists every flaky job and test with the evidence behind it.
• Wasted CI time and cost per item, so you can fix the expensive ones first.
• Dismiss / restore, a one-click retry rule for flaky jobs, and alerts when a new flaky item is detected.
• A Known flaky note on pull request failure comments, so authors know when a failure is a known flake.
• Ask the AI agent — "Which tests are flaky, and what do they cost us?" — for the same data in chat, including wasted CI time and 30/90-day trends.

Jobs vs testsCopy Anchor Link

A flaky test usually makes its job flaky too. On the Flaky tests page, the tests of a flaky job are grouped under the job row — expand N flaky tests in this job to see them. The job row tells you where the flake shows up; the test rows tell you which test causes it. A flaky test whose job is not flagged as flaky on its own is listed as a separate row.

Because job detection looks only at job outcomes (not test log output), any GitHub Actions job that meets the signals — including build, lint, or deploy jobs that do not run tests, as well as test suites whose console output is not parsed — appears as a flaky Job row. Because GitHub Actions bills runner minutes and executes auto-retry rules at the job level, surfacing unreliable jobs alongside tests lets you spot and stabilize every source of non-deterministic CI failures in one place. With the Tests kind filter (or any Framework filter), job rows are hidden and every flaky test is listed on its own.

How detection worksCopy Anchor Link

Detection runs every 6 hours (00:00, 06:00, 12:00, 18:00 UTC) over the last 14 days of runs, for every organization on a paid plan.

The three signalsCopy Anchor Link

Pass on retry is the strongest signal: the code did not change between the two attempts, only the outcome did. Manually triggered runs (workflow_dispatch) are not used for the same-commit signal because their inputs can differ between runs.

ThresholdsCopy Anchor Link

• Pass on retry and same commit must be seen on at least 2 different commits before an item becomes flaky. One unlucky run is not enough.
• Flip rate is a separate path and is enough on its own for jobs: a job whose default-branch runs keep flipping between pass and fail is flagged once there are at least 10 runs, even with no retries and no commit that both failed and passed. Such a job can show 0 commits.
• A job or test that fails in 80% or more of its runs is broken, not flaky, and is not flagged as new — fix it as a regular failure in Failures.
• An item that is already active and becomes broken stays in Active with a Consistently failing badge instead of moving to Resolved, since nothing has recovered. The badge clears once it fails on fewer than 80% of runs.
• Cancelled and skipped runs are ignored.
• Matrix jobs are tracked separately by job name, so a test that only flakes on one OS or version is flagged on that job.

LifecycleCopy Anchor Link

Each item moves through Active → Resolved or Dismissed:

• Resolved automatically when it has not flaked for 7 days. The delay prevents an item from flipping back and forth. Until then, every detection run refreshes its metrics (commits, failure rate, wasted time), so the numbers stay current even while it is not flaking.
• Dismissed by you. A dismissed item stays quiet for a 30-day cooldown so a known issue does not keep alerting. Restore it at any time.

How tests are identifiedCopy Anchor Link

Gitera reads the log of every failed job, finds the tests that failed using the framework's standard console output, and stores each failed test with the run, attempt, and commit it failed on. A test is flagged when it failed in a job whose same commit also produced a passing run of that job — a passing job means its tests passed.

Supported frameworks:

Test names are normalized so the same test has the same identity on every run: durations, line numbers, retry suffixes, and runner paths are removed. Up to 200 failed tests are stored per job, and for very large job logs (over 5 million characters) log analysis reads the first 1 million and the last 4 million characters, so end-of-run failure summaries are always included.

The Flaky tests pageCopy Anchor Link

Open Flaky tests under Monitoring. The page follows the app's light and dark theme.

• Summary cards — active flaky jobs (and how many flaky tests on top of them), wasted CI minutes over the last 14 days (with the estimated runner cost), and the number of workflows affected. Wasted time in the cards is counted once per job: a job and the flaky tests inside it share the same failed runs, so they are not added together. The cards always cover every active item; when you switch to another tab or apply filters, a note under the cards says so.

• Analytics — a tab next to Active, Resolved, and Dismissed that shows how flakiness evolves over the last 30 days (switch to 90 days):

  • Tiles for items active now (with the change since the start of the range, and the split into jobs and tests), new and resolved items in the range, and the peak day.
  • Active flaky items — a stacked area chart of active jobs and tests per day.
  • New vs resolved — items detected (above the line) and resolved (below) each day.
  • By repository — active flaky jobs and tests in each repository, largest first. Up to six repositories get their own bar. When there are more, the five biggest get their own bar and the rest are grouped into one.
  • By test framework — active flaky tests by the framework that reported them (Jest, Playwright, pytest, and so on). Jobs have no framework, so they are not counted here.

  Counts are calculated from when each item was detected, resolved, or dismissed, so the full range is available right away. An item that was dismissed and later restored counts as active for the time it was dismissed. The repository and framework charts always show the items active now, whichever range is selected. The open tab is kept in the URL (view=analytics).

• Tabs — Active, Resolved, and Dismissed. Switching tabs does not reload data. Active always lists every item; Resolved and Dismissed list the 500 most recent items and say so when older ones are left out.

• Filters, search and sort — on the same row as the tabs:

  • Filters — one pill per filter: Kind (All · Jobs · Tests), the test Framework, the Repository, and the Workflow (listed by file name). Click a pill to pick values in its own small menu. A pill in use turns teal and shows what you picked, such as Repository | acme/web +2. Selecting a framework narrows the list to tests of that framework, since jobs have no framework.
  • Search — the round search button at the end of the row. Click it, or press /, and it opens into a search field. While it holds text, it stays as a small pill showing your search, with ✕ to clear it. Search matches test names, job and step names, workflow name or file, and repository. Every word you type must match, so web login finds login tests in the web repository.
  • Sort — the round sort button: Most wasted time (default), Last flaked, Highest failure rate, Most flaky commits, or Recently detected. Hover it to see the current sort; a dot shows when it is not the default.

  Counts next to each option, and on each tab, reflect the other active filters. Once a filter or search narrows the list, Showing X of Y and Reset filters appear on the same row.

• Shareable view — the selected tab, search, filters, and sort are kept in the page URL. Refreshing the page keeps them, and you can copy the URL to share the exact view with a teammate.

• Links to one item — links from alerts and pull request comments open the page on that item: Gitera switches to its tab, clears any filter that would hide it, highlights it, opens the job group a linked test belongs to, and expands Details. If the item is no longer listed (for example, it resolved long ago), the page says so. The highlight goes away as soon as you change the tab, filters, or sort.

What a row showsCopy Anchor Link

For Test rows, Details expands the three most recent failures of that test with the commit, a link to the exact run attempt on GitHub, the file the failure was reported in, and the captured error message — usually enough to start debugging without opening the log.

For Job rows, Details lists the five most recent failed attempts of the job in the last 14 days: when it failed, the commit, the run attempt, whether it failed or timed out, the step that failed, a link to the job log, and — when a later attempt of the same run passed — Passed on attempt N, the pass-on-retry evidence at a glance.

On the Free plan, or if the page cannot load its data, the summary cards and the list are hidden instead of showing zeros, so an empty page never reads as "no flaky items". Animations follow your system's reduce-motion setting.

Wasted time is the duration of failed runs on commits where the item flaked. Cost uses the same per-SKU list price as the Cost page — standard, larger, arm64, GPU, and ubuntu-slim runners each at their own rate, each job rounded up to the whole minute. Self-hosted runners and runners whose SKU is not known are counted as zero cost. Organization discounts and included minutes are not applied. For a flaky test, wasted time is the full duration of the job runs it failed in, since the whole job had to be re-run — so several flaky tests in the same job each show that job's time.

What to do with a flaky itemCopy Anchor Link

1. Open Details on a test or job and read the error, or follow its links to the run and job log on GitHub.
2. Fix the cause: shared state, timing, ordering, or an unreliable external service.
3. For a flaky job you cannot fix yet, choose Create retry rule (not offered for Consistently failing jobs, where retrying would not help). It opens Auto-retry with the repository, workflow, and job already filled in, one retry, and the failure conclusion. Review and save it — nothing is created until you do.
4. Dismiss items you have decided to live with. Gitera resolves fixed items on its own after 7 days without a flake.

Dismissing or restoring takes effect in the list right away, and the confirmation has an Undo button. Each item is dismissed or restored from its own card.

Alerts for new flaky itemsCopy Anchor Link

Configure alerts in Notifications → Flaky tests. Each rule has:

• Repositories — specific repositories, or all repositories when none are selected.
• Notify about — Jobs, Tests, or both.
• Destination — email addresses, Slack channels, or both. Slack channels come from Notifications; only channels of the current organization can be selected.
• An active switch to pause the rule without deleting it.

After each detection run, a rule sends one digest listing the items detected for the first time that match it — up to 10 items, then and N more. Each item links to its row on the Flaky tests page, next to the link to its last flaky run on GitHub. Items that are already active do not alert again, and a dismissed item does not alert during its cooldown. In Slack, test and job names are shown as plain text (links or mentions inside a name are not rendered), and very long names are shortened.

Known flaky on pull requestsCopy Anchor Link

When AI failure summary comments are enabled, a failure on a job that is currently an active flaky item gets a line on the pull request comment: Known flaky since DATE — flaked on N commits, with a link that opens the Flaky tests page on that job. Authors can tell at a glance whether to re-run or investigate.

Requirements and limitsCopy Anchor Link

• A paid plan (Basic, Pro, or Enterprise). On the Free plan the page shows an upgrade prompt, detection does not run, and failed tests are not recorded. See Plans & usage limits.
• The Gitera GitHub App installed on the repository, so run and job events arrive. See how Gitera works.
• Recorded test failures are kept for 30 days, then deleted automatically.
• No backfill of test failures. Failed tests are recorded as runs arrive, from the time the feature is active. Historical runs imported from GitHub contain only the latest attempt of each run, so pass on retry needs live runs; same commit and flip rate work on imported history.
• A test is inferred to have passed when its job passed. If your suite shards tests dynamically or skips tests on re-runs, a passing job does not always mean the test ran.

TroubleshootingCopy Anchor Link

RelatedCopy Anchor Link

• Failures & incidents — failures that are not flaky.
• Auto-retry — retries for flaky jobs.
• Notifications & alerts — Slack channels and email.