JSON and JUnit Reports in Cucumber

What Are JSON and JUnit Reports?

JSON and JUnit reports are machine-readable report formats generated after a Cucumber test execution. They are not mainly designed for a tester to open and read like a normal web page. Instead, they are designed for tools, pipelines, dashboards, reporting libraries, and build servers that need structured execution data. Cucumber can run feature files and print output to the console, but console output is temporary and difficult to process. JSON and JUnit XML reports convert the execution result into files that other systems can understand.

A JSON report stores detailed execution information in a structured JSON format. It can contain feature names, scenario names, step text, step status, duration, tags, errors, embeddings, and other metadata. Because JSON is easy for programs to parse, many reporting tools use it as a source for richer dashboards. A JUnit XML report stores results in an XML format understood by most CI/CD tools. Jenkins, Azure DevOps, Bamboo, TeamCity, GitHub Actions integrations, and many build systems can read JUnit-style XML and show test results directly in the pipeline.

In simple terms, the JSON report is useful when another reporting or analytics tool needs detailed Cucumber execution data, while the JUnit XML report is useful when a CI/CD server needs to understand whether tests passed, failed, or were skipped. A real automation framework often generates both formats in the same run because they serve different purposes.

Why JSON and JUnit Reports Are Important

Enterprise automation does not stop when tests execute. The value of automation comes from feedback, and feedback must reach the right people quickly. A local tester may watch the console, but a larger team needs results inside build tools, release dashboards, email notifications, defect triage workflows, and historical reports. JSON and JUnit reports make this possible by turning Cucumber execution into reusable data.

CI/CD pipelines depend heavily on machine-readable output. If a Jenkins job runs a Cucumber suite, Jenkins needs a way to know how many scenarios passed, how many failed, and which test cases caused the build to fail. A JUnit XML report gives Jenkins that information in a standard structure. Without it, the build may show only that a command failed, which is not enough for practical investigation.

JSON reports are equally important, but for a different reason. They preserve richer Cucumber-specific details. Tools such as advanced report generators, custom dashboards, analytics scripts, and report-merging utilities can consume JSON and produce more useful views. If a team wants trends by tag, module, feature, browser, environment, or scenario, JSON output gives them a strong base for that analysis.

Reporting Flow

The reporting flow begins before execution starts. The automation engineer configures reporting plugins in the Cucumber runner, Cucumber properties, Maven command, Gradle task, or framework configuration. When the test suite runs, Cucumber reads feature files, matches steps to step definitions, executes hooks, runs the scenario logic, records status, and passes the result data to configured plugins. Those plugins write files into the configured output directory.

Feature Files
  -> Runner
  -> Cucumber Execution
  -> JSON Report
  -> JUnit XML Report
  -> CI/CD
  -> Dashboard

This flow matters because report generation is not separate from test execution. If the runner is configured incorrectly, reports may not be produced even if scenarios run successfully. If output paths are wrong, reports may be generated in a folder that the pipeline never archives. If parallel execution overwrites files, only partial results may remain. Reporting should therefore be treated as part of framework design, not as an afterthought.

Built-In Report Types

Cucumber supports multiple output formats, and each format has a different audience. Pretty output is useful in the console. HTML output is useful for quick manual review in a browser. JSON output is useful for report generation and custom processing. JUnit XML output is useful for CI/CD build result publishing. Message or NDJSON output is useful for newer integrations that need event-based execution data.

Cucumber Reports
  |-- Pretty
  |-- HTML
  |-- JSON
  |-- JUnit XML
  |-- Message / NDJSON

Beginners sometimes ask which report format is best. The better answer is that no single format is best for every need. A local tester may prefer HTML. A build server may require JUnit XML. A reporting library may need JSON. An enterprise team may generate several outputs together and let each consumer use the file it understands best.

Plugin Configuration

In many Cucumber JVM projects, reports are configured through the runner class using @CucumberOptions. A single execution can generate multiple reports by listing multiple plugins. The output path usually points to the Maven target directory because target is the standard place for generated build artifacts.

@CucumberOptions(
  plugin = {
    "pretty",
    "html:target/cucumber-report.html",
    "json:target/cucumber.json",
    "junit:target/cucumber.xml"
  }
)

This configuration tells Cucumber to print readable console output, create an HTML report, create a JSON report, and create a JUnit XML report. The framework does not need separate executions for each output. Cucumber can produce all configured reports from the same scenario execution, which keeps results consistent across formats.

JSON Report

The JSON report is generated by using the json plugin. The plugin value includes the report type and output path. A common configuration is json:target/cucumber.json. After execution, Cucumber writes a JSON file at that location.

plugin = {
  "json:target/cucumber.json"
}

target/
  cucumber.json

This file is structured data. It is not meant to be pretty documentation for business users. It is meant to be input for another tool. A reporting library can read it and build charts. A custom dashboard can read it and store historical results. A merge tool can combine multiple JSON files from parallel executions. Because of this flexibility, JSON is one of the most important Cucumber reporting outputs in automation frameworks.

JSON Report Contents

A JSON report can store execution details from the feature level down to the step level. It may include feature information, scenario information, background steps, tags, step results, durations, error messages, hooks, embeddings, and metadata depending on version and configuration. The exact structure can vary across Cucumber versions, but the purpose remains the same: preserve detailed execution results in a structured form.

Feature
  -> Scenario
  -> Step
  -> Status
  -> Duration
  -> Error Message
  -> Tags

A simplified JSON example may look small, but real Cucumber JSON output is more detailed. A reporting tool needs enough information to recreate the execution hierarchy, identify failures, calculate timing, group scenarios by tags, and display errors. That detail is why JSON is more useful than a simple pass/fail text file.

[
  {
    "feature": "Login",
    "scenario": "Valid Login",
    "status": "passed"
  }
]

JSON Report Usage

JSON reports are commonly used by reporting tools such as Allure adapters, Extent report integrations, custom report generators, and internal dashboards. They can also be used for analytics. For example, a team may parse JSON reports across builds to identify the slowest scenarios, most unstable tags, frequently failing modules, or scenarios that fail only in one browser. This kind of analysis is not practical from console logs.

JSON is also useful when test execution is split. A large regression suite may run across multiple machines, browsers, or modules. Each execution can generate its own JSON file. Later, a merge process can combine those files into a single report source. This is one of the reasons JSON is popular in parallel automation frameworks.

JUnit XML Report

The JUnit XML report is generated using the junit plugin. A common configuration is junit:target/cucumber.xml. After execution, Cucumber writes an XML file that follows a structure understood by build servers and CI/CD tools.

plugin = {
  "junit:target/cucumber.xml"
}

target/
  cucumber.xml

The name JUnit can be confusing for beginners. It does not mean the tests must be written as pure JUnit unit tests. In this context, JUnit XML refers to a widely supported test result format. Many tools know how to parse it because it has become a standard way to publish test results in automation pipelines.

JUnit XML Contents

A JUnit XML report usually contains test suites, test cases, failures, skipped tests, execution time, and error information. The details are presented in a format that CI tools can parse quickly. Build servers can then show total tests, failed tests, skipped tests, duration, trends, and build health.

Test Suite
  -> Test Cases
  -> Passed
  -> Failed
  -> Skipped
  -> Execution Time

A simplified XML structure may look like this:

<testsuite>
  <testcase name="Valid Login">
  </testcase>
</testsuite>

Real reports include more information than this simplified example. When a scenario fails, the XML can include failure details that the CI tool displays in its test results section. This helps developers inspect failures without downloading every artifact manually.

JUnit XML Usage

JUnit XML is mainly used by CI/CD systems. Jenkins can publish JUnit XML files and display test result trends. Azure DevOps can publish test results from XML files and show them in the pipeline summary. GitHub Actions workflows can use additional actions to publish or annotate test failures. Bamboo and TeamCity also understand similar report formats. This makes JUnit XML the bridge between Cucumber execution and build visibility.

In a well-designed pipeline, the Cucumber command runs first, the XML file is generated, and the CI server publishes the report even if tests fail. This is important. If the pipeline stops immediately after a failed test command and never publishes the XML, the team loses structured failure information. CI configuration should preserve reports regardless of test outcome.

Jenkins Flow

In Jenkins, the usual pattern is to run the test command and then publish JUnit XML results from the target folder. The Jenkins job may run Maven, execute the Cucumber runner, generate target/cucumber.xml, and then use the JUnit publisher to read the XML file. Jenkins can then show failed test counts, trends, test case details, and links to failed tests.

Pipeline
  -> Run Maven Tests
  -> Generate cucumber.xml
  -> Publish JUnit Results
  -> Show Test Trend

Jenkins archiving should also include JSON, HTML, screenshots, logs, and other artifacts when available. JUnit XML gives Jenkins build-level understanding, but JSON and HTML may provide richer debugging details. The best Jenkins setup usually publishes XML for build status and archives HTML or advanced reports for investigation.

Azure DevOps Flow

Azure DevOps follows a similar idea. The pipeline runs tests, Cucumber generates XML, and the pipeline publishes the XML as test results. Azure DevOps can then display passed, failed, and skipped tests in the pipeline interface. This is useful for teams that rely on pipeline summaries instead of manually browsing build folders.

Azure Pipeline
  -> Execute Cucumber Tests
  -> Generate JUnit XML
  -> Publish Test Results
  -> Review Pipeline Summary

The same principle applies to most modern CI/CD tools. They may differ in syntax, but they all need a machine-readable result file. JUnit XML provides that standard result file. JSON provides additional data when the team wants richer custom reporting.

JSON vs XML Generation

JSON and XML reports can be generated in the same execution. There is no need to choose only one. The JSON file can feed reporting tools, while the XML file can feed the CI/CD server. This approach avoids duplicate test execution and ensures that every report format describes the same run.

@CucumberOptions(
  plugin = {
    "json:target/cucumber.json",
    "junit:target/cucumber.xml"
  }
)

Generating both is a practical default in serious automation frameworks. HTML may also be added for quick browser viewing. When all three formats are generated together, the team gets human-readable reports, machine-readable details, and CI-readable test status from the same execution.

Output Directory

The output directory should be predictable. Most Maven-based frameworks use the target folder. Gradle projects may use a build folder. The key is consistency. CI jobs, report publishers, and artifact archivers should know exactly where to find generated files.

target/
  cucumber-report.html
  cucumber.json
  cucumber.xml

A common mistake is writing reports to unclear locations or changing paths frequently. If one runner writes to target/cucumber.json and another writes to reports/result.json, publishing becomes confusing. If the CI job expects one path but the runner writes another, the report will appear missing even though Cucumber generated it correctly.

Report Consumers

A report consumer is any tool or person that uses the generated report. HTML is consumed by humans. JSON is consumed by reporting tools, dashboards, analytics scripts, and merge utilities. JUnit XML is consumed by CI/CD servers. Understanding the consumer helps decide which report formats to generate.

This distinction is important in interviews and real projects. Saying "Cucumber report" is too general. A strong answer explains which report format is being used, who consumes it, and why that format is suitable.

Why Not Use HTML Everywhere?

HTML reports are useful, but they are not ideal for every purpose. HTML is designed for visual display. A human can open it in a browser and read it, but a pipeline usually does not want to scrape an HTML page to understand test status. CI/CD tools need structured, predictable data. JUnit XML is much better for that purpose.

Similarly, advanced reporting tools often need raw execution details, tags, timings, and failure metadata. JSON is easier to parse and transform than HTML. This is why enterprise frameworks often generate HTML, JSON, and JUnit XML together. Each format supports a different layer of reporting.

Multiple Executions and Report Merging

Large automation suites are often split across modules, browsers, environments, or parallel workers. For example, one execution may run login scenarios, another may run payment scenarios, and another may run order scenarios. Each execution can create its own JSON file and XML file. The challenge is combining the results without overwriting data.

When parallel execution is used, each worker should write to a unique file name or folder. For example, one worker can write target/reports/json/cucumber-1.json and another can write target/reports/json/cucumber-2.json. A report generation step can then merge those JSON files into a single dashboard. If every worker writes to the same file, later executions may overwrite earlier results.

JUnit XML files can also be generated per module or per runner. CI tools often support file patterns such as target/**/*.xml. This allows the pipeline to publish all test result files together. Naming and folder strategy become important when execution grows beyond a single runner.

JSON Reports for Advanced Dashboards

JSON reports are often the foundation for advanced dashboards. A custom dashboard can parse Cucumber JSON files, store results in a database, and show trends across builds. It can identify slow scenarios, flaky scenarios, high-failure modules, and tag-level stability. It can also connect automation results with release quality metrics.

This is one reason JSON is valuable even when the team already has HTML reports. HTML shows one execution. JSON can become historical data. When stored over time, JSON results can answer larger questions such as which features are becoming unstable, which tags take the most time, and whether automation reliability is improving.

JUnit XML for Build Health

JUnit XML is especially useful for build health. CI tools can show whether the current build introduced test failures, whether a branch is stable, and whether failed tests are increasing over time. The XML format allows the pipeline to treat Cucumber scenarios as published test cases. This makes BDD automation visible in the same way unit tests or integration tests are visible.

Build health reporting should be simple and immediate. A developer should not need to download a ZIP file and manually inspect logs just to know which tests failed. JUnit XML helps surface that information directly in the pipeline. This makes failures easier to triage and makes automation feedback more useful to the whole team.

Report Archiving

Generated reports should be archived after every important execution. In CI/CD, reports are often generated inside temporary workspace folders that may be deleted after the build. If the files are not archived or published, the team may lose the evidence. At minimum, important builds should preserve JUnit XML, JSON reports, HTML reports, screenshots, logs, and configuration details.

Archived reports should be connected to build number, branch name, commit ID, environment, execution time, and suite type. A report without context is difficult to interpret later. For example, a failure in a QA environment may mean something different from a failure in staging. A failure in Chrome may mean something different from a failure in Firefox. Good report archiving includes enough context to make old results meaningful.

Common Mistakes

One common mistake is generating only HTML reports and expecting CI tools to understand them. HTML is good for humans, but CI tools usually need XML. Another mistake is generating JSON but never using it. If no reporting tool or dashboard consumes the JSON file, the team should either add a consumer or understand why the file is being generated.

A frequent problem in parallel execution is report overwriting. If multiple runners write to the same cucumber.json file, the final file may contain only one runner's output. Another mistake is storing generated reports in source directories and accidentally committing them. Reports should usually be generated under target or build output folders.

Teams also forget to publish reports when tests fail. In many pipelines, a failed test command stops the job before the report publishing step runs. The pipeline should be designed so report publishing happens even after failures. Otherwise, the most important reports disappear exactly when the team needs them most.

Best Practices

Generate multiple report formats from the same execution. Use HTML for manual review, JSON for advanced reporting, and JUnit XML for CI/CD publishing. Keep report paths stable. Store generated files under target or build directories. Use unique file names for parallel executions. Archive reports after every CI run that matters.

Keep feature names, scenario names, step names, and tags clean because reports reflect the quality of the underlying suite. A JSON report can contain all the right data and still be hard to use if scenarios are poorly named. A JUnit report can publish failures, but vague scenario names make failures difficult to understand. Reporting quality begins with readable Gherkin.

Mask sensitive data before attaching request payloads, response bodies, screenshots, or logs. Reports are often shared widely. They may include tokens, usernames, customer data, environment URLs, headers, or file paths if the framework is not careful. A good reporting strategy balances useful evidence with security and privacy.

Enterprise Reporting Architecture

In enterprise projects, reporting usually has multiple layers. The Cucumber runner generates JSON, XML, and HTML. The CI server publishes JUnit XML to show build health. The build archives HTML, screenshots, and logs for manual review. A reporting library or custom dashboard consumes JSON to create richer reports. Historical data may be stored in a database for trend analysis.

Cucumber Execution
  -> JSON
  -> Advanced Report / Dashboard

Cucumber Execution
  -> JUnit XML
  -> CI Test Results

Cucumber Execution
  -> HTML
  -> Human Review

This layered approach is practical because one report format cannot satisfy every need. Managers may want trend summaries. Testers may want failed-step evidence. Developers may want stack traces and logs. CI tools want structured XML. Dashboards want structured JSON. A mature framework gives each consumer the right output.

JSON vs JUnit XML

JSON and JUnit XML should not be treated as competitors. JSON is richer and more Cucumber-specific. JUnit XML is more standardized for CI test publishing. JSON is better when a tool needs detailed scenario, step, tag, and duration information. JUnit XML is better when the CI server needs to show test result status in its normal test reporting interface.

A strong framework often enables both. The JSON report supports detailed reporting and future analytics. The JUnit XML report supports immediate build visibility. Together, they make Cucumber automation easier to integrate into real delivery pipelines.

JSON vs HTML

JSON and HTML serve different audiences. HTML is for people. JSON is for tools. If a tester wants to quickly open a report and inspect scenario results, HTML is convenient. If a reporting library wants to build a chart, calculate statistics, or merge parallel results, JSON is more useful.

This difference is important because teams sometimes expect one report to do everything. HTML reports are easy to read but not ideal for automation processing. JSON reports are powerful for processing but not pleasant for manual review. Generating both gives the team flexibility without rerunning the suite.

JUnit XML vs HTML

JUnit XML and HTML also solve different problems. HTML helps a person review the execution hierarchy. JUnit XML helps a CI/CD system publish test results. A pipeline may use XML to mark tests as failed and display the failing test names, while the archived HTML report gives testers and developers a more readable breakdown.

In release pipelines, both can be valuable. The XML result can influence the build status and pipeline dashboard. The HTML report can be attached as evidence for QA review. When screenshots or detailed logs are available, the HTML or advanced report may become the main debugging artifact, while XML remains the main CI integration artifact.

Report Quality Depends on Good Gherkin

Reports are only as useful as the tests they describe. If feature files use vague names, report consumers will struggle. A scenario named Verify login functionality is less helpful than Successful login with valid credentials. A feature named Test Feature is less useful than Customer Account Authentication. Reporting does not fix poor BDD design; it exposes it.

Good Gherkin improves every report format. JSON consumers get meaningful names and tags. JUnit XML publishers show clear test case names. HTML reports become readable. Managers and developers can understand failures without asking the automation engineer what a scenario means. This is why reporting strategy and scenario design should be reviewed together.

Using JSON Reports for Flaky Test Analysis

Flaky tests pass sometimes and fail at other times without a clear product change. JSON reports are useful for detecting these patterns because they can be collected across many executions. A dashboard can compare scenario names and statuses over time to identify tests that frequently alternate between pass and fail. It can also compare duration trends to identify scenarios that are becoming slower or unstable.

This kind of analysis helps teams move beyond reacting to individual failures. If a scenario fails once, the team investigates that run. If the same scenario fails in 30 percent of builds, the team has an automation reliability problem. JSON data gives the team the raw material to find those patterns and prioritize stability work.

Publishing JUnit XML in CI Pipelines

Publishing JUnit XML should be a standard part of pipeline configuration. The test execution step creates the XML file. The publish step reads the file and displays results in the build system. The publish step should run even when tests fail, because failed tests are exactly when published results are most valuable.

In Jenkins, this usually means using a post-build or post-stage action to publish JUnit results. In Azure DevOps, it may mean using a publish test results task. In GitHub Actions, it may involve a reporting action or artifact upload step. The syntax changes by platform, but the principle remains the same: generate XML, publish it, and keep it visible.

Report File Naming for Parallel Runs

Parallel runs require careful file naming. If ten runners execute at the same time and all write to target/cucumber.json, the final file may be corrupted or overwritten. Each runner should write a unique file, such as cucumber-login.json, cucumber-payment.json, or cucumber-thread-1.json. The same principle applies to XML reports.

After execution, the framework or CI job can collect all files using a pattern. For JSON reports, a merge or report-generation step can combine them. For XML reports, CI tools can often publish multiple XML files directly. This makes reporting reliable even as execution scales across modules and workers.

Security and Privacy in Generated Reports

Reports can accidentally expose sensitive data. API automation may log request headers, tokens, passwords, customer identifiers, or response bodies. UI automation may attach screenshots containing personal information. Environment details may reveal internal URLs. Because reports are often stored as artifacts and shared widely, they must be treated as potentially sensitive documents.

A secure framework masks tokens, passwords, secret headers, and personal data before writing attachments. It avoids printing raw credentials in assertion messages. It separates public reporting from internal debugging artifacts when needed. Reporting should make failures easier to understand without leaking data that should not be shared.

Troubleshooting Missing JSON or XML Reports

If JSON or XML reports are missing, start with the runner configuration. Confirm that the plugin name is correct and that the output path exists or can be created. Then confirm that the runner being executed is the same runner where plugins are configured. In projects with multiple runners, it is common to update one runner and execute another by mistake.

Next, check whether the build cleaned the output folder after execution or whether the CI job is looking in the wrong path. Also check parallel execution. Multiple runners writing to the same report file can cause missing or incomplete output. Finally, confirm that the Cucumber version and runner style support the configuration approach being used. Some projects configure plugins through annotations, system properties, command-line options, or platform configuration files.

Keeping Machine-Readable Reports Trustworthy

A report must be trustworthy. If reports are sometimes missing, overwritten, incomplete, or attached to the wrong build, teams stop relying on them. Trust comes from predictable paths, consistent naming, stable pipeline publishing, and clear ownership. The team should know where reports are generated, where they are archived, and how long they are retained.

Trust also depends on clean test data and reliable scenario status. If scenarios fail because of unstable environments or poor waits, reports may become noisy. A reporting system can show the noise, but it cannot solve it by itself. Good automation engineering, stable environments, clean data, and clear reporting all work together.

Interview-Ready Summary

JSON and JUnit reports are important Cucumber reporting outputs used mainly by tools rather than humans. JSON reports store detailed execution data in a structured format and are commonly consumed by advanced reporting tools, dashboards, analytics systems, and merge utilities. JUnit XML reports follow a standard test result format and are commonly consumed by CI/CD tools such as Jenkins, Azure DevOps, Bamboo, and TeamCity.

In a real automation framework, HTML, JSON, and JUnit XML reports are often generated together. HTML helps manual review. JSON supports rich reporting and custom analysis. JUnit XML supports CI/CD build result publishing. A good reporting strategy defines stable output paths, archives reports after execution, avoids overwriting during parallel runs, masks sensitive data, and keeps scenario names meaningful.

Golden Rules

Generate JSON reports when reporting tools or dashboards need detailed Cucumber execution data. Generate JUnit XML reports when CI/CD systems need to publish test results. Generate both from the same execution when possible. Store reports in predictable build folders. Use unique file names for parallel execution. Publish reports even when tests fail. Archive reports with build, branch, environment, and execution context.

The practical takeaway is simple: JSON reports feed reporting intelligence, and JUnit XML reports feed CI/CD visibility. When both are configured correctly, Cucumber automation becomes easier to monitor, easier to debug, easier to integrate, and far more useful in real delivery pipelines.