Poodle’s Reporters

           ^\               ARF!    |\|\
 /        //o__o     ARF!          ..    \       .
/\       /  __/           ARF!   o--     \\    / @)
\ \______\  /   ARF!   ARF!       v__///\\\\__/ @
 \         /        ARF!    ARF!    {           }
  \ \----\ \           ARF!          {  } \\\{  }
   \_\_   \_\_   ARF!     ARF!       <_|      <_|

Builtin Reporter Names

summary	Summary Reporter
not_found	Not Found Reporter
json	JSON Reporter
html	HTML Reporter

Summary Reporter

The summary reporter prints a list of basic statistics about the testing.

Example Report:

*** Results Summary ***
Testing found 50.0% of Mutants.
 - 10 mutant(s) were not found.
 - 10 mutant(s) caused trial to timeout.
 - 10 mutant(s) could not be tested due to an error.

Not Found Reporter

The “Not Found” Reporter prints a list of all mutations that were not found. It includes information about the Mutation, and any error messages generated by the testing tool.

Example Report:

*** Mutants Not Found ***

Mutant Trial Result: Mutant Not Found
Mutator: String
--- /home/runner/work/poodle/poodle/src/poodle/runners/command_line.py
+++ [Mutant] /home/runner/work/poodle/poodle/src/poodle/runners/command_line.py:74
@@ -71,7 +71,7 @@
     return MutantTrialResult(
         passed=True,
         reason_code=MutantTrialResult.RC_OTHER,
-        reason_desc=result.stdout.decode("utf-8", errors="replace")
+        reason_desc=result.stdout.decode("utf-8", errors='XXreplaceXX')
         + "\n"
         + result.stderr.decode("utf-8", errors="replace"),
     )

Options:

not_found_file

If specified, not found report is printed to specified file instead of sysout.

Default: None

poodle_config.py

reporter_opts = {
  "not_found_file":"mutation-testing-report.txt",
}

poodle.toml

[poodle.reporter_opts]
not_found_file = "mutation-testing-report.txt"

pyproject.toml

[tool.poodle.reporter_opts]
not_found_file = "mutation-testing-report.txt"

JSON Reporter

This reporter will create a file with a JSON dump of the Test Summary data, along with information about selected Mutants. This is most helpful when passing testing results to other processes, or if you’d like to query the results with tools like jq. The json data can also be used to create a mutation coverage badge.

Example Report

JSON Structure:

{
  "mutant_trials": [
    {
      "mutant": {
        "mutator_name": "",
        "lineno": 0,
        "col_offset": 0,
        "end_lineno": 0,
        "end_col_offset": 0,
        "text": "",
        "source_folder": "",
        "source_file": "",
        "unified_diff": ""
      },
      "result": {
        "found": true,
        "reason_code": "",
        "reason_desc": ""
      },
      "duration": 0.0
    }
  ],
  "summary": {
    "trials": 0,
    "tested": 0,
    "found": 0,
    "not_found": 0,
    "timeout": 0,
    "errors": 0,
    "success_rate": 0.0,
    "coverage_display": "0.00%"
  }
}  

Options:

json_include_summary

If False, the “summary” section of the json is excluded.

Default: True

poodle_config.py

reporter_opts = {
  "json_include_summary":False,
}

poodle.toml

[poodle.reporter_opts]
json_include_summary = false

pyproject.toml

[tool.poodle.reporter_opts]
json_include_summary = false

json_report_file

Specifies which file to use for storing the testing report json.

If value is the string “sysout”, json will be printed to sysout.

Default: mutation-testing-report.json

poodle_config.py

reporter_opts = {
  "json_report_file":"report.json",
}

poodle.toml

[poodle.reporter_opts]
json_report_file = "report.json"

pyproject.toml

[tool.poodle.reporter_opts]
json_report_file = "report.json"

json_report_found

If False, the “mutant_trials” list will NOT include trials where “found” is true

Default: False

poodle_config.py

reporter_opts = {
  "json_report_found":True,
}

poodle.toml

[poodle.reporter_opts]
json_report_found = true

pyproject.toml

[tool.poodle.reporter_opts]
json_report_found = true

json_report_not_found

If False, the “mutant_trials” list will NOT include trials where “found” is false

Default: True

poodle_config.py

reporter_opts = {
  "json_report_not_found":False,
}

poodle.toml

[poodle.reporter_opts]
json_report_not_found = false

pyproject.toml

[tool.poodle.reporter_opts]
json_report_not_found = false

HTML Reporter

This reporter creates a set of HTML and related files in a folder. These provide a summary of the mutation testing results, and statistics for each module tested. It also lists all the mutants that were not found. Finally, it creates pages for each module with the full source text and mutations applied to each line.

Example Report from Poodle version 1.2.0

Options:

report_folder

Folder where HTML report will be stored.

Default: “mutation_reports”

poodle_config.py

reporter_opts = {
  "html": {
    "report_folder": "mutation_reports",
  }
}

poodle.toml

[poodle.reporter_opts.html]
report_folder = "mutation_reports"

pyproject.toml

[tool.poodle.reporter_opts.html]
report_folder = "mutation_reports"

include_found_trials_on_index

The main (index) page of the Mutation Coverage Report, includes a list of mutations that were not found in each module. If this option is True, all found mutations will be included as well.

Default: False

poodle_config.py

reporter_opts = {
  "html": {
    "include_found_trials_on_index": True,
  }
}

poodle.toml

[poodle.reporter_opts.html]
include_found_trials_on_index = true

pyproject.toml

[tool.poodle.reporter_opts.html]
include_found_trials_on_index = true

include_found_trials_with_source

The HTML reporter generates a page for each module with source code, and all mutations. If this option is set to false, mutations that were found are excluded.

Default: True

poodle_config.py

reporter_opts = {
  "html": {
    "include_found_trials_with_source": False,
  }
}

poodle.toml

[poodle.reporter_opts.html]
include_found_trials_with_source = false

pyproject.toml

[tool.poodle.reporter_opts.html]
include_found_trials_with_source = false