Using Poodle#

 _   _
/(. .)\    )
  (*)____/|
  /       |
 /   |--\ |
(_)(_)  (_)

Getting Started#

The most important thing is to tell poodle where your source code is. By default, it will look for a folder named “src” or “lib”. But if the parent folder for your project is something else, you’ll need to tell poodle where to look.

Tip

In some systems it’s necessary to prefix poodle command like: python3 -m poodle

Project with “src” folder:

Example:

  • src

    • main.py

    • util.py

    • query

      • __init__.py

      • database.py

  • test

    • __init__.py

    • test_main.py

    • test_util.py

  • pyproject.toml

  • requirements.txt

Running with just poodle will find the folder.

Project with “sources” folder:

Example:

  • sources

    • main.py

    • util.py

    • query

      • __init__.py

      • database.py

  • tests

    • __init__.py

    • test_main.py

    • test_util.py

  • pyproject.toml

  • requirements.txt

In this case, need to specify source folder like: poodle sources

DO NOT specify the package folders like “sources/query” as a source. Poodle needs the folder that contains the package folder.

Project with multiple sources to mutate:

Example:

  • application

    • main.py

    • query

      • __init__.py

      • database.py

  • layer

    • util.py

  • tests

    • __init__.py

    • test_main.py

    • test_util.py

  • pyproject.toml

  • requirements.txt

You can specify multiple folders to mutate like: poodle application layer

DO NOT specify the package folders like “application/query” as a source. Poodle needs the folder that contains the package folder.

Flat project:

Example:

  • query

    • __init__.py

    • database.py

  • main.py

  • pyproject.toml

  • requirements.txt

  • test_main.py

  • test_util.py

  • util.py

In this case, specify the source as the current folder: poodle .

DO NOT specify the package folders like “query” as a source

Flat Package project:

Example:

  • app

    • __init__.py

    • main.py

    • util.py

    • query

      • __init__.py

      • database.py

  • test

    • __init__.py

    • test_main.py

    • test_util.py

  • pyproject.toml

  • requirements.txt

In this case, specify the source as the current folder: poodle .

DO NOT specify the package folders like “app” or “app/query” as a source. Poodle needs the folder that contains the package folder.

Configuration File#

Poodle accepts several types of configuration files.

First is you can create poodle_config.py module. This is the most flexible option especially if you are adding custom code.

Second is a key/value config file like pyproject.toml or poodle.toml. All options are listed in the Options page.

if you need to specify folders for your project, recommend putting that setting in your chosen config file with option source_folders

Start Small#

If you have a lot of modules to scan, it can be helpful to start by scanning one or two at a time. This can be accomplished with the --only flag.

poodle --only main.py --only database.py

Terminology#

Mutation Testing can introduce some confusing language. For example, we run the test suite and a test case failed. In mutation testing, we want to testing to fail, so the test suite passed. passed == failed?

So, in this application, I use the following terms whenever possible to help make things clearer

Mutation

An intentional bug added to the code. For example, changing x + y to x - y.

Trial

A run of the entire test suite. Usually to validate if the Test Suite can find the Mutation.

Passed

When a Test Suite or Test Case ends without an error.

Failure

When a Test Suite or Test Case ends with an error or non-successful return code.

Found

If a Trial of a Mutation results in a Failure, then the Mutation was Found.

Not Found

If a Trial of a Mutation results in a successful completion, then the Mutation was Not Found.

Timeout

If the time to run a Trial exceeds a reasonable limit, the Trial is reported as Timeout instead of Found or Not Found.

Errors

If a Trial results in an exception that is not normally expected from the Test Suite.

Whitelisting#

Whitelisting is used to prevent mutations from being tested. This can be helpful in cases where mutation can’t reasonably be tested. There are several methods to whitelist code in poodle.

Line Comments#

The best way to block a mutation on a specific line is to add a comment to the line.

x = y + 3  # nomut: Number

This prevents only the “Number” mutator from mutating the statement.

x = y + 3  # nomut: Number, BinOp

This prevents only the “Number” and “BinOp” mutators from mutating the statement.

x = y + 3  # nomut

or

x = y + 3  # nomut: all

or

x = y + 3  # pragma: no mutate

This prevents all mutators from mutating the statement.

x = y + 3  # nomut: start
x += 4
y *= 6  # nomut: end

or

x = y + 3  # nomut: on
x += 4
y *= 6  # nomut: off

This prevents all mutators from mutating all three of these statements.

Whitelisting Entire Files#

The best way to prevent mutation on an entire file is to add it to the file_filters option in your configuration file.

Disabling a Mutator#

You can also disable a mutator completely using the skip_mutators option in your configuration file.

Create a Badge#

Like Badges, heres how I created this one: Mutation Coverage

  1. https://shields.io/badges/dynamic-json-badge

  2. Enter the following data:

    • URL: Enter link to the RAW JSON report file that includes summary data.

    • Query: $.summary.coverage_display

    • Label: Mutation Coverage

    • Color: 3776b5

    • Logo: (optional)

data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTYuOTMzbW0iIGhlaWdodD0iMTMuNTNtbSIgdmVyc2lvbj0iMS4xIiB2aWV3Qm94PSIwIDAgMTYuOTMzIDEzLjUzIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjxnIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLXdpZHRoPSIxLjEyODYiPjxwYXRoIGQ9Im0zLjEwMDIgMTIuODA0Yy0xLjA2NjItMC41NzEzOC0xLjk3MzUtMS41MTctMi4yMjgyLTIuNzI5OC0wLjM3ODA3LTEuNTU4OS0wLjM1MTE1LTMuMTk0MS0wLjE0NDc0LTQuNzc1IDAuMjE3OTQtMS4zODUgMS4zNjQzLTIuMzQ1NCAyLjQxMjQtMy4xNTI3IDAuOTk3NzQtMC42NzIwMSAyLjEwMzMtMS4yNjM1IDMuMjg3Ny0xLjQ5OTUgMS40MjY3LTAuMTcxNTcgMy4xMzQ0LTAuMTY3ODcgNC4xNTg5IDEuMDA4NiAwLjQ5OTQxIDAuNTI4NTEgMC42OTI5NiAxLjI1NzkgMC42MjI4NSAxLjk2OS0wLjAwNDkgMC44OTcxNCAwLjMwMzkzIDEuOTA3OCAxLjE4IDIuMzE0MyAxLjE0MzkgMC42NTU2OCAyLjUzMDggMC41NTMxMSAzLjcyMTQgMS4wODA4IDAuNzE1OTQgMC43MzgzMi0wLjI1NTQ2IDEuNjMzOC0wLjUxMDYxIDIuMzY1LTAuMzM5MDEgMC45MzMyLTEuMjkzOCAxLjIzNzYtMi4xMzMxIDEuNTQ5Ni0xLjYzNDggMC42NDgzNC0zLjExNTkgMC45NjU3NS01LjIwNTcgMS40MTAzLTIuNzcxMyAwLjcwMjA4LTMuODU0MSAwLjc0ODExLTUuMTYwOSAwLjQ1OTI3eiIgZmlsbD0iIzM3NzZiNSIvPjxwYXRoIGQ9Im0xNS4wMTcgOS40ODYyYy0wLjY4NjM5IDAuMDcyMzA4LTEuNDAxNS0xLjIzNjQtMS4zNDc4LTEuOTc4NSAwLjIwODIxLTAuOTk3NDcgMC44NDk1NS0wLjI1MTkzIDEuMTI1NC0wLjM5OTQgMC4yNTMzNC0wLjEzNDMxIDAuNTEyMTYtMC4xNjYzMSAwLjc3ODk2LTAuMDQ5MzI0IDAuNjA1NzMgMC4zMzk2OCAwLjExNDY0IDEuMjg2Ny0wLjEzNzY2IDEuNzUzOHoiLz48ZWxsaXBzZSBjeD0iOS4zMDM2IiBjeT0iNi4zOTUiIHJ4PSIxLjM0ODEiIHJ5PSIuODA0MDEiLz48L2c+PHBhdGggZD0ibTQuNzc3OSA0LjA3YzAuNjk1ODggMC4zMDQzMiAwLjYwODk5IDAuMzQwMjYgMC41OTUyNiAwLjkxMjk2LTAuMDExNTI1IDEuNjg0MiAwLjY0ODUyIDMuMzU1OCAwLjYwMzg5IDUuMTAxMiAwLjAxNTY2MyAwLjc1NjMzLTAuMDY2MTc3IDAuODQ3NzMtMC4zOTg5NiAxLjc2NDYtMC43NDMyNiAwLjM0MjAyLTEuNDQwOCAwLjE1ODY0LTIuMjc4LTAuMDA1MzM4LTAuODczNDgtMC42MTc4NC0xLjAxODktMC43NTkzMi0xLjQxODQtMi4wMDYzLTAuMzI0MjUtMS44NjQzLTAuMzIwMzYtMy4yOTM3LTAuMDgyODQ2LTQuOTU5NiAwLjE4OTYxLTAuOTY0NTEgMi42MjMyLTAuOTYxNzQgMi45Nzg5LTAuODA3NTYiIGZpbGw9IiMzNzNjNDMiIHN0cm9rZT0iIzM3M2M0MyIgc3Ryb2tlLXdpZHRoPSIxLjEyODYiLz48L3N2Zz4=