Exhaustive Testing

Exhaustive testing is a testing or quality assurance approach in which all possible combinations of scenarios and use/test cases are used for testing.

In testing software, it is all the possible combinations of every type of input as well as every permutation and variation of how the input is carried out to ensure that everything works as expected.

Techopedia

Demo project distributed with Puppetry contains sandbox-exhaustive.json suite, which is a show-case of exhaustive testing. It points at the form on https://puppetry.app/demo/ page. Our goal is to seed the form with various data and check if it responds as intended on submission. Here we want to test if special (non-latin and emoji) character in First Name field are tolerated. In real-life application form data are sent to the server. The server stores them in the DB. We need to ensure the server sanitizes the payload and DB is set properly (e.g. MySQL default collation is utf8mb4_general_ci and doesn’t support extended 4-bytes characters such as Emoji).

On this example we use a snippet "Fill out the form". It types in the First Name field a value of FIRST_NAME template variable, which will be passed to the snippet from test case. The snippet also seeds other form fields, but our focus here is on First Name field.

A snippet to fill out the form

In this demo when user submits the form it show an Alert dialog with a message containing either OK or FAIL. So in the test case we start with page.closeDialog method to dismiss dialogs as they pop up. Then we call the snippet for every combination of input values and assert the dialog message is OK

A test case of exhaustive testing

When adding snippet references we set FIRST_NAME variable a value (latin, non-latin and emoji)

Passing data into snippet

As we run the test it seeds the First Name field with test data and checks the form feedback after every submit. So we penetrate the application under test to ensure both client and server side tolerate any kind of input.