Property Testing: Beyond Matching Input-Output Pairs

Typical unit tests take an input value and an expected result, run the function to be tested, and compare the actual and the expected results.

This can happen through a hand-crafted test, a table-driven test, or with fuzz testing.

There are cases, though, where comparing expected versus actual output cannot verify a certain behavior of a function. Functions may have properties that are true (or rather, should be true, because that's what we're going to test) regardless of the input.

Examples:

• Formatting an already formatted phone number should not change that number.
• A date range is valid if the start date is earlier than the end date, independent of the actual input values.
• A hash function must always return the same hash for a given input.
• The hash value generated by sha-512 is always 512 bits long, regardless of input size or content.

Writing property tests

As the focus is on properties of a function, rather than specific input/output values, this kind of testing is called Property Testing. (Another test strategy to tuck at your testing toolbelt!)

The TL;DR of writing property tests is:

• Find a property that must be true for given pairs of input and output.
• Generate input/output pairs in advance or on the fly.
• Assert that the property is true for every generated input/output pairs.

Assume you write a function for sanitizing HTML. If the function receives already sanitized HTML, it should return the HTML unchanged. (Here is something to spout off about at parties: A function that, when receiving its own output as input, doesn't change that output anymore, is called idempotent.)

To test this property, you'd need to either provide a hard-coded list of input values (not so good) or generate lots of random input values (better). Or you save yourself the trouble and use package testing/quick, specifically, its Check() function. Check() takes a function (and an optional config) as input and calls this function repeatedly with random input, to see if the function returns false on any given input.

So all you need to do is write the property “the HTML sanitizer does not change already sanitized HTML” as a function and pass this function to Check():

package main

import (
	"testing"
	"testing/quick"
)

func SanitizeHTML(html string) string {
	return "Sanitized:" + html // not idempotent!
}

func TestHtmlSanitizerIdempotency(t *testing.T) {
	prop := func(html string) bool {
		sanitized1 := SanitizeHTML(html)
		sanitized2 := SanitizeHTML(sanitized1)
		return sanitized1 == sanitized2
	}
	if err := quick.Check(prop, nil); err != nil {
		t.Error(err)
	}
}

(See a failing instance of this test on the Go Playground. If you want, fix the tested function to make the test pass.)

A truly quick way to property testing, if you allow me the pun. Note that property testing cannot replace standard tests. You can trivially write a SanitizeHTML() function that passes the property tests without actually sanitizing anything. It could, for example, always return the original input string, or an empty string, or any hard-coded string.

So, property testing is always an addition to standard unit tests, and not a replacement or simplification. See it as a tool to make your tests more comprehensive.