Property Based Testing

In this chapter, we'll explore property-based testing - a powerful technique that generates test cases automatically. We'll cover:

• What is property-based testing?
• Introduction to Hypothesis
• Defining strategies for test data
• Finding edge cases automatically
• Combining with example-based tests
• When property-based testing shines

What is property-based testing?

Traditional (example-based) testing:

def test_reverse_string():
    assert reverse("hello") == "olleh"
    assert reverse("") == ""
    assert reverse("a") == "a"

Property-based testing:

from hypothesis import given
from hypothesis import strategies as st


@given(st.text())
def test_reverse_twice_is_original(s):
    assert reverse(reverse(s)) == s

Instead of specific examples, we define properties that should hold for any input.

Introduction to Hypothesis

Hypothesis is Python's premier property-based testing library:

pip install hypothesis

Basic usage

from hypothesis import given
from hypothesis import strategies as st


@given(st.integers())
def test_integer_addition_is_commutative(x):
    # Hypothesis generates many integers and tests this property
    assert x + 0 == x

Hypothesis will:

1. Generate many random integers
2. Run the test with each one
3. Report any failures with a minimal example

Strategies

Strategies define how to generate test data:

Built-in strategies

from hypothesis import strategies as st

st.integers()                    # Any integer
st.integers(min_value=0)         # Non-negative integers
st.integers(min_value=1, max_value=100)  # Range

st.floats()                      # Any float
st.floats(allow_nan=False)       # No NaN values

st.text()                        # Any Unicode string
st.text(min_size=1, max_size=10) # Length limits
st.text(alphabet="abc")          # Limited characters

st.booleans()                    # True or False

st.none()                        # Always None

st.lists(st.integers())          # List of integers
st.lists(st.text(), min_size=1)  # Non-empty list of strings

st.tuples(st.integers(), st.text())  # (int, str) tuples

st.dictionaries(st.text(), st.integers())  # Dict[str, int]

Combining strategies

# Either an integer or None
st.integers() | st.none()

# One of specific values
st.sampled_from(["red", "green", "blue"])

# Complex structure
st.fixed_dictionaries({
    "name": st.text(min_size=1),
    "age": st.integers(min_value=0, max_value=150),
    "email": st.emails(),
})

Testing properties

Reversibility

@given(st.text())
def test_encode_decode_roundtrip(s):
    encoded = base64_encode(s)
    decoded = base64_decode(encoded)
    assert decoded == s

Invariants

@given(st.lists(st.integers()))
def test_sorted_list_is_ordered(lst):
    result = sorted(lst)
    
    # Property: sorted list has same elements
    assert sorted(result) == sorted(lst)
    
    # Property: each element <= next element
    for i in range(len(result) - 1):
        assert result[i] <= result[i + 1]

Idempotence

@given(st.text())
def test_normalize_is_idempotent(s):
    once = normalize(s)
    twice = normalize(normalize(s))
    assert once == twice

Commutativity

@given(st.integers(), st.integers())
def test_addition_is_commutative(a, b):
    assert a + b == b + a

Finding edge cases

Hypothesis excels at finding edge cases you might miss:

from hypothesis import given
from hypothesis import strategies as st


def divide(a, b):
    return a / b


@given(st.integers(), st.integers())
def test_divide(a, b):
    result = divide(a, b)
    assert result * b == a

Hypothesis will find:

• b = 0 causes ZeroDivisionError
• Very large numbers may cause overflow
• Edge cases with negative numbers

Fixing with assume

from hypothesis import given, assume
from hypothesis import strategies as st


@given(st.integers(), st.integers())
def test_divide(a, b):
    assume(b != 0)  # Skip when b is 0
    result = divide(a, b)
    # Note: floating point may not be exact
    assert abs(result * b - a) < 0.0001

Custom strategies

Using @composite

from hypothesis import given
from hypothesis import strategies as st
from hypothesis.strategies import composite


@composite
def valid_user(draw):
    name = draw(st.text(min_size=1, max_size=50))
    age = draw(st.integers(min_value=0, max_value=150))
    email = draw(st.emails())
    return {"name": name, "age": age, "email": email}


@given(valid_user())
def test_user_creation(user_data):
    user = User(**user_data)
    assert user.name == user_data["name"]
    assert user.age == user_data["age"]

Filtering strategies

# Even integers only
even_integers = st.integers().filter(lambda x: x % 2 == 0)

# Or use assume in the test
@given(st.integers())
def test_even_numbers(n):
    assume(n % 2 == 0)
    assert n // 2 * 2 == n

Mapping strategies

# Generate positive integers as strings
positive_int_strings = st.integers(min_value=1).map(str)

@given(positive_int_strings)
def test_parse_positive(s):
    assert int(s) > 0

Hypothesis settings

Configure Hypothesis behavior:

from hypothesis import given, settings
from hypothesis import strategies as st


@settings(max_examples=500)  # Run more examples
@given(st.integers())
def test_with_more_examples(n):
    assert n == n


@settings(deadline=None)  # No time limit per example
@given(st.text())
def test_slow_operation(s):
    # Slow operation that might timeout
    process(s)

Profile settings

# conftest.py
from hypothesis import settings

# Faster for CI
settings.register_profile("ci", max_examples=100)

# Thorough for local
settings.register_profile("dev", max_examples=1000)

# Use with: pytest --hypothesis-profile=ci

Combining with example-based tests

Use @example for specific cases you always want tested:

from hypothesis import given, example
from hypothesis import strategies as st


@given(st.text())
@example("")           # Always test empty string
@example("hello")      # Always test this specific case
@example("🎉🎊")       # Always test emoji
def test_process_text(s):
    result = process(s)
    assert isinstance(result, str)

Stateful testing

Test sequences of operations:

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
from hypothesis import strategies as st


class SetMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.model = set()      # What we expect
        self.actual = MySet()   # What we're testing
    
    @rule(value=st.integers())
    def add(self, value):
        self.model.add(value)
        self.actual.add(value)
    
    @rule(value=st.integers())
    def remove(self, value):
        self.model.discard(value)
        self.actual.discard(value)
    
    @invariant()
    def sets_match(self):
        assert set(self.actual) == self.model


TestMySet = SetMachine.TestCase

Real-world example: JSON serialization

from hypothesis import given
from hypothesis import strategies as st
import json


# Strategy for JSON-serializable data
json_data = st.recursive(
    st.none() | st.booleans() | st.floats(allow_nan=False) | st.text(),
    lambda children: st.lists(children) | st.dictionaries(st.text(), children),
    max_leaves=20
)


@given(json_data)
def test_json_roundtrip(data):
    serialized = json.dumps(data)
    deserialized = json.loads(serialized)
    assert deserialized == data

When property-based testing shines

Good use cases

1. Parsers: parse(format(x)) == x
2. Serialization: deserialize(serialize(x)) == x
3. Data structures: Invariants always hold
4. Mathematical functions: Known properties
5. String manipulation: Reversibility, idempotence

Less suitable

1. UI tests: Hard to define properties
2. Integration tests: Too slow/complex
3. Tests that need specific values: Use example-based

Best practices

1. Start simple

# Start with basic properties
@given(st.lists(st.integers()))
def test_length_preserved(lst):
    result = my_sort(lst)
    assert len(result) == len(lst)

2. Combine with example-based tests

# Example-based for specific scenarios
def test_sort_empty():
    assert my_sort([]) == []


# Property-based for general correctness
@given(st.lists(st.integers()))
def test_sort_is_sorted(lst):
    result = my_sort(lst)
    assert all(result[i] <= result[i+1] for i in range(len(result)-1))

3. Document the property

@given(st.text())
def test_strip_removes_whitespace(s):
    """
    Property: strip() removes leading/trailing whitespace
    and the result contains no leading/trailing whitespace.
    """
    result = s.strip()
    assert not result.startswith((" ", "\t", "\n"))
    assert not result.endswith((" ", "\t", "\n"))

Wrapping up

We've covered:

• Property-based testing - Define properties, not examples
• Hypothesis - Python's property-based testing library
• Strategies - Generate test data
• Finding edge cases - Hypothesis finds bugs you'd miss
• Custom strategies - Build complex test data
• Stateful testing - Test sequences of operations

Key takeaways

1. Properties describe what should always be true
2. Hypothesis generates many test cases automatically
3. It finds edge cases you wouldn't think of
4. Combine with example-based tests for best coverage
5. Great for serialization, parsing, and data structures

Property-based testing is a powerful addition to your testing toolkit. It complements example-based testing by exploring the input space more thoroughly than you ever could manually.