Introduction to pytest: A Beginner's Guide

Welcome to our beginner's guide on pytest! pytest is a powerful and easy-to-use testing framework in Python. In this blog post, we'll introduce you to the essential features of pytest, using code examples and explanations.

Table of contents

How to install and run Pytest
Test outcomes in pytest
How to write Pytest tests (with a better example)
Testing a data store using pytest and fixtures
Conclusion

If you'd rather watch the YouTube video for this article, check it out below!

When we write code—let's say one function—it's easy to focus on just the one situation where we will use it. When that happens, we tend to forget about what will happen when we use the function in other circumstances, and also on how the function operates with the data available in our program.

What I mean to say (please bear with the super-simplified example) is we may write a function like this:

def divide(x, y):
    return x / y

And forget that the function may be called with y=0, which will result in an error.

Writing tests as we go along (or even before we write the code) can help put us in a mindset of "what do I want this function to do when given certain inputs?" and this moves from an implementation mindset into a more mathematical mindset.

In this article, I'll show you how to use pytest to write tests for your Python code!

How to install and run Pytest

To install pytest, just run:

pip install pytest

Remember that if you're using pytest in a project, you should first create a virtual environment. I also recommend working with pyenv so you can easily use multiple different Python versions in your computer.

Whenever you start a new project, I recommend using the latest Python version available (that is compatible with the libraries you plan to use) and always creating a virtual environment for the project.

Now that you've got your virtual environment created and pytest installed, let's write our first test! In a new file called test_simple.py (naming is important, should start with test_):

def test_simple_add():
    assert 1 + 1 == 2

Note the naming here is also important. Pytest tests are functions, and the functions names should start with test_.

Pytest uses Python assertions, so it's super simple to get started. In Python, 1 + 1 == 2 evaluates to the boolean value True, so assert True will pass and therefore your test will pass. Doing assert False will raise an AssertionError which would cause the test to fail. Simple, right?

To run all your tests (just 1, for now), activate your virtual environment and then run:

pytest

You should see something like this output (among potentially a few other things):

platform darwin -- Python 3.11.0, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/josesalvatierra/Documents/Teclado/Blog/posts/2023/02-07-testing-with-pytest/project
collected 1 item                                                                 

test_simple.py .           [100%]

=== 1 passed in 0.00s ===

The main thing there is:

test_simple.py .           [100%]

This tells you that in the file test_simple.py there was 1 passing test (denoted by the .). The [100%] at the end tells us that 100% of the tests have ran.

If you have many files, the file names and the . (or other characters, if the test doesn't pass) will appear as the tests run. Similarly the percentages will add up over a few (or many) lines until you reach 100%.

If you want to run a specific file because you have multiple files, you could always run:

pytest test_simple.py

And if you want to run a specific test, you can do so like this:

pytest -k test_simple_add

The -k flag must be my most-used one. Any tests that contain the string test_simple_add in their function name will run. At the moment it's just one, but you can see how you could do for example pytest -k register to run all tests related to user registration, for example.

Let's write another test:

def test_simple_add():
    assert 1 + 1 == 2


def test_failing():  # will fail
    assert 'a' == 'b'

This new test will fail. When you run pytest, you'll see this output (I'll skip showing you the printed boilerplate from now on):

test_simple.py .F                                             [100%]

===== FAILURES =====
_____ test_failing _____

    def test_failing():
>       assert 'a' == 'b'
E       AssertionError: assert 'a' == 'b'
E         - b
E         + a

test_simple.py:6: AssertionError
===== short test summary info =====
FAILED test_simple.py::test_failing - AssertionError: assert 'a' == 'b'

Now this is quite handy! You can see that in test_simple.py there is one passing test and one failing test with .F.

At the end of the test run, pytest shows you which tests have failed, and when there is an assertion that failed, it shows you the values and what the expectation was. Here we had 'b', but the assertion expected to see 'a' on the right side.

Test outcomes in pytest

This actually brings us nicely to the different test outcomes. So far we've seen:

PASSED, denoted by .
FAILED, denoted by F

But there are others too:

SKIPPED (s)
XPASS (X), a.k.a. "unexpectedly passed"
ERROR (E)

Expected failure in pytest with xfail

Let's say that for whatever reason, we want to keep our second test but we want to tell pytest that it should fail. That is, that "success" for that test means "it fails".

We can mark it with a decorator:

import pytest


def test_simple_add():
    assert 1 + 1 == 2


@pytest.mark.xfail
def test_failing():
    assert 'a' == 'b'

Running these tests will now show us this:

test_simple.py .x

And all tests passed! The second test passed, because we said we expected it to fail. Confusing, right? It's not used very often, but it's there if you need it.

Note that the outcome was x, and not X. That's because we said the test should fail, and it did. It didn't "unexpectedly pass", it "expectedly failed".

If we change it to this:

import pytest


def test_simple_add():
    assert 1 + 1 == 2


@pytest.mark.xfail
def test_failing():
    assert 'a' == 'a'  # this will pass

Then the outcome will be:

test_simple.py .X

Pytest now says 1 passed, 1 xpassed.

How to skip tests with pytest

Using similar syntax, we can skip tests:

import pytest


def test_simple_add():
    assert 1 + 1 == 2


@pytest.mark.skip
def test_failing():
    assert 'a' == 'b'

And now the output:

test_simple.py .s

Again, all tests pass. You shouldn't skip tests very often, but sometimes there isn't anything you can do about a test failure (temporarily), so you can skip it.

You can also skip tests only under certain conditions. For example, if a certain test doesn't work on Mac computers because the library you're using isn't supported there, you can use skipif:

import os

import pytest


def test_simple_add():
    assert 1 + 1 == 2


@pytest.mark.skipif(os.name == 'posix', reason='does not run on mac')
def test_failing():
    assert 'a' == 'b'

Again, test is skipped on Mac computers, but not on Windows (where it would fail, since 'a' == 'b' always fails).

The error test outcome in pytest

The final test outcome, error, is pretty self-explanatory. If there's an error such as ValueError, NameError, or any other error during the runtime of the test that isn't AssertionError, the test will fail with ERROR (shown as E in the console).

How to write Pytest tests (with a better example)

Let's move on to a different example for the rest of this article. Let's say you are writing a class to represent replies in a forum thread. Something like this (in a file called reply.py):

import datetime


class Reply:
    def __init__(self, body: str, user: str, created: datetime.datetime) -> None:
        self.body = body
        self.user = user
        self.created = created
    
    def __repr__(self) -> str:
        return f"<Reply from '{self.user}' on '{self.created.strftime('%Y-%m-%d %H:%M')}'>"

    def __eq__(self, __value: object) -> bool:
        if not isinstance(__value, Reply):
            return False
        return (
            self.body == __value.body and
            self.user == __value.user and
            self.created == __value.created
        )

Now let's write some tests for this!

First we need to think about what different tests we may want to write:

Test that creating a Reply object works (testing __init__).
Test that calling repr(reply) works correctly (testing __repr__).
Test __eq__:
- Two Reply objects with the same data are equal when doing reply1 == reply2.
- Two Reply objects with different data aren't equal.
- A Reply object and something else (like a string) aren't equal.

The Given-When-Then structure for tests

Create a file test_reply.py for the next few tests.

First, we'd add our imports:

import datetime

from reply import Reply

Then let's test that we can create a Reply object:

def test_init():
    # Given
    body = "Hello, World!"
    user = "user1"
    created = datetime.datetime.now()

    # When
    reply = Reply(body, user, created)

    # Then
    assert reply.body == body
    assert reply.user == user
    assert reply.created == created

Here you can see the Given-When-Then structure of tests coming into play. It's a simple but effective way to structure your tests:

Start by defining the data your test will use ("given that this data is available...")
Then write the actual test ("when this happens...")
Finally, check that the result is what you expect ("then the result is...")

How to write tests for functions that process data

In our __repr__ method, objects have access to self.created, and they will transform it to a string to be displayed.

We may feel like doing something like this is warranted:

def test_repr():
    body = "Hello, World!"
    user = "user1"
    created = datetime.datetime.now()

    reply = Reply(body, user, created)
    expected_repr = f"<Reply from '{user}' on '{created.strftime('%Y-%m-%d %H:%M')'>"

    assert repr(reply) == expected_repr

But this would be a bit of a mistake, because the code in the Reply objects calls strftime, and so does our testing code. They're both doing the same thing, so you aren't really testing anything.

Instead, it's better to actually write the expected value in the test. It also makes it more readable!

def test_repr():
    body = "Hello, World!"
    user = "user1"
    created = datetime.datetime(2023, 1, 1, 0, 0, 0)

    reply = Reply(body, user, created)
    expected_repr = f"<Reply from '{user}' on '2023-01-01 00:00'>"

    assert repr(reply) == expected_repr

By the way, I should mention that if your tests fail and you're not sure why, running Pytest with the -v flag can be very helpful. It gives you a bit more information about the expected and actual assertion values.

pytest -v

Testing for expected exceptions

What happens when a user passes something that isn't a datetime object to a Reply?

Well, nothing until __repr__ is called. Let's write a test for that.

Here we will use an assertion helped, pytest.raises, to say that an exception should be raised:

def test_repr_without_date():
    body = "Hello, World!"
    user = "user1"
    created = "2021-02-07 12:00"

    reply = Reply(body, user, created)

    with pytest.raises(AttributeError):
        assert repr(reply) == f"<Reply from '{user}' on '2021-02-07 12:00'>"

Note that with pytest.raises(AttributeError) says that when we call repr(reply), an AttributeError should be raised because we can't call created.strftime() (since created isn't a datetime object).

There are other helpers that are worth knowing about. For example, pytest.approx(), which allows you to do things like:

import pytest

def test_almost_equal():
    x = 5
    y = 5.01
    assert x == pytest.approx(y, 0.1)  # True
    assert x == pytest.approx(y, 0.01)  # True
    assert x == pytest.approx(y, 0.001)  # False

This one checks that the two values are within an acceptable range of each other, given by the second argument. Handy, especially when comparing floats (did you know most programming languages don't represent floats very well in some cases?).

Testing the equality of two objects

To write our tests for testing equality, we do something similar to what we've been doing! First make variables for the values, then create the objects, and finally compare them:

def test_eq_same_values():
    body = "Hello, World!"
    user = "user1"
    created = datetime.datetime.now()

    reply1 = Reply(body, user, created)
    reply2 = Reply(body, user, created)

    assert reply1 == reply2


def test_eq_different_values():
    body1 = "Hello, World!"
    user1 = "user1"
    created1 = datetime.datetime.now()

    body2 = "Goodbye, World!"
    user2 = "user2"
    created2 = created1 + datetime.timedelta(minutes=1)

    reply1 = Reply(body1, user1, created1)
    reply2 = Reply(body2, user2, created2)

    assert reply1 != reply2


def test_eq_different_types():
    body = "Hello, World!"
    user = "user1"
    created = datetime.datetime.now()

    reply = Reply(body, user, created)
    non_reply = "Not a reply object"

    assert reply != non_reply

Testing a data store using pytest and fixtures

In order to showcase what "fixtures" are, let's add another class to our system:

from reply import Reply


class Thread:
    def __init__(self, title: str, user: str, replies: list[Reply] = None) -> None:
        self.title = title
        self.user = user
        self.replies = replies or []
    
    def __repr__(self) -> str:
        return f"<Thread '{self.title}' by '{self.user}'>"
    
    def reply(self, reply: Reply) -> None:
        if reply.created > self.replies[-1].created:
            raise ValueError("New reply is older than the last reply")
        self.replies.append(reply)

Here we've got a thread, with a title, the user who created the thread, and a list of replies.

It has one particularly interesting method, reply(), which takes in a Reply object and appends it to the replies list. But it only does so if the new reply is not older than the last reply in the thread. Or does it? I've introduced a bug in there to show you how easy it is to make a mistake here. Our testing will help us realise and fix the mistake!

Our first tests are fairly straightforward and introduce nothing new:

from thread import Thread
from reply import Reply

import datetime

def test_init():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime.now()),
        Reply("Goodbye, World!", "user2", datetime.datetime.now()),
    ]

    thread = Thread(title, user, replies)

    assert thread.title == title
    assert thread.user == user
    assert thread.replies == replies


def test_repr():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime.now()),
        Reply("Goodbye, World!", "user2", datetime.datetime.now()),
    ]

    thread = Thread(title, user, replies)

    assert repr(thread) == f"<Thread '{title}' by '{user}'>"

But now let's test the reply method:

def test_reply():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]

    thread = Thread(title, user, replies)
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))

    thread.reply(new_reply)

    assert new_reply in thread.replies

If we run this (which by the way, you should be running your tests continuously, certainly after every new test, but preferably after every relevant code change)...

self = <Thread 'Hello, World!' by 'user1'>, reply = <Reply from 'user1' on '2023-01-03 00:00'>

    def reply(self, reply: Reply) -> None:
        if reply.created > self.replies[-1].created:
>           raise ValueError("New reply is older than the last reply")
E           ValueError: New reply is older than the last reply

simple_tests/thread.py:15: ValueError
===== short test summary info =====
FAILED simple_tests/test_thread.py::test_reply - ValueError: New reply is older than the last reply

How could this be? Our replies are correct:

2023-03-01
2023-03-02
2023-03-03

The first thing would be to check that the test is correct. But it seems it is, so maybe we've introduced a bug in the code!

After looking through it, you'll be able to see that we've got the > the wrong way round. It should be <:

if reply.created < self.replies[-1].created:

Changing that and re-running the tests shows us they pass. Excellent!

Now let's write a test that should fail:

def test_reply_more_recent():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]

    thread = Thread(title, user, replies)
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))

    thread.reply(new_reply)

This one fails, so let's add pytest.raises(ValueError) at the end:

# At top of file
import pytest


def test_reply_more_recent():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]

    thread = Thread(title, user, replies)
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))

    with pytest.raises(ValueError):
        thread.reply(new_reply)

Using fixtures with pytest

You may have noticed that all tests start the same way, defining the title, user, and replies of our thread. We can extract them to a variable (but it won't work):

import datetime

import pytest
from reply import Reply
from thread import Thread

title = "Hello, World!"
user = "user1"
replies = [
    Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
    Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)


def test_init():
    assert thread.title == title
    assert thread.user == user
    assert thread.replies == replies


def test_repr():
    assert repr(thread) == f"<Thread '{title}' by '{user}'>"


def test_reply():
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))

    thread.reply(new_reply)

    assert new_reply in thread.replies


def test_reply_more_recent():
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))

    with pytest.raises(ValueError):
        thread.reply(new_reply)

Here the more experienced among you will notice that we've done something taboo! We've got a global variable, thread, and in our tests we're calling thread.reply(), which modifies the global variable.

This is a recipe for disaster because now the order in which our tests runs matters. You never want your tests to depend on each other.

Think about it, if test_reply runs before test_init, the latter will fail because the thread will have 3 replies in it instead of the expected 2.

So instead, we need to create a variable that is unique to each test. We can do that with a fixture:

@pytest.fixture(scope="function")
def thread():
    """This fixture returns a Thread object"""
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]
    return Thread(title, user, replies)

Now each test can gain access to this variable using dependency injection. This is just a fancy way to say: add thread as a parameter to the test, and Pytest will automatically call the thread() function and give the test its value when the test runs.

So let's modify our tests so they have a thread parameter (note I've not used this fixture in test_init):

import datetime

import pytest
from reply import Reply
from thread import Thread


@pytest.fixture
def thread():
    """This fixture returns a Thread object"""
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]
    return Thread(title, user, replies)


def test_init():
    title = "Hello, World!"
    user = "user1"
    replies = [
        Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
        Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
    ]

    thread = Thread(title, user, replies)
    
    assert thread.title == "Hello, World!"
    assert thread.user == "user1"
    assert thread.replies == replies


def test_repr(thread):
    assert assert repr(thread) == "<Thread 'Hello, World!' by 'user1'>"


def test_reply(thread):
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))

    thread.reply(new_reply)

    assert new_reply in thread.replies


def test_reply_more_recent(thread):
    new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))

    with pytest.raises(ValueError):
        thread.reply(new_reply)

So this helps us answer the question: "what are fixtures?"

Fixtures are functions that allow us to reuse data in our tests, and can be called once per test by Pytest.

Note that scope="function" can take other values! If you want the same fixture value to be reused in the entire file, you can say scope="module". If you want it to be reused in the entire test suite, you can use scope="session". There's also scope="class", but that only becomes relevant when you use an object-oriented approach to writing your tests, which is something I almost never do.

By default, fixtures can only be used in the file where they are defined. If you want to share a fixture between different test files, you can put the fixtures inside a file called conftest.py. Then the fixtures will be visible in any test file in the folder that contains conftest.py, or any sub-folder.

Setup and teardown in Pytest fixtures

Every fixture in Pytest can run some code before and after returning the value used in the tests.

Here's a non-useful example of how this works:

@pytest.fixture
def my_fixture():
    print("Hello, fixture!")
    yield 42
    print("Bye, fixture!")


def test_nothing(my_fixture):
    print("In the test!")
    print(f"Value from fixture: {my_fixture}")

Running this, you'll see this output is captured:

Hello, fixture!
In the test!
Value from fixture: 42
Bye, fixture!

Obviously, that example isn't very useful, but I'm sure you can see when it may be useful:

If your tests need to interact with a file, you could use the setup and teardown functionality to open and close the file before and after giving it to your tests.
If your tests use a database, you can use this to connect to the database and close the connection when you're done.

How to use multiple fixtures in a Pytest test

Using multiple fixtures is easy as pie! Just add multiple parameters to the tests, and Pytest will make sure the fixtures are called and the values passed:

def test_with_two_fixtures(fixture_one, fixture_two):
    pass

Tracing fixture execution and finding where fixtures are defined

When you execute the pytest command to run your tests, you can pass the --fixtures flag and it will show you all the fixtures that are available for your test functions to use. This includes fixtures Pytest provides, as well as your own:

cache -- .venv/lib/python3.11/site-packages/_pytest/cacheprovider.py:509
    Return a cache object that can persist state between testing sessions.

capsys -- .venv/lib/python3.11/site-packages/_pytest/capture.py:905
    Enable text capturing of writes to ``sys.stdout`` and ``sys.stderr``.

capsysbinary -- .venv/lib/python3.11/site-packages/_pytest/capture.py:933
    Enable bytes capturing of writes to ``sys.stdout`` and ``sys.stderr``.

... (there are a few)

----- fixtures defined from test_thread -----
thread -- simple_tests/test_thread.py:9
    This is a Thread object

If you want to see what fixtures each test used, you can use --fixtures-per-test:

$ pytest --fixtures-per-test -k test_thread
...
----- fixtures used by test_repr -----
----- (simple_tests/test_thread.py:36) -----
thread -- simple_tests/test_thread.py:9
    This fixture returns a Thread object

----- fixtures used by test_reply -----
----- (simple_tests/test_thread.py:40) -----
thread -- simple_tests/test_thread.py:9
    This fixture returns a Thread object

----- fixtures used by test_reply_more_recent -----
----- (simple_tests/test_thread.py:48) -----
thread -- simple_tests/test_thread.py:9
    This fixture returns a Thread object

Conclusion

In this beginner's guide to pytest, we've covered the basics of how to write tests using pytest, including assertions, fixtures, and selecting tests to run. We've also covered some advanced features like fixture scopes and sharing fixtures among multiple files using conftest.py.

Writing tests is an important part of software development, and pytest is a great tool for making testing in Python easy and effective. With the knowledge you've gained from this guide, you should be well on your way to writing better tests and building more reliable software!

If you want to learn more about Python generally, consider enrolling in our Complete Python Course! This comprehensive course teaches from the basics to the advanced, covering topics such as object-oriented programming, async development, GUI programming, and more.

For further reading, I strongly recommend the official documentation. Brian Okken has also written an excellent book:

The official pytest documentation
Python Testing with pytest by Brian Okken, an excellent book that covers everything you could want to know about Pytest

That's all for today! Thank you for joining me, I hope you've enjoyed the read, and I'll see you next time.