Welcome to our beginner's guide on pytest! pytest is a powerful and easy-to-use testing framework in Python. In this blog post, we'll introduce you to the essential features of pytest, using code examples and explanations.
Table of contents
If you'd rather watch the YouTube video for this article, check it out below!
When we write code—let's say one function—it's easy to focus on just the one situation where we will use it. When that happens, we tend to forget about what will happen when we use the function in other circumstances, and also on how the function operates with the data available in our program.
What I mean to say (please bear with the super-simplified example) is we may write a function like this:
def divide(x, y):
return x / y
And forget that the function may be called with y=0, which will result in an error.
Writing tests as we go along (or even before we write the code) can help put us in a mindset of "what do I want this function to do when given certain inputs?" and this moves from an implementation mindset into a more mathematical mindset.
In this article, I'll show you how to use pytest to write tests for your Python code!
How to install and run Pytest
To install pytest, just run:
pip install pytest
Remember that if you're using pytest in a project, you should first create a virtual environment. I also recommend working with pyenv so you can easily use multiple different Python versions in your computer.
Whenever you start a new project, I recommend using the latest Python version available (that is compatible with the libraries you plan to use) and always creating a virtual environment for the project.
Now that you've got your virtual environment created and pytest installed, let's write our first test! In a new file called test_simple.py (naming is important, should start with test_):
def test_simple_add():
assert 1 + 1 == 2
Note the naming here is also important. Pytest tests are functions, and the functions names should start with test_.
Pytest uses Python assertions, so it's super simple to get started. In Python, 1 + 1 == 2 evaluates to the boolean value True, so assert True will pass and therefore your test will pass. Doing assert False will raise an AssertionError which would cause the test to fail. Simple, right?
To run all your tests (just 1, for now), activate your virtual environment and then run:
pytest
You should see something like this output (among potentially a few other things):
platform darwin -- Python 3.11.0, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/josesalvatierra/Documents/Teclado/Blog/posts/2023/02-07-testing-with-pytest/project
collected 1 item
test_simple.py . [100%]
=== 1 passed in 0.00s ===
The main thing there is:
test_simple.py . [100%]
This tells you that in the file test_simple.py there was 1 passing test (denoted by the .). The [100%] at the end tells us that 100% of the tests have ran.
If you have many files, the file names and the . (or other characters, if the test doesn't pass) will appear as the tests run. Similarly the percentages will add up over a few (or many) lines until you reach 100%.
If you want to run a specific file because you have multiple files, you could always run:
pytest test_simple.py
And if you want to run a specific test, you can do so like this:
pytest -k test_simple_add
The -k flag must be my most-used one. Any tests that contain the string test_simple_add in their function name will run. At the moment it's just one, but you can see how you could do for example pytest -k register to run all tests related to user registration, for example.
Let's write another test:
def test_simple_add():
assert 1 + 1 == 2
def test_failing(): # will fail
assert 'a' == 'b'
This new test will fail. When you run pytest, you'll see this output (I'll skip showing you the printed boilerplate from now on):
test_simple.py .F [100%]
===== FAILURES =====
_____ test_failing _____
def test_failing():
> assert 'a' == 'b'
E AssertionError: assert 'a' == 'b'
E - b
E + a
test_simple.py:6: AssertionError
===== short test summary info =====
FAILED test_simple.py::test_failing - AssertionError: assert 'a' == 'b'
Now this is quite handy! You can see that in test_simple.py there is one passing test and one failing test with .F.
At the end of the test run, pytest shows you which tests have failed, and when there is an assertion that failed, it shows you the values and what the expectation was. Here we had 'b', but the assertion expected to see 'a' on the right side.
Test outcomes in pytest
This actually brings us nicely to the different test outcomes. So far we've seen:
PASSED, denoted by.FAILED, denoted byF
But there are others too:
SKIPPED (s)XPASS (X), a.k.a. "unexpectedly passed"ERROR (E)
Expected failure in pytest with xfail
Let's say that for whatever reason, we want to keep our second test but we want to tell pytest that it should fail. That is, that "success" for that test means "it fails".
We can mark it with a decorator:
import pytest
def test_simple_add():
assert 1 + 1 == 2
@pytest.mark.xfail
def test_failing():
assert 'a' == 'b'
Running these tests will now show us this:
test_simple.py .x
And all tests passed! The second test passed, because we said we expected it to fail. Confusing, right? It's not used very often, but it's there if you need it.
Note that the outcome was x, and not X. That's because we said the test should fail, and it did. It didn't "unexpectedly pass", it "expectedly failed".
If we change it to this:
import pytest
def test_simple_add():
assert 1 + 1 == 2
@pytest.mark.xfail
def test_failing():
assert 'a' == 'a' # this will pass
Then the outcome will be:
test_simple.py .X
Pytest now says 1 passed, 1 xpassed.
How to skip tests with pytest
Using similar syntax, we can skip tests:
import pytest
def test_simple_add():
assert 1 + 1 == 2
@pytest.mark.skip
def test_failing():
assert 'a' == 'b'
And now the output:
test_simple.py .s
Again, all tests pass. You shouldn't skip tests very often, but sometimes there isn't anything you can do about a test failure (temporarily), so you can skip it.
You can also skip tests only under certain conditions. For example, if a certain test doesn't work on Mac computers because the library you're using isn't supported there, you can use skipif:
import os
import pytest
def test_simple_add():
assert 1 + 1 == 2
@pytest.mark.skipif(os.name == 'posix', reason='does not run on mac')
def test_failing():
assert 'a' == 'b'
Again, test is skipped on Mac computers, but not on Windows (where it would fail, since 'a' == 'b' always fails).
The error test outcome in pytest
The final test outcome, error, is pretty self-explanatory. If there's an error such as ValueError, NameError, or any other error during the runtime of the test that isn't AssertionError, the test will fail with ERROR (shown as E in the console).
How to write Pytest tests (with a better example)
Let's move on to a different example for the rest of this article. Let's say you are writing a class to represent replies in a forum thread. Something like this (in a file called reply.py):
import datetime
class Reply:
def __init__(self, body: str, user: str, created: datetime.datetime) -> None:
self.body = body
self.user = user
self.created = created
def __repr__(self) -> str:
return f"<Reply from '{self.user}' on '{self.created.strftime('%Y-%m-%d %H:%M')}'>"
def __eq__(self, __value: object) -> bool:
if not isinstance(__value, Reply):
return False
return (
self.body == __value.body and
self.user == __value.user and
self.created == __value.created
)
Now let's write some tests for this!
First we need to think about what different tests we may want to write:
- Test that creating a
Replyobject works (testing__init__). - Test that calling
repr(reply)works correctly (testing__repr__). - Test
__eq__:- Two
Replyobjects with the same data are equal when doingreply1 == reply2. - Two
Replyobjects with different data aren't equal. - A
Replyobject and something else (like a string) aren't equal.
- Two
The Given-When-Then structure for tests
Create a file test_reply.py for the next few tests.
First, we'd add our imports:
import datetime
from reply import Reply
Then let's test that we can create a Reply object:
def test_init():
# Given
body = "Hello, World!"
user = "user1"
created = datetime.datetime.now()
# When
reply = Reply(body, user, created)
# Then
assert reply.body == body
assert reply.user == user
assert reply.created == created
Here you can see the Given-When-Then structure of tests coming into play. It's a simple but effective way to structure your tests:
- Start by defining the data your test will use ("given that this data is available...")
- Then write the actual test ("when this happens...")
- Finally, check that the result is what you expect ("then the result is...")
How to write tests for functions that process data
In our __repr__ method, objects have access to self.created, and they will transform it to a string to be displayed.
We may feel like doing something like this is warranted:
def test_repr():
body = "Hello, World!"
user = "user1"
created = datetime.datetime.now()
reply = Reply(body, user, created)
expected_repr = f"<Reply from '{user}' on '{created.strftime('%Y-%m-%d %H:%M')'>"
assert repr(reply) == expected_repr
But this would be a bit of a mistake, because the code in the Reply objects calls strftime, and so does our testing code. They're both doing the same thing, so you aren't really testing anything.
Instead, it's better to actually write the expected value in the test. It also makes it more readable!
def test_repr():
body = "Hello, World!"
user = "user1"
created = datetime.datetime(2023, 1, 1, 0, 0, 0)
reply = Reply(body, user, created)
expected_repr = f"<Reply from '{user}' on '2023-01-01 00:00'>"
assert repr(reply) == expected_repr
By the way, I should mention that if your tests fail and you're not sure why, running Pytest with the -v flag can be very helpful. It gives you a bit more information about the expected and actual assertion values.
pytest -v
Testing for expected exceptions
What happens when a user passes something that isn't a datetime object to a Reply?
Well, nothing until __repr__ is called. Let's write a test for that.
Here we will use an assertion helped, pytest.raises, to say that an exception should be raised:
def test_repr_without_date():
body = "Hello, World!"
user = "user1"
created = "2021-02-07 12:00"
reply = Reply(body, user, created)
with pytest.raises(AttributeError):
assert repr(reply) == f"<Reply from '{user}' on '2021-02-07 12:00'>"
Note that with pytest.raises(AttributeError) says that when we call repr(reply), an AttributeError should be raised because we can't call created.strftime() (since created isn't a datetime object).
There are other helpers that are worth knowing about. For example, pytest.approx(), which allows you to do things like:
import pytest
def test_almost_equal():
x = 5
y = 5.01
assert x == pytest.approx(y, 0.1) # True
assert x == pytest.approx(y, 0.01) # True
assert x == pytest.approx(y, 0.001) # False
This one checks that the two values are within an acceptable range of each other, given by the second argument. Handy, especially when comparing floats (did you know most programming languages don't represent floats very well in some cases?).
Testing the equality of two objects
To write our tests for testing equality, we do something similar to what we've been doing! First make variables for the values, then create the objects, and finally compare them:
def test_eq_same_values():
body = "Hello, World!"
user = "user1"
created = datetime.datetime.now()
reply1 = Reply(body, user, created)
reply2 = Reply(body, user, created)
assert reply1 == reply2
def test_eq_different_values():
body1 = "Hello, World!"
user1 = "user1"
created1 = datetime.datetime.now()
body2 = "Goodbye, World!"
user2 = "user2"
created2 = created1 + datetime.timedelta(minutes=1)
reply1 = Reply(body1, user1, created1)
reply2 = Reply(body2, user2, created2)
assert reply1 != reply2
def test_eq_different_types():
body = "Hello, World!"
user = "user1"
created = datetime.datetime.now()
reply = Reply(body, user, created)
non_reply = "Not a reply object"
assert reply != non_reply
Testing a data store using pytest and fixtures
In order to showcase what "fixtures" are, let's add another class to our system:
from reply import Reply
class Thread:
def __init__(self, title: str, user: str, replies: list[Reply] = None) -> None:
self.title = title
self.user = user
self.replies = replies or []
def __repr__(self) -> str:
return f"<Thread '{self.title}' by '{self.user}'>"
def reply(self, reply: Reply) -> None:
if reply.created > self.replies[-1].created:
raise ValueError("New reply is older than the last reply")
self.replies.append(reply)
Here we've got a thread, with a title, the user who created the thread, and a list of replies.
It has one particularly interesting method, reply(), which takes in a Reply object and appends it to the replies list. But it only does so if the new reply is not older than the last reply in the thread. Or does it? I've introduced a bug in there to show you how easy it is to make a mistake here. Our testing will help us realise and fix the mistake!
Our first tests are fairly straightforward and introduce nothing new:
from thread import Thread
from reply import Reply
import datetime
def test_init():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime.now()),
Reply("Goodbye, World!", "user2", datetime.datetime.now()),
]
thread = Thread(title, user, replies)
assert thread.title == title
assert thread.user == user
assert thread.replies == replies
def test_repr():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime.now()),
Reply("Goodbye, World!", "user2", datetime.datetime.now()),
]
thread = Thread(title, user, replies)
assert repr(thread) == f"<Thread '{title}' by '{user}'>"
But now let's test the reply method:
def test_reply():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))
thread.reply(new_reply)
assert new_reply in thread.replies
If we run this (which by the way, you should be running your tests continuously, certainly after every new test, but preferably after every relevant code change)...
self = <Thread 'Hello, World!' by 'user1'>, reply = <Reply from 'user1' on '2023-01-03 00:00'>
def reply(self, reply: Reply) -> None:
if reply.created > self.replies[-1].created:
> raise ValueError("New reply is older than the last reply")
E ValueError: New reply is older than the last reply
simple_tests/thread.py:15: ValueError
===== short test summary info =====
FAILED simple_tests/test_thread.py::test_reply - ValueError: New reply is older than the last reply
How could this be? Our replies are correct:
2023-03-012023-03-022023-03-03
The first thing would be to check that the test is correct. But it seems it is, so maybe we've introduced a bug in the code!
After looking through it, you'll be able to see that we've got the > the wrong way round. It should be <:
if reply.created < self.replies[-1].created:
Changing that and re-running the tests shows us they pass. Excellent!
Now let's write a test that should fail:
def test_reply_more_recent():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))
thread.reply(new_reply)
This one fails, so let's add pytest.raises(ValueError) at the end:
# At top of file
import pytest
def test_reply_more_recent():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))
with pytest.raises(ValueError):
thread.reply(new_reply)
Using fixtures with pytest
You may have noticed that all tests start the same way, defining the title, user, and replies of our thread. We can extract them to a variable (but it won't work):
import datetime
import pytest
from reply import Reply
from thread import Thread
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)
def test_init():
assert thread.title == title
assert thread.user == user
assert thread.replies == replies
def test_repr():
assert repr(thread) == f"<Thread '{title}' by '{user}'>"
def test_reply():
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))
thread.reply(new_reply)
assert new_reply in thread.replies
def test_reply_more_recent():
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))
with pytest.raises(ValueError):
thread.reply(new_reply)
Here the more experienced among you will notice that we've done something taboo! We've got a global variable, thread, and in our tests we're calling thread.reply(), which modifies the global variable.
This is a recipe for disaster because now the order in which our tests runs matters. You never want your tests to depend on each other.
Think about it, if test_reply runs before test_init, the latter will fail because the thread will have 3 replies in it instead of the expected 2.
So instead, we need to create a variable that is unique to each test. We can do that with a fixture:
@pytest.fixture(scope="function")
def thread():
"""This fixture returns a Thread object"""
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
return Thread(title, user, replies)
Now each test can gain access to this variable using dependency injection. This is just a fancy way to say: add thread as a parameter to the test, and Pytest will automatically call the thread() function and give the test its value when the test runs.
So let's modify our tests so they have a thread parameter (note I've not used this fixture in test_init):
import datetime
import pytest
from reply import Reply
from thread import Thread
@pytest.fixture
def thread():
"""This fixture returns a Thread object"""
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
return Thread(title, user, replies)
def test_init():
title = "Hello, World!"
user = "user1"
replies = [
Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0)),
Reply("Goodbye, World!", "user2", datetime.datetime(2023, 1, 2, 0, 0, 0)),
]
thread = Thread(title, user, replies)
assert thread.title == "Hello, World!"
assert thread.user == "user1"
assert thread.replies == replies
def test_repr(thread):
assert assert repr(thread) == "<Thread 'Hello, World!' by 'user1'>"
def test_reply(thread):
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 3, 0, 0, 0))
thread.reply(new_reply)
assert new_reply in thread.replies
def test_reply_more_recent(thread):
new_reply = Reply("Hello, World!", "user1", datetime.datetime(2023, 1, 1, 0, 0, 0))
with pytest.raises(ValueError):
thread.reply(new_reply)
So this helps us answer the question: "what are fixtures?"
Fixtures are functions that allow us to reuse data in our tests, and can be called once per test by Pytest.
Note that scope="function" can take other values! If you want the same fixture value to be reused in the entire file, you can say scope="module". If you want it to be reused in the entire test suite, you can use scope="session". There's also scope="class", but that only becomes relevant when you use an object-oriented approach to writing your tests, which is something I almost never do.
By default, fixtures can only be used in the file where they are defined. If you want to share a fixture between different test files, you can put the fixtures inside a file called conftest.py. Then the fixtures will be visible in any test file in the folder that contains conftest.py, or any sub-folder.
Setup and teardown in Pytest fixtures
Every fixture in Pytest can run some code before and after returning the value used in the tests.
Here's a non-useful example of how this works:
@pytest.fixture
def my_fixture():
print("Hello, fixture!")
yield 42
print("Bye, fixture!")
def test_nothing(my_fixture):
print("In the test!")
print(f"Value from fixture: {my_fixture}")
Running this, you'll see this output is captured:
Hello, fixture!
In the test!
Value from fixture: 42
Bye, fixture!
Obviously, that example isn't very useful, but I'm sure you can see when it may be useful:
- If your tests need to interact with a file, you could use the setup and teardown functionality to open and close the file before and after giving it to your tests.
- If your tests use a database, you can use this to connect to the database and close the connection when you're done.
How to use multiple fixtures in a Pytest test
Using multiple fixtures is easy as pie! Just add multiple parameters to the tests, and Pytest will make sure the fixtures are called and the values passed:
def test_with_two_fixtures(fixture_one, fixture_two):
pass
Tracing fixture execution and finding where fixtures are defined
When you execute the pytest command to run your tests, you can pass the --fixtures flag and it will show you all the fixtures that are available for your test functions to use. This includes fixtures Pytest provides, as well as your own:
cache -- .venv/lib/python3.11/site-packages/_pytest/cacheprovider.py:509
Return a cache object that can persist state between testing sessions.
capsys -- .venv/lib/python3.11/site-packages/_pytest/capture.py:905
Enable text capturing of writes to ``sys.stdout`` and ``sys.stderr``.
capsysbinary -- .venv/lib/python3.11/site-packages/_pytest/capture.py:933
Enable bytes capturing of writes to ``sys.stdout`` and ``sys.stderr``.
... (there are a few)
----- fixtures defined from test_thread -----
thread -- simple_tests/test_thread.py:9
This is a Thread object
If you want to see what fixtures each test used, you can use --fixtures-per-test:
$ pytest --fixtures-per-test -k test_thread
...
----- fixtures used by test_repr -----
----- (simple_tests/test_thread.py:36) -----
thread -- simple_tests/test_thread.py:9
This fixture returns a Thread object
----- fixtures used by test_reply -----
----- (simple_tests/test_thread.py:40) -----
thread -- simple_tests/test_thread.py:9
This fixture returns a Thread object
----- fixtures used by test_reply_more_recent -----
----- (simple_tests/test_thread.py:48) -----
thread -- simple_tests/test_thread.py:9
This fixture returns a Thread object
Conclusion
In this beginner's guide to pytest, we've covered the basics of how to write tests using pytest, including assertions, fixtures, and selecting tests to run. We've also covered some advanced features like fixture scopes and sharing fixtures among multiple files using conftest.py.
Writing tests is an important part of software development, and pytest is a great tool for making testing in Python easy and effective. With the knowledge you've gained from this guide, you should be well on your way to writing better tests and building more reliable software!
If you want to learn more about Python generally, consider enrolling in our Complete Python Course! This comprehensive course teaches from the basics to the advanced, covering topics such as object-oriented programming, async development, GUI programming, and more.
For further reading, I strongly recommend the official documentation. Brian Okken has also written an excellent book:
- The official pytest documentation
- Python Testing with pytest by Brian Okken, an excellent book that covers everything you could want to know about Pytest
That's all for today! Thank you for joining me, I hope you've enjoyed the read, and I'll see you next time.
