Most Python projects we develop will have dependencies: code written by other people that we install and use, so that we don't have to re-write it ourselves.

For example, the requests library is a popular third-party dependency to make HTTP requests. The flask library is used to create web applications.

There are thousands upon thousands of libraries to do all kinds of things, and there are tens or hundreds of thousands of people working on those libraries.

We install and use libraries so that we don't have to re-write all that code written by those people ourselves!

The libraries change over time, as new and better ways to do things are discovered and implemented.

As such, when we install a library for one of our projects, it will be a specific version. Months or years down the line, if try to install it again, it may well be a different version with different functionality.

Therefore, it's possible that when we update a library, our code will no longer work because the way that we should use the library has changed.

Virtual environments exist so that we can separate the dependencies of one project from the dependencies of another project. That way, they can have different versions of the same library. That often happens when we start projects at different times.

What is a virtual environment?

There are two key parts to a virtual environment:

  • A specific version of Python, for example Python 3.9.
  • A folder of third-party libraries that you've installed.

Every virtual environment we create can be created with a different Python version. That lets us work on projects that use different Python versions very easily. Similarly, because each virtual environment has its own folder of third-party libraries, they can have different libraries or the same libraries in the same or different versions.

How does Python import libraries?

When we run a Python file, we have access to import paths, or where Python will look when we try to import things.

We can see what the import paths are by running this code:

import sys

print(sys.path)

If you run this, you may get something that looks like this:

["/Users/youruser/Documents/projects/myproject", "/Applications/Python/Python39/bin", "Applications/Python/Python39/lib", "Applications/Python/Python39"]

What this means is that, if we try to import something, Python will look for something.py on the first path in the output list.

If it finds it, it will import it. If it doesn't find it, it will move onto the second path of that list. And then the third, and so forth until it is found.

If it isn't found, Python will raise an ImportError with a message telling you that the thing you tried to import doesn't exist.

Now, this is going to be relevant to virtual environments in a moment, as we'll learn!

How to create new virtual environments

Creating new virtual environments with recent Python versions is as easy as running this command on your console:

python -m venv venvName

This will create a new virtual environment called venvName in the current folder.

Remember that if you have multiple versions of Python installed, you may have to do python3.9 -m venv venvName or similar, depending on your Python version.

What happens when you activate a virtual environment?

Now that you've got the virtual environment created, we can activate it!

You can activate the virtual environment using this command (on Mac or Linux):

source venvName/bin/activate

If you're using Windows with WSL or Git Bash:

source venvName/Scripts/activate

Or, if you're using Windows with cmd.exe or PowerShell:

.\\venvName\\Scripts\\activate.bat

When we activate a virtual environment, two main things happen:

  • We get to use the python and pip commands in the console, and they use the Python that is inside the virtual environment. When we use pip with the virtual environment activated, new libraries will be installed inside that virtual environment.
  • When we run Python using the python command, the sys.path values are modified so that any imports are looked for inside the virtual environment's third-party library folder. So anything we've installed in the virtual environment will be available, but things we've installed in other virtual environments won't be available.

You can easily verify this by running the code that prints sys.path again, after activating the virtual environment first. You'll see that it's a bit different!

Virtualenv vs. venv vs. Pipenv

There are a few different packages for handling virtual environments, such as virtualenv, venv, or Pipenv. They're all a bit different, and they have pros and cons.

I recommend you use venv, since it comes with the new Python versions and it's very easy to use.

The virtualenv package is an older package we had to install in older Python versions. It's no longer necessary and it doesn't do anything that venv doesn't do.

The Pipenv package does a bit more than venv, but it deserves its own blog post to explain all the differences. I don't think it's worth using at the moment.

Saving dependencies in a text file

Saving your dependencies to a text file can be useful, so that when you share your code with other people, they know what to install in order to run your project.

When we use venv, it's common to save the names of the dependencies to a file called requirements.txt.

For example, here's a requirements.txt file for one of my projects:

requests
flask
gunicorn
pymongo[srv]

This has 4 dependencies, one per line. If I download the entire code and want to install the latest versions of the dependencies detailed in requirements.txt, I just do this:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

It's common to name the virtual environment .venv, so that's what I normally do!

Including version numbers

Any requirements.txt file can also include version numbers, so that when you install the dependencies from the file, you'll get the exact versions detailed there.

This can be useful because, like I mentioned at the beginning, libraries change over time, and your code may not work with the latest versions.

Usually, we'll write down the versions that we used when developing the code. Updating the version number of any dependency should be a conscious decision, since sometimes we'll need to make changes to the code in order to accommodate the changes to the library.

Here's how to include version numbers in your requirements.txt file:

requests==1.0.0
flask>=1.1.2
gunicorn==20.0.4
pymongo[srv]==3.11

There are a few ways to write this. Shown above:

  • == means "install exactly this version".
  • >= means "install the latest version, which must be above the following".

Handling major version changes

Often, libraries will make major changes when they change the "major version".

In library versioning, the "major version" is the first number, so these two would have the same major version:

  • 1.0.0
  • 1.2.1

But they are on different minor versions (middle number) and patch versions (last number).

Oftentimes, we want to install the latest version of a library, while staying within a major version. That way, if we install the dependency in the future, we'll install the same major version that we were working with in the past, but we'll gain any small improvements the library has made in its minor and patch versions.

It's worth repeating, that usually libraries don't add breaking changes to minor and patch versions, that's why this is a common thing to do. However, a library could break at any version, so this is all about tradeoffs and risk!

To install the latest minor/patch version of a library, we can do this:

flask>=1.1.2,<2.0

Conclusion

That's about everything for this post! I hope it's been helpful to learn a bit more about virtual environments, how they work, and how you can use them in your projects!

If you want to learn more about Python, consider enrolling in our Complete Python Course which takes you from beginner all the way to advanced Python knowledge (including web scraping, async development, and much more!).

Thanks for reading, and I'll see you next time!

References