#039: The Importance of Virtual Environments

A virtual environment is an isolated directory on your computer that holds the Python packages, and the exact versions of them, for one project. Most modern data tools, dbt and Prefect among them, are Python based, so each project you run pulls in its own set of dependencies. Without isolation those sets collide: one project needs one version of a module and another project needs a different one. A virtual environment keeps each project's packages separate, lets you hop in and out with a single command, and gives you a requirements file that teammates can use to rebuild the same setup. Python ships with venv, which is all most data engineers need.

What virtual environments are

Many data tools today are code based and built on top of Python. You don't have to be a Python wizard, but you do need to get comfortable building and maintaining Python style projects.

When you work with Python you're dealing with a lot of packages and modules, and for things to work as expected you need specific versions of them on your machine.

A virtual environment is an isolated area on your computer that holds all of the dependencies and modules for a given project. Nothing else on the machine sees them, and they don't see anything else.

Why it matters for data engineers

Different projects need different versions. Say a module is required at one version for your dbt project. Then you start an Airflow project that needs the same module at a different version.

Install both directly on your computer and you get conflicts. One of the two projects breaks, and usually it's the one you weren't working on when you installed the other.

With a virtual environment per project, each version lives in its own place. You hop into the environment for the project you're working on, and it always works.

Sharing the environment

The other benefit comes later in the workflow. You can get a list of every package in a project with its exact version, share it through version control, and everyone on the team knows precisely what they need to mimic your setup.

I learned this one the hard way. Early on, a case study interview had me build a custom Python script. The script worked, but one of the first questions I was pressed on was virtual environments, because I hadn't used one. I got the job anyway, and I haven't started a Python project without one since.

Common tools to use

There's more than one way to create a virtual environment. The four most common:

1. venv

A module that comes prebuilt with Python 3.3 and later. It's bare bones: it creates an isolated environment and not much else, and that's exactly what I want most of the time. It's what I use and what the example below uses.

2. virtualenv

Supports both Python 2 and Python 3, so it's a bit more flexible than venv. The trade-off is a separate pip install before you can use it, where venv works straight out of the box.

3. conda

Handles virtual environments and also manages all of your packages, plus some other things. More features, but it's a separate tool to install and there's more to it than a single built-in command.

4. pyenv

I haven't used this one myself, but it lets you switch and manage the Python version itself per project, alongside the environment. Useful if you're juggling Python versions, not just package versions.

Which one you pick depends on how much complexity you need and what your team already uses. If you just want to isolate projects, venv gets you going.

Create a virtual environment

To show what isolation actually looks like, start in a plain directory with no environment active and check for two tools:

dbt --version
prefect version

Both come back with command not found. Neither tool exists on the machine. The plan is to install each one in its own environment and watch how they stay separate.

Create and activate

With venv, the command is the Python module plus a name for the environment:

python3 -m venv my_env

That creates a my_env directory with a handful of files inside. You don't need to know what's in there. The one that matters is the activate script.

Activation differs by operating system. On Mac or Linux:

source my_env/bin/activate

On Windows:

my_env\Scripts\activate

Your prompt now shows (my_env) at the front. You've probably seen that prefix in other projects without knowing where it came from. To leave:

deactivate

The prefix disappears. Create a second environment the same way, say another_env, and you can hop between them with the same activate and deactivate commands.

Install dbt in one environment

Activate my_env and install the dbt adapter for Snowflake:

source my_env/bin/activate
pip3 install dbt-snowflake

Notice how much gets installed. It isn't just dbt, it's a long list of dependencies that come with it. Imagine that list on your main machine, next to other projects that need different versions of the same things.

Once it finishes, dbt --version works and prints the installed version.

Prove the isolation

Now walk through the environments and check:

• Deactivate and run dbt --version. Command not found.
• Activate another_env and run it again. Still not found.
• Deactivate, then activate my_env and run it once more. It works.

dbt only exists inside the environment it was installed in. That's the whole concept.

Install Prefect in the other one

Repeat it with a second tool in the second environment:

source another_env/bin/activate
pip install prefect
prefect version

Prefect pulls in its own long list of modules, and prefect version works here. Deactivate and it's gone. Activate my_env and it's not there either. Two tools, two environments, no overlap.

Export and share dependencies

The last piece is handing your environment to someone else. Inside an active environment, pip freeze prints every installed module and its version:

pip freeze

In my_env that list includes dbt-snowflake and everything it brought with it.

Write a requirements file

Redirect that output into a file:

pip freeze > requirements.txt

requirements.txt is the standard way of sharing a dependency list. It's a plain text file that lives in your project and goes into version control, so anyone who clones the repository can see exactly what's needed.

Commit the requirements file, not the environment directory. The directory is machine specific and large. The file is a few lines and rebuilds it anywhere.

Rebuild it somewhere else

To mimic the environment exactly, deactivate, create a fresh one, and install from the file:

python3 -m venv third_env
source third_env/bin/activate
pip install -r requirements.txt

pip reads the file and installs every dependency at the listed version. When it's done, dbt --version works in third_env just as it did in my_env.

That's the full loop. One person builds the environment, freezes it, commits the file. Everyone else creates their own environment from that file and works under the same conditions.

Related reading

• Why Data Teams Need Version Control
• Getting Started with Prefect (Task Orchestration)
• A Simple 4-Step Process for Creating dbt Models
• dbt Environments vs Targets: What's the Difference?

Final takeaway

Every dbt or Prefect project I set up with a client starts with a virtual environment and a requirements file, before a single model is written. It costs two commands, and it's the difference between a project a new teammate can run in five minutes and one that only works on the laptop it was built on.