If the working directory and the venv are on different disks, venv doesn't work in bash on Windows

[Zabolekar]

Environment

• OS: Windows 10
• Architecture: x86_64
• Shell: bash 4.4.23, part of the git 2.37.0 install
• Python version: 3.11.1
• Output of path/to/python -m pip list:

Package    Version
---------- -------
pip        23.0.1
setuptools 65.5.0

Issue

When I create a venv on D: and use it from C:, executables like pip or python can't be found:

$ pwd
/d
$ /path/to/python -m venv myenv
$ . myenv/Scripts/activate
$ pip --version
pip 22.3.1 from D:\myenv\Lib\site-packages\pip (python 3.11)
$ cd /c
$ pip --version
bash: \myenv/Scripts/pip: No such file or directory

Linked PRs

• gh-103088: Fix scripts for Windows #103325
• gh-103088: fix activate script not working in Cygwin #103470
• gh-103088: Fix test error message #103500
• gh-103088: Ensure POSIX venv scripts always use LF line endings #103591

[eryksun]

A POSIX shell delimits PATH with ":". Thus "D:\myenv" represents two separate pathnames "D" and "\myenv". If a path begins with "\", MSYS apparently resolves it against the drive path of the current working directory (e.g. "/c" or "/d").

You can work around this issue by editing "{env_dir}/Scripts/activate" to replace the value of VIRTUAL_ENV with the equivalent MSYS path (e.g. "D:\myenv" -> "/d/myenv").

Maybe the "activate" script could detect MSYS or Cygwin and automatically translate the value of VIRTUAL_ENV via export VIRTUAL_ENV=$(cygpath -u "$VIRTUAL_ENV").

[Zabolekar]

Thanks for the answer. My current workaround was to move venvs and code to the same disk.

Now that you mention Cygwin I've tested the script with Cygwin's version of bash as well. The script fails because it contains \r. After applying dos2unix, it works but has the same issue as MSYS bash.

Let's say I want to fix the script for everyone and not just for me. What would be the process, and what bash variants other than MSYS and Cygwin should I be aware of?

[eryksun]

Let's say I want to fix the script for everyone and not just for me. What would be the process, and what bash variants other than MSYS and Cygwin should I be aware of?

Cygwin and MSYS2 are the common POSIX environments on Windows, other than WSL. Create a pull request that modifies the template script "Lib/venv/scripts/common/activate".

cpython/Lib/venv/scripts/common/activate

Lines 41 to 42 in e375bff

	VIRTUAL_ENV="__VENV_DIR__"
	export VIRTUAL_ENV

Apparently MSYS and Cygwin can be detected on bash using the OSTYPE environment variable. The following change seems to do the job. I haven't checked whether it causes a problem for an environment that's created by the MSYS2 or Cygwin versions of Python, but I don't think it should cause a problem. cygpath -u should only modify a Windows path such as "D:\myenv", not a POSIX path such as "/d/myenv" or "/cygdrive/d/myenv".

if [ "$OSTYPE" = "cygwin" ] || [ "$OSTYPE" = "msys" ] ; then
    export VIRTUAL_ENV=$(cygpath -u "__VENV_DIR__")
else
    export VIRTUAL_ENV="__VENV_DIR__"
fi

The venv module replaces the __VENV_DIR__ template parameter with the path of the created virtual environment.

[Zabolekar]

Thanks for your help. I didn't even know MSYS had cygpath, too. I've tested creating venvs in cmd and in MSYS2, then activating them in cmd, MSYS2, and cygwin, and looking at pip --version and python --version. Everything works as expected. Here is a pull request.

Two things I've noticed while doing this, not sure whether those are bugs or not:

• the script has CR LF, so Cygwin only understands it after I run dos2unix on it
• https://devguide.python.org/ advises to build on Windows with -d by default (set the configuration to Debug). So the resulting venvs don't contain python.exe, only python_d.exe, and calling pip directly doesn't work ("Fatal error in launcher"). If Python has been built without -d, everything works as expected.

[zooba]

• the script has CR LF, so Cygwin only understands it after I run dos2unix on it

It may depend on your Git settings, if you've cloned and built from source. Or maybe this file gets modified when venv copies it? @vsajip any thoughts?

• resulting venvs don't contain python.exe, only python_d.exe, and calling pip directly doesn't work

Guessing this is a script launcher limitation, or maybe they aren't being generated properly in this scenario. In any case, it's external - I believe pip still gets their launchers from distlib?

[Zabolekar]

It may depend on your Git settings, if you've cloned and built from source

It doesn't only affect those who build from source: Python 3.11.1 that I installed from python.org has CR LF in Lib/venv/scripts/common/activate, too.

It doesn't feel right to just clone everything with Unix line endings on Windows. I'm not sure whether all relevant programs can handle it correctly. On my computer, it seems to suffice if I add Lib/venv/scripts/common/activate text eol=lf to .gitattributes, then delete the file and do git checkout-index --force --all to recreate it. But I don't know how to fix it for the installer as well.

[zooba]

If that's added to .gitattributes then it should work fine for the installer. We don't do anything to that file.

I'd be more concerned about venv creation reading, modifying and writing the file with different line endings. If it doesn't do that, then modifying the attributes should be enough.

[vsajip]

Or maybe this file gets modified when venv copies it?

The activate script in my repo on Linux has LF terminators, and the copying code reads and writes the files as binary (there are some variable substitutions, but none of those should introduce CRLF line endings).