How I install pipenv and why

tl;dr:

pip3 install --user pipx
pipx install pipenv

Background

I personally enjoy writing a lot of python code, I've done it for community projects, personal projects, and for work. One of the parts of python, at least for me, (and it seems like for others as well) that's...let's say hairy... about writing python is the managing dependencies for long running projects.  While the pypi ecosystem is vast and expansive, sanely taming how you manage your dependencies for developing tools you create can be...frustrating at best 🙃.

Solution for Dependencies

Ever since I watched the YouTube video below, I've been trying to enlighten others about using pipenv to manage any project I see that has just a requirements.txt file.

The main allure (IMO) is that it pins the dependencies in a .lock file, but lets your regular Pipfile be human readable and easy to modify.

Another amazing feature, that I'd only found out about in the past year, was the scripts section (wbm). While it can be a bit kludgy IMO (but I don't know a better way either), it reminds me of npm scripts, and it's super useful if you don't want to show your co-workers how to deal with pipenv environments (pipenv shell) or use longer commands in a CI/CD system.

Reason for this post

Overall whenever I've done pip3 install --user pipenv everything has pretty much worked.  I've run into a few issues along the way that I hadn't expected and mostly they make complete sense, but are still very frustrating.

Python Upgrades

While this doesn't happen very often on most Linux distributions, I use Kali Linux for a lot of systems at work.  Unfortuently they upgrade python at a lot faster cadence than typical Long Term Support (LTS) distributions, since they're a rolling distribution (which makes complete sense), so I have to deal with this a lot more often than I'd like to 😅.

Below is a picture of what I'd typically get when the python version was upgraded. Ironically this picture was taken while writing this blog post on one of my LTS distributions.

Weird warning messages during usage

While this isn't necessarily bad, I've run into weird situations (and of course I can't replicate it now for a screenshot...) were I've gotten warning messages about not being installed on a known python version.  This typically was on my LTS systems, and I'm imagining it's because of the Canonical team patching their python version and giving it some weird internal versioning scheme. So, again while it's not necessarily bad it's annoying.

Solution

After learning about pipx from byt3bl33d3r (source), I figured I could avoid both of these issues by using pipx to install pipenv 😅.

So, while it seems kind of counter intuitive to have an entire virtual environment setup for a virtual environment manager...it's been working flawlessly for quite a while now 😁.

So, all you need to do is install pipx and have access to the binary:

pip3 install --user pipx

##################
## the rest of this is optional if you already have ~/.local/bin
#    in your PATH, if you don't know what any of that means
#    it won't hurt to follow along :)
# to ensure you can access the pipx binary
python3 -m pipx ensurepath
# to reload your path to access the pipx binary
source ~/.bashrc
# or on zsh (kali)
source ~/.zshrc

One caveat is that if you're on a debian based distro then you also need to install the packages listed here.

Then install pipenv:

pipx install pipenv

You should be good to go, have fun coding your python project 🥳

Optional Additional Configuration/Information

Python Version (Hack)

One of the things about using pipenv is that you can pin your python version in the lock file.

[requires]
python_version = "3.10"

This is extremely useful to have an even more deterministic level of control of how your code works.  While this sounds good in theory it's also something that's not always something you think of when you're on a linux distro that has could have a different system python version. So a quick hack around it, but I don't recommend this on production applications, is you can specify the version like this: python_version = "3" which is telling it any version of python 3. Unfortunelty you can't specify a range of versions (src), but I can't fault them since I'm not going to create the PR to fix it 😅.

Python Version (more official)

Instead of manually installing the different python versions and making them accessible to pipenv's path, pipenv actually suggests using pyenv or asdf.

Both of these tools are really useful for having multiple version of python installed, but IMO if you're only doing python code you can probably safely choose pyenv since it's dedicated to python.

If you were planning on having multiple version of ruby & python (i.e. static analysis of a company's code), then I'd recommend using asdf because it's extensible ecosystem allows you to install more than just python.

If you go with pyenv, then it's pretty easy to get started by just using they're "curl | bash" script (read here about how you can examine the code before intalling it (recommended)). Once that's installed pipenv should pick up the path to the binary and automatically handle the installation. It'll prompt you to make sure it's okay, like the picture below.

Then once you hit enter (since Y means yes is default), then it'll automatically download the source code for the python version, compile it on you computer, and make it accessible to pipenv.

Python Version (automation - ansible)

I've done this (install pip3 + pyenv + pipenv) so many times now that I've created an ansible collection around it so I can automate, if you don't have any different use cases it should just work 😅.

It's not the most elegant solution yet (haven't had time to create tests and make it more streamlined), but when I have the time I plan to.

Caveat

When it comes to installing different version of python with your pipenv install my role doesn't handle that very well...so here are two hacks you can do around that.

If you've heard of expect scripts before:

#!/usr/bin/expect

set timeout 680

spawn pipenv --bare sync

expect "Y/n" { send "\r" } \
    "All dependencies are now up-to-date!" { }

expect eof

I haven't converted my role for pipenv to use an expect script yet, but I plan to.

Alternatively here's a script you can use to wrap around the original binary you want to run (if you're using something like a pex file):

#!/usr/bin/env bash

# https://elrey.casa/bash/scripting/harden
set -${-//[sc]/}eu${DEBUG:+xv}o pipefail

function install_python_version(){
  # check for the python version you're expecting
  if [[ -d "/root/.pyenv/versions/${pyenv_python_version}" ]] ; then
    # if it's there, do nothing
    :
  else
    # else install the python version
    pyenv install "${pyenv_python_version}"
  fi  
}   

function set_python_version(){
  # if the .python-version file exists, it means you've
  #  set the local python version for that folder
  # REPLACE THE LINE BELOW WITH THE PATH TO YOUR PROJECT
  if [[ -f "<path_to_project>/.python-version" ]] ; then
   # if it exists, then do nothing
   :
  else
    # set the local python version
    pyenv local "${pyenv_python_version}"
  fi  
}   

function main(){
  # specify the full python version (pyenv install --list)
  pyenv_python_version='3.7.16'
  # load pyenv to ensure you can access the binary
  source ~/.bashrc

  # calling the function we defined before
  install_python_version

  # REPLACE THE LINE BELOW WITH THE PATH TO YOUR PROJECT
  pushd <path_to_project>
  # calling the function we defined before
  set_python_version

  # execute the original binary
  ./<original_binary>.orig
}   

# https://elrey.casa/bash/scripting/main
if [[ "${0}" = "${BASH_SOURCE[0]:-bash}" ]] ; then
  main "${@}"
fi

Appendix: Debian Distros

If you're on Debian based distros (i.e. Ubuntu), then you'll need to install 2 dependencies for pipx to work properly. Run the following command, and you should be good.

sudo apt update && sudo apt install -y python3-{pip,venv}