I wear many hats in open source software spaces, including maintaining critical projects that serve as the foundational digital infrastructure for the ecosystem of the Python programming language, making it easier to create good technical documentation websites and being involved in community events.

My current day job is to make it easier for software developers to write software in Python; as part of Bloomberg Engineering’s Python Infrastructure team.

This page is where I document stuff that I am responsible for, am involved in, etc. For things like work experience and education, see Linkedin.

Open Source Software

I contribute to a lot of open source software projects.

Community Organisation and Engagement

Recognition

Talks

I am an experienced public speaker, having presented at numerous conferences and meetups on topics related to Python and software development. My talks have covered a range of subjects, such as open source project management, documentation best practices, UX design, and Python packaging.

Interviews & Features

In a variety of interviews and podcast appearances, I have had the opportunity to discuss my work in the Python community as well as my insights and experiences as an open source developer.

Older Projects

These are projects that I have stepped away from.

How it works

The dotfiles are managed by a Python script. In broad strokes, the script will:

• read a TOML file
• locate the configured paths
• create symlinks, based on custom marker text in the filenames, for files in subdirectories under the configured paths¹

If there’s a conflict (i.e. two configured paths provide the same symlink target location), the TOML file contains the resolution for it (i.e. specifies one of the paths as the “winner” for that file). If a resolution doesn’t exist, the script errors out.

This setup exists mainly to allow me to have work-only dotfiles managed in work’s VCS with all their corporate stuff while keeping them separate yet cooperative with my dotfiles managed publicly on GitHub.

The problem

Now, this effectively means that my dotfiles manager is a Python script. It also has a bunch of dependencies (specifically, rich because I like colors and tomli because it should probably run on all supported Python versions).

This usually means that you need to create a virtual environment to run it. That’s gotten a bit tedious and fragile given how frequently I keep changing my Python installations (gotta keep up with the latest and greatest!).

The solution

The implementation

I figured out a way to ensure that the script can be run without needing to manage a virtual environment myself. Now, it can be run on any machine with a working curl, bash, python3 and the ability to download from https://github.com. It also gracefully triggers MacOS’ prompt for “Hey, do you want to install our developer tooling stuff?” (XCode Command Line Tools, via the python3 shim they install on a new Mac).

This is made possible by pipx, inline dependency metadata and Python’s zipapps. There are also a few shenanigans to make this script a valid Bash script and a valid Python script. A """"true serves as our little gem of polyglot magic to make that possible.²

How it works: Bash

Let me show you the script first, with Bash’s syntax highlighting:

#!/usr/bin/env bash
#
# /// script
# requires-python = ">=3.8"
# dependencies = [
#   "rich",
#   "tomli",
# ]
# ///
#
""""true
if [[ ! -f '/tmp/pipx-dotfiles.pyz' ]]; then
  echo "Downloading pipx..."
  curl --proto '=https' --tlsv1.2 -sSLf https://github.com/pypa/pipx/releases/latest/download/pipx.pyz -o /tmp/pipx-dotfiles.pyz
fi
python3 /tmp/pipx-dotfiles.pyz run "$0" -- "$@"
exit
"""

import rich
print(rich.__version__)

From Bash’s perspective, this script has a bunch of comments, then a few commands, and then an exit.

The way Bash processes a script is by reading it as a buffer and executing it before moving on. This allows Bash scripts containing syntax issues to be run, as long as the syntax issue is somewhere that isn’t being read by Bash (i.e. after the point of exit). We’re definitely using that since the Python code (which is not valid Bash syntax) is present after the exit.

To Bash, our little gem of magic (""""true) is the same as a true command since it’s a bunch of empty strings concatenated together with true. And, true is used as a no-op command (it exits with code 0 and does nothing else).³ It is treated the same as a true on its own line and Bash proceeds with the rest of the logic as a regular Bash script.

The rest of the logic in the Bash script is to download pipx’s zipapp (if it’s not already downloaded) and then run the script itself with python3 <pipx-zipapp> run <script> -- <any arguments passed to the script>.

How it works: zipapp

Wait, a zipapp?

So, that’s a fun Python feature: it can execute a zip file as if it were a script. Python will look for a __main__.py in the zip file and, if it’s there, the __main__.py as if it were a script. See the documentation if you want to learn more about this.

In this case, the pipx maintainers create a maintain a zipapp and attach it to their GitHub releases. By using GitHub’s latest release URL, we can fetch the zipapp for the latest release (with a redirect, hence the -L to curl) without much additional complexity.

How it works: pipx run

pipx run enables running a script with the dependencies being installed by pipx in a cached virtual environment that is managed by pipx.

Notably, it supports PEP 723 (inline script metadata) which enables declaring dependency information inline. This is what the /// script and /// are serving as markers for. The requires-python and dependencies are bits of metadata that pipx will use to determine what needs to be present for running the script.

Assuming you’re on a compatible Python, pipx will parse that chunk, install the dependencies in a cached virtual environment and then run the script within that virtual environment. This means that the dependencies are installed by pipx and can/will be reused across multiple runs of the script by pipx.

How it works: Python

Let’s look at the script again, this time with Python’s syntax highlighting:

#!/usr/bin/env bash
#
# /// script
# requires-python = ">=3.8"
# dependencies = [
#   "rich",
#   "tomli",
# ]
# ///
#
""""true
if [[ ! -f '/tmp/pipx-dotfiles.pyz' ]]; then
  echo "Downloading pipx..."
  curl --proto '=https' --tlsv1.2 -sSLf https://github.com/pypa/pipx/releases/latest/download/pipx.pyz -o /tmp/pipx-dotfiles.pyz
fi
python3 /tmp/pipx-dotfiles.pyz run "$0" -- "$@"
exit
"""

import rich
print(rich.__version__)

From Python’s perspective, this script has a bunch of comments, then a docstring, and then the whole Python script.

To Python, our little gem of magic (""""true) is the start of a multiline string which starts with the content "true\n. This is valid Python syntax, and Python will happily treat it as the start of the string literal. The end of the string literal is the next """ it encounters, which is the one at the end of the Bash parts of the script.

This means that the Bash script is treated as a multiline string by Python. It’s treated as the docstring of the Python script. I am not too concerned about that since this is a script that doesn’t need a docstring.

I’d like to highlight a major change in this blog post: the removal of support for “legacy” versions and dependency specification, or as I like to call it, nonsensical versions and dependencies. We’re releasing this as a beta to get feedback from the community on how this change affects their workflows.

Nonsensical versions and dependencies

For a bunch of historical reasons, pip has allowed many arbitrary strings as versions and dependency specification. There’s so many kinds of bad behaviours that this “feature” has enabled.

My favourite example is one I wrote back in 2022 on the packaging¹ project’s issue tracker:

>>> packaging.version.parse("This is a completely random string")
<LegacyVersion('This is a completely random string')>

Or providing weird version comparisons:

>>> packaging.version.parse("1.0") > packaging.version.parse("99999.0.whatever")
True
>>> packaging.version.parse("1.0")
<Version('1.0')>
>>> packaging.version.parse("99999.0.whatever")
<LegacyVersion('99999.0.whatever')>

Or hiding mistakes in dependency declarations:

>>> all_dependencies = [
...     "package == 1"
...     "another-package",
...     "yet-another-package",
]
>>> [packaging.requirements.Requirement(s) for s in all_dependencies]
[<Requirement('package==1another-package')>, <Requirement('yet-another-package')>]

Over a decade ago, the Python packaging ecosystem adopted PEP 440 which defined a standard for versions and dependency specification. While this standard has been adopted by basically every single tool in the ecosystem, support for “legacy” versions and requirements has been maintained even though it has caused a lot of user confusion and bugs in the Python ecosystem.

Getting rid of the nonsense

Over the course of the last decade, across multiple projects in the Python packaging ecosystem, many PyPA projects have been working to make this behaviour not possible. The package index, https://pypi.org, has not accepted uploads with bad versions for many years now and is also enforcing validity of the requirements on all uploads. The tooling that generate packages (build-backends) also now refuse to generate packages with such bad metadata in them.

This effort has been undertaken by many people across the Python packaging ecosystem, and I’m very grateful for all the work that has gone into this. It regularly surprises me how much work goes into making sure that the foundational pieces of Python’s packaging ecosystem are improving, even absent any institutional investment into it.

This pip release marks the next major step in the direction of improving behaviours in this area: updating the package installer to stop accepting such packages. Starting with this pip release, pip will refuse to install packages with bad versions or dependency specification and will ignore any files with such bad versions or dependency specification during the dependency resolution step.

Where we can’t see the nonsense

While a substantial amount of effort has been put into making sure pip can work correctly when enforcing correctness around these versions and dependency specification, we want to be cautious in rolling out this change. There are many places that Python is used where we can’t access the packages (e.g. private package indexes hosting proprietary code), and thus can’t ensure that packages have appropriate metadata in those spaces.

We’re releasing this as a beta to get feedback from the community on any issues that might arise from this change. This will help us better understand how this change affects user workflows and what actionable guidance can be provided to users in various contexts.

Help us test the changes

We’re looking for feedback from Python users on this beta release. Notably, we’d like for users who are using package indexes other than PyPI to test their workflows against this beta release of pip and provide us with feedback.

You can let us know about issues related to this change on pip’s issue tracker (please check for duplicates before filing, and use reactions rather than saying “me too”!). Please also feel welcome to tell us about the good stuff, like if this release made dependency resolves quicker – we like hearing about that too! :)

Acknowledgements

While there’s a lot of people who have worked on things I mentioned in this blog post, I can’t really list them all here – you know who you are, and thank you for all the work you’ve done toward this as well as the other cool things you’ve worked on.

I do want to thank Paul Moore, Stéphane Bidoul, Richard Si and Pavithra Eswaramoorthy for reviewing various drafts of this post and providing feedback on it.

The problem

The “new” GitHub Notifications UI¹ was based on Octobox. It was a step in the right direction to help with my notification “workload”, although I recognise that it was not perceived as such by many others.

That said, they copied over the exact same design issue I had with Octobox: the action buttons are on the completely opposite side from where my attention is.

I am going to scan and read the content of the notification, which is left aligned and very close to the left edge. All the action buttons are on the right edge. This makes it difficult when trying to triage notifications, especially when I’m trying to do so in bulk where stuff like “oh, this is not gonna need my attention” requires me to drag my cursor over to the other side of the page.

If there’s a keyboard shortcut for this, I don’t know it and I don’t want to learn it anyway. I want to be able to use the mouse to do what I need here because that’s how I use a computer.

The solution

I wrote a blurb of CSS to remove this annoyance.

It moves the action buttons toward the left side, so that I can quickly triage notifications without having to move my cursor across the screen. It also moves the text describing the content that the notification is for (repository and notification subject) to accommodate for the buttons.

If you know me, you know that I will animate moving text to different locations. In the end, this was fairly easy to achieve in this case.

I’ve been using this for quite a few months now.

The stylesheet

Recently, while screen sharing with a friend, they asked me about my GitHub notifications doing this really cool thing. That’s when I realised that I had completely forgotten that this was something I had been injecting into the page. I shared it with them and realised that some other people might find this useful too.

If you also find this change to be an improvement, the CSS blurb that needs to be injected is provided below. You can add this via Refined GitHub’s preferences page or use a more generic extension like Stylus to inject this into the GitHub Notifications page.

.notification-list-item-actions {
  right: unset !important;
  left: 4rem !important;
  padding: 0 8px;
}

.notifications-list-item:hover .note.notification-list-item-hide-on-hover {
  visibility: visible !important;
}

.notifications-list-item .notification-list-item-link [id^="notification"] {
  margin-left: 0;
  transition: margin-left 100ms cubic-bezier(0, 0.8, 0.2, 1);
}
.notifications-list-item:hover
  .notification-list-item-link
  [id^="notification"] {
  margin-left: 8rem;
}

.notification-list-item-link > :nth-child(1) {
  align-self: unset !important;
}
.notifications-list-item
  > div:nth-child(1)
  > div:nth-child(1)
  > div:nth-child(1) {
  align-self: unset !important;
}

This is basically my research document as I am exploring what I want the release process (and supporting development workflow) to look like for the Python packages I maintain, so that I can reduce the overhead caused by the various differences in these projects as well as the need to make all these decisions myself. I’m posting this publicly so that I can get feedback from a few people I know about this and whether they think it makes sense.

None of this is related to the whole “what build-backend/environment management/script runner tool should I use” situation but rather it’s everything “around” that. I’m not going into the test running or documentation building aspects here, “just” the mechanics around making a package release.

My design goals

This is basically “what I want” in terms of using existing communication channels, ensuring consistency and the development workflow.

• Have the version number managed as-is in the source tree.
• Have the version number be {last_release+1}.dev{N} during development.
• Have a git tag pointing to the commit with a release’s code.
• Have a changelog auto-generated based on the Git history or source tree.
• Include the changelog in the documentation at the release tag.
• Publish to pypi.org via Trusted Publishers, triggered automatically but requiring manual approval¹.
• Publish a GitHub release.
  • Include the changelog in the GitHub release description.
  • Include the release files on the GitHub release.
• The git tag and GitHub release must not be published if the release cannot be uploaded to PyPI.
• Releasing to PyPI requires a manual approval even after PR merge.

What those design goals imply

• changelog:
  • needs to be maintained in Markdown, specifically the intersection of GFM and MyST.
  • needs to be generated before making the commit that the release would be tagged on, since the documentation within the release should contain the generated changelog.
  • needs to only describe what happened between this release and the last one (so that it makes sense in the GitHub releases UI).
• PyPI release:
  • needs to happen on GitHub Actions, and the release needs to be triggered by a git tag (with protection rules on git tags).
  • needs to also upload the files to GitHub release.
• version number:
  • need a figure out the logic for bumping to next development version.
  • need some way to detect when a PR changes the version number and determine the commit that should trigger a release.

Let’s figure out the details now

The specific flow that seems to be needed here is:

• Merge a PR that bumps the release version.
• The merge will trigger a “release” workflow that has the ability to push a commit (containing the changelog) and trigger a different “publish” workflow.
• The triggered “publish” workflow needs approval to run and has the ability to publish to PyPI, push a git tag, publish a GitHub release and include the description as well as release files².

Changelog is not the same everywhere

Well, the changelog in documentation needs to do some things that the GitHub UI would do for “free”: have a link to the original diff between releases and a link to individual PRs in changelog entries.

Additionally, the changelog in documentation should also have a preview when looking at “latest” documentation, to communicate how things would work.

Deciding on how to detect version number changes

There’s basically 2 options here:

• Try to look at the source of truth that lives in the source code (like src/package/__init__.py -> __version__ / pyproject.toml -> project.version).
• Look at the relevant PR and bump the version based on some immutable thing about the PR.

I think the first is a better experience, since it’s a bit more intuitive – the version number changed and a release was automatically made. With this, an obvious place to make the release tag would be the “first” commit that bumps the version, which would need to vary based on merge strategy…

• merge commit: Use the merge commit into main, that has the multiple parents.
• squash merges: Use the first commit that bumps the version.
• rebase merges: Use the first commit that bumps the version… but wait…

Oh heck, rebase merges

Ohk, let’s think through this. I have a PR that contains 3 commits:

• 3pstrel Awesome change after the version bump
• 2verbmp Version bump
• 1prerel Awesome change before the version bump
• 0staquo (main) Status quo

Assuming that this is rebase-merged, I think the release commit should be based on top of 2verbmp. But, the only way to have it make sense with 3pstrel is to have the release commit be based on 2verbmp and then later to do a merge commit – which defeats the point of rebase merges (linear history).

I guess there’s 2 options here:

• Detect such cases and error out.
• Don’t bother merging back the release tag commit.

If we don’t merge back the release tag commit, then the release process needs to not require any code change other than the version bump. The changelog needs to be managed out-of-tree somehow (eg: in git?). I like having the ability to update a changelog entry that was contributed by a fly-by contributor after the PR has been merged, so this is a no-go for me.

In terms of detecting such cases, I guess I could have a GitHub Actions check that there isn’t a commit in the PR after one that bumps the version.

Looking at the source of truth

Doing a quick search on the internet for how to find when a line has changed with git, I found a useful StackOverflow question which points me to git log -L <range>:<file>. Trying it out on pip tells me this is a good option:

❯ git ls -L 3,3:src/pip/__init__.py -q
c0cce3ca6 Bump for development [Stéphane Bidoul]
e3dc91dad (tag: 23.3) Bump for release [Stéphane Bidoul]
b6a267059 Bump for development [Paul Moore]
a3c2c43c5 (tag: 23.2) Bump for release [Paul Moore]
2fd3e408d Bump for development [Paul Moore]
6424ac460 (tag: 23.1) Bump for release [Paul Moore]
d21af1c98 Bump for development [Pradyun Gedam]
368c7b4c5 (tag: 23.0) Bump for release [Pradyun Gedam]
c8ae28001 Bump for development [Paul Moore]
0a76da3a9 (tag: 22.3) Bump for release [Paul Moore]
2132eb4cd Bump for development [Stéphane Bidoul]
8e7e76e60 (tag: 22.2) Bump for release [Stéphane Bidoul]
88d565cc3 Bump for development [Pradyun Gedam]
3c953322c (tag: 22.1) Bump for release [Pradyun Gedam]
6012b48e5 Bump for development [Pradyun Gedam]
9b203d5af (tag: 22.1b1) Bump for release [Pradyun Gedam]
0a916125e Bump for development [Pradyun Gedam]
1742af7bd (tag: 22.0) Bump for release [Pradyun Gedam]
2bf32f5f9 Bump for development [Pradyun Gedam]
abec8a701 (tag: 21.3) Bump for release [Pradyun Gedam]
[...]

And, that the existing automation around pip releases is quite nice.

I still need to figure out how to gracefully handle changes to the source of truth for versions as well as moving it around without logical changes (eg: editing the docstring, or changing quoting etc).

Oh, this reminded me about pip’s releases…

Let’s think about pip

I’m not the only one who makes the calls on this but I think it would be nice to move pip to Trusted Publishers for publishing releases. Looking at the documented pip process, the main things that’d need to different if it adopted this release process would be:

1. Update get-pip.py’s GitHub repository to reflect the new pip release on PyPI.
2. Update an AUTHORS file.

Those don’t seem particularly bad. For 1, we’d need to tweak the generation logic over on get-pip.py to listen for repository_dispatch and trigger that event manually with the relevant payload and that’ll need pypa/pip to store an access token to pypa/get-pip that is only exposed for that specific trigger, which… seems doable.

What needs to be created?

• 3 GitHub Actions reusable workflows for individual “tasks”
  • A “release” workflow that determines if a release needs to be made and triggers the “publish” workflow via a workflow_dispatch.
  • A “publish” workflow that does all the mechanics of making the release.
  • A “release-check” workflow that checks the commit situation to protect against “commit after version bump” edge cases.
• 1 GitHub Actions reusable workflow for consolidated handling in project CI, to enable the project’s release.yml to be a short file that’s “just” on: [...] and uses: [...].

Naming: The second hardest problem in computer science

I feel like I should come up with a name for this, because it’d make communicating about this easier for me and because this workflow feels like it would be useful to more than just me.

This is automation that makes a release when it detects by a version change in source code and publishes to PyPI and GitHub, with an auto-generated changelog.

For now, this is a problem for future me and depends on whether he decides that he needs to maintain another piece of reusable software.

PDM claims to implement PEP 582. However, if you look at what it implements, it is something completely different from the standard.

PDM’s file system structure

<root>
    __pypackages__
        3.10
            bottle
    myscript.py

PDM will scan for the __pypackages__ directory up to 5 folders above the “current” one.

PEP 582’s file system structure

<root>
    __pypackages__
        lib
            python3.10
                site-packages
                    bottle
    myscript.py

PEP 582 says that the __pypackages__ directory should be next to the script being executed, and there’s no discovery logic “above” the script.

How did this happen?

PDM implemented a draft PEP. Draft PEPs are not final, and are subject to changes. However, PDM cannot evolve with the PEP since that means frequent breakages to the user experience as the standard evolves.

This leads to a situation where PDM is advertising implementing a (draft) PEP, while not implementing that PEP since the draft has evolved.

Why does this matter?

Well, outside of the issues with implementing draft proposals in workflow tooling, I think this is reflective of the issues with the competitive nature of the Python packaging ecosystem’s workflow tooling space.

I’m willing to trust that PDM’s authors are well intentioned and didn’t intentionally aim to end up with “false advertising” in the first feature the project lists in the README.

Besides, I only took a closer look at this because I was talking to one of PEP 582’s authors recently, and LWN¹ linked to someone on the internet was claiming that PDM is a solution to Python packaging.

Note: I’ve updated this to cover an aspect of the recent LWN article on the topic as well.

TL;DR

This post is a bit long, so here’s the key points I’m making:

• The Python packaging ecosystem unintentionally became the type of competitive space that it is today.
• The community needs to make an explicit decision if it should continue operating under the model that led to status quo.
• Pick from N different tools that do N different things is a good model.
• Pick from N ~equivalent choices is a really bad user experience.
• Picking a default doesn’t make other approaches illegal.
• Communication about the Python packaging ecosystem is fragmented, and we should improve that.

Python users are not software engineers

Many of the users who write Python code are not primarily full-time software engineers or “developers”. They are not particularly interested in this aspect of their job. They’re using Python as a tool to get their job done. They’re not interested in the details of how the tool works, or even how complicated things are under the hood.

As Thea (Stargirl) Flowers said recently:

The reason there are so many tools for managing Python dependencies is because Python is not a monoculture and different folks need different things.

User expectations for a “default” workflow

A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), cargo (Rust), gem (Ruby), pub (Dart), dotnet (C#/.NET), etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post.

Certain other ecosystems have this in their “default” tool, providing a much more streamlined experience for users. I have first hand experience of this for NodeJS and Rust, where they have a single tool that users invoke to do the majority of their work:

• create a new project.
• install/manage dependencies.
• run their project w/ those dependencies.
• test their project.
• publish their project.
• more?!

Today, each of these pieces is a separate tool for Python and doesn’t have a strict 1:1 mapping to the “best practices”/“secure” workflows. This is at odds with the expectations that these users have. This class of users, by and large, want consolidation and a single-tool experience.

We know that this class of users exists because we have:

• a number of popular tools that are attempting to provide this experience.
• results from user surveys we’ve done¹ clearly indicate this as well.
• had multiple community members say this in multiple ways, across multiple channels.

Flexibility leads to complexity

Brian Skinn said recently:

You can package darn near anything in Python, even though it may take figuring out a complicated three-step-and-a-hop process to get there… and I suspect that this has been part of what’s enabled Python to grow into its “second best programming language for every task” aphorism.

I think there’s some truth to that. “An Overview of Packaging for Python” on packaging.python.org expresses a similar tone as well:

Packaging in Python has a bit of a reputation for being a bumpy ride. This impression is mostly a byproduct of Python’s versatility.

I do want to contrast this with the fact that the overview page takes over 3000 words, to provide the “high-level” overview for how one can approach packaging a Python project. It doesn’t even touch the specifics of configuration or provide any specific workflow guidance with that much digital ink.

The bumpy ride reputation is not misplaced, and is the most frequent user complaint (more on this later). There are consequences to the degree of flexibility afforded to users by the Python packaging ecosystem:

• every Python project has to make multiple decisions about how they want to do certain things
• every Python user has to make choices for how they wanna manage their Python installation and workflow tools
• it leads to multiple ways to achieve the same thing when trying to use Python packaging tools, with some of these ways being subtly wrong
• it leads to a larger surface area of behaviours of existing/established tools that users rely upon.
• it makes it much more likely that new users hit edge cases and paper-cuts, that more-experienced developers won’t hit because they have adapted their workflow to avoid certain failure modes over time.

The flexibility is great to have when you need it but, without a “default” workflow, it serves to create more user confusion than it resolves. It contributes to the bumpy ride reputation and to the perceived complexity.

Placing the Python packaging ecosystem on the community spectrum

When I was reading “The community spectrum: caring to combative” - Insight From Alex Bayley on Sumana Harihareswara’s blog, it flagged something amusing to me. I recommend reading the article, but I’ll quote a portion that provides sufficient context for the rest of this post.

The Competitive Spectrum describes communities as being:

• Caring: members are motivated by helping each other.
• Collaborative: members share goals and help each other to achieve them.
• Cordial: members have their own goals which do not conflict with each other.
• Competitive: members share the same goals, and compete against each other to achieve them.
• Combative: members must achieve their goals by preventing others from being doing so.

PyPA and Conda

Within this spectrum, I think the relationship between Conda and PyPA projects is definitely collaborative.

The two groups of maintainers have worked together to solve problems that affect both groups. Conda packages are often built up from Python packages that are built with PyPA tools. Heck, at the time of writing, one of the founders of the PyPA currently sits on the Conda Steering Council.

Python build-backends

The goal of enabling pyproject.toml-based builds² was to move Python packaging from a collaborative (or at least, cordial) model, to a competitive model for the build mechanisms.

The stated goal of PEP 517 is:

The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods.

Moving to a competitive model for build mechanisms was intended to enable the ecosystem to move away from the “only one way” of building Python packages (the quote is from the Zen of Python) because the implementation we had of “only one way” was exceedingly difficult to evolve.

Poetry / PDM / Hatch / PyFlow / etc

These tools are firmly in a competitive model. They’re competing for users. They’re competing to be the “best” solution to the “workflow” problem. They’re, arguably, even competing for contributors.

Other than the obvious sign that these tools can’t be used together on the same codebase (mostly), this can be seen in other aspects of these projects:

• the way they’re marketed/documented – they have an incentive to invest in this, because they’re competing for users and good-looking/flashy documentation is a good way to attract users.
• the way they do community management: some have dedicated community Discord servers, mention $tool-specific ecosystems, have their own $tool plugin ecosystems, etc.
• the way their users advocate for them as the one-true-solution on the internet :)

This competition also leads to incentives for projects to do things like implementing draft standards that are not accepted or settled and claims that standards are implemented, even when the implementation does not match the standard.

Unintended competition

In my opinion, ending up with multiple competing workflow tools in the Python ecosystem was not an intentional choice by any individual or group.

While providing alternatives to distutils/setuptools was the intent of pyproject.toml-based build systems, I don’t think it was intended nor was there ever consensus that we wanted to end up with an ecosystem of competing tools which use the pyproject.toml-based build system and provide an end-to-end workflow.

I don’t see any discussion of such tooling in the corresponding mailing list discussions and the PEPs certainly don’t talk about trying to enable building alternative workflow-related tooling. There’s extensive discussion about the technical design of the final solution and on many aspects of “how to build distribution files”, but there’s no discussion about how competing workflow tools would be enabled.

Another reason is… well… let’s dig into some “history”. The Python Packaging Authority has publicly written goals for them:

• Although it’s still being defined, to work towards a “Meta-Packaging” system that:
  • Clearly delineates the phases of distribution
  • Allows for multiple interacting tools vs one monolithic tool
  • Specifically allows for alternative build systems, i.e. a “MetaBuild” system.

These goals were written in a different “era” of Python packaging, before PEP 516/517/518 were being debated, when the ecosystem was still in an entirely collaborative/cordial model (on the other side of a combative era³), and when the “Python ecosystem” was much smaller than it is today⁴.⁵

The intent was to enable the ecosystem to build multiple tools that interacted with each other and allow for alternative build tooling to solve problems that were difficult to solve with distutils/setuptools.

A competitive ecosystem for workflow tooling is an unintended consequence of the pyproject.toml-based build system. The PyPA’s focus on standardisation made it easier to build workflow tooling that interacted with other packages, with no mechanisms to check whether these tools are reciprocating on interoperability.

On existing workflow tools

I expect the most relevant people already know this, but I’ll state it explicitly: I have a lot of respect for the work done by the authors and/or maintainers of tooling like Poetry, PDM, Hatch, Pipenv, PyFlow, etc. I think they each, individually, contribute meaningfully and positively to the ecosystem.

The most popular workflow tools for Python handle the underlying details for the user and give them a single unified tool that has install / publish / run commands. They also provide functionality that the “default” tools do not (eg: environment-agnostic lockfiles, automated environment management etc). Serving as an end-to-end tool enables them to trim the scope and define it as they deem appropriate.

Conda also does this to a certain extent, by tying the environment management and package management together into a single tool, but it also operates on a different level than these tools – packaging “everything” rather than just Python packages.

The reasons for the existence of workflow tools

Some/all of the “workflow tools” that exist today because the “default” tooling did not cover more of the user’s workflow with a single piece.

• Pipenv’s Pipfile was created with the express goal of being for pip⁶.
• Poetry was created to provide a single config file with a single tool experience, and a better dependency resolution model.
• Hatch was created as “a productivity tool designed to make your workflow easier and more efficient, while also reducing the number of other tools you need to know.”.
• PDM was created as a “Python package manager with PEP 582 support” (which is notable, given that PDM does not implement PEP 582).

These tools have all now dropped that language from their documentation (or at least evolved it) to reflect that they’re now focused on providing a complete and unified experience for the user. This can be seen by the fact that they’ve all gone ahead and invented their own build-backends (except Pipenv, which is maintaining a fork of pip within it), since providing a complete and unified experience requires that the tool also controls how projects are built.

Now, to state the obvious, the folks who created these tools are not fools who like to create work for themselves or enjoy reinventing the wheel. They created because they felt that the existing tooling wasn’t meeting their needs and that there was no clear path to improving the pre-existing tooling to meet those needs. Each of these tools solves the problem by making different calls for what the right trade-offs are.

I am certain that it is not possible to create a single “workflow” tool for Python software. What we have today, an ecosystem of tooling where each makes different design choices and technical trade-offs, is a part of why Python is as widespread as it is today. This flexibility and availability of choice is, however, both a blessing and a curse. That’s actually a great segue to talk about…

pip: A privileged player

Today, pip is uniquely positioned within in the Python packaging ecosystem. It is the only⁷ piece of the Python packaging tooling that ships with Python, and is guaranteed⁸ to be installed in every environment. Nearly every Python user⁹ who wishes to share code (or use shared code) uses it today, directly or indirectly.

The fact that I’ve added 3 footnotes in the last paragraph is a symptom of something and I’m not sure if it’s a good thing or a bad thing.

pip: A disadvantaged player

This point is easier to make with an example, let’s take lockfiles: it’s technically feasible to implement an arbitrary lockfile format in pip, that evolves with pip, in lock-step with it (something of this sort is implemented already in the form of pip-tools).

Given the privileged position that pip has within the ecosystem (i.e. ships with the language), whatever it does would become the de-facto standard and commercial tools/IDEs will add support for that model much quicker (eg: requirements.txt) than something similar from PDM, Poetry, Pipenv etc.

Now, on the face of it, this is a completely different direction from PyPA’s model of “interoperability standards through concensus” because, effectively, whatever pip implements would become a de-facto standard for what the ecosystem and tooling supports.

On the other hand, if we focus on designing for interoperability through concensus before implementing functionality in pip, implementing vital workflow improvements is now blocked on an exhaustingly long process of a non-iterative, waterfall-style design process. Further, an interoperable lockfile format also has to try to satisfy needs of tools that use completely different resolution models, even semantically incorrect ones¹⁰.

Taking a slightly iterative approach of “we’ll cover the more complicated case later” caused the last proposal for a lockfile format standard to be rejected after months of discussion.

We’re in a state where the process for adding new functionality to our default “out of the box” experience is designed to be both very difficult and very “energy” intensive – leading to it being pretty slow. The reason the default tooling doesn’t improve is that making meaningful improvements to it is blocked on trying to cater to all workflows; in ways that alternative tooling is not.

The effect of this is that our “out of the box” experience is always going to be worse than the experience with other tools in the ecosystem. In a competitive ecosystem, is that what we want? Heck, at that point, do we even want to be in a competitive ecosystem?

On build-backends tied to workflow tools

Hatchling, pdm-backend and poetry-core are all examples of this. Flit/flit-core is another slightly-weaker example of this¹¹. They each have build-backends that are either (a) tied to a particular workflow tool in some way or (b) promoted along with a workflow tool.

The build backends are all solving the “build a wheel from Python code” problem, but with different user experiences tacked onto them. Building multiple tools that solve the same problem is duplicated effort.

Part of the problem here is that these tools (except for PDM) are not built with interoperability in their design, and these tools have basically little to no incentive to take on the complexities of providing interoperability.

Flit can only be used with flit_core, and flit build doesn’t build the same artefacts as python -m build would.

Hatch is tied to hatchling, and “it would be an extraordinary amount of effort” to support using a different build-backend for your project when using Hatch.¹²

Poetry has its own dependency specification format, and the corresponding build-backend enables it to build packages that use that format.

PDM is better on this front, in that it has greater backend-agnostic behaviours to it. However, pdm-backend is undergoing a rewrite to “provide a similar extensible interface to hatchling”.

These were all made possible by the explicit focus of the PyPA on designing an interoperability model – i.e. Unix-like approach – which these tools have used to create tool-specific build-backends. :)

While trying to enforce that one way of building packages/managing dependencies for all Python users is not feasible… having 4 build-backends that all handle pure-Python packages with the same file holding their configuration, while providing slightly different user experiences is also not a good place to be in IMO.

Pick from N ~equivalent choices is really bad UX

When you package Python software, a user has to make a lot of choices. There are a lot, and I do mean a lot, of “A vs B” comparisons that you can make when it comes to figuring out the scaffolding for packaging and distributing Python software.

The problem with not making a default recommendation for these largely-inconsequential choices is that it means that every user has to make these choices. Instead of making a choice once and then being able to build upon that as an ecosystem, we keep moving in circles on these topics because we’ve got two groups now and picking either choice means that the other group is unhappy.

This also leads to the same problems being solved twice – there’s duplicated effort and duplicated documentation. Each project will design their own approach and there’s incentive for projects to try to “out-compete” each other by providing more features or by providing better documentation, rather than contributing to improving a common corpus.

This is how, for example, we end up with packaging.python.org not having a standard structure for declaring metadata that is implemented by ~every build-backend documented, even though there was extensive documentation for the same in multiple tools’ own documentation for months. Each of those tools’ authors have had incentives to document it for their users and it was easier to do so in their own documentation, where they don’t have to worry about the concerns of other tools or “being generic enough”.

“not a PyPA project”

The only reason various Python packaging projects (notably, Poetry and PDM) are not PyPA projects is because they’ve never asked to become one.

As it stands, the PyPA views itself as a big umbrella. Basically, any established Python packaging project that asks to be included, will be accepted. If Poetry and PDM ever asked to join, as it stands, there’s no version of this timeline where the existing PyPA members say no.

From the discussions I’ve had, the reasons have ranged from some sense of maintaining control (which doesn’t really have good precedence), to logistical issues like GitHub Actions queues, as well as a sense of being able to “be successful without the tag”.

Also, to say that these tools are “are not participating in the PyPA”¹³ is incorrect. PDM’s whole pitch today is that it is “A modern Python package and dependency manager supporting the latest PEP standards”. Poetry’s authors somewhat regularly interact with the interoperability discussions, and its original author has even co-authored a PEP.

On the Python Packaging Authority

I think there’s a need to reconsider what the Python Packaging Authority should be trying to do and what it even is. We’ve been cruising on the premise that we’re maintaining foundational tools and designing for interoperability is the “right” model for the Python packaging ecosystem. I’m not sure that’s the case.

Between the user surveys, having a $work role where I’m directly influencing user workflows beyond the installer, spending time helping out with scientific Python tooling and with pyOpenSci, and the discussions in the strategy thread… I’m starting to think that our current approach is not working and is harmful¹⁴ unintentionally. Each projects’ maintainers effectively decide on different aspects of the overall UX. Each project acts as its own project. There is no broader guiding “roadmap”. Making decisions about how the default tooling of the ecosystem works is “not appropriate” for our process to hashing out technical design proposals. There is no “blessed” tool and yet there are defaults, things that ship with the Python standard library and PyPA recommendations.

I’m not sure what the right answer is, but I don’t think we’re in a good place right now. Here’s where we are:

• even co-operating tools are viewed as being in competition with each other
• interoperability standards that are written but are not enforced
  • on the principle of “consenting adults” or “be permissive in what you accept”
  • for backwards compatibility reasons
  • because “we should trust the XYZ authors to do the right thing”
  • difficult to answer basic questions like “what can a source distribution file be named” because the standard says one thing while the tools do something else because “the author prefers that”
  • implemented and publicized as features in tools, despite not being something that is “accepted”
  • absolutely ignored by tool authors “because it’s not a priority”
• duplicated effort because multiple tools are competing
• users are confused about what to use, what is deprecated, what is the “right” way to do things etc and there is no authoritative answer
• say that PEPs are not documentation but, also, go read this PEP for the details on how this feature works because that’s the only place we wrote it.
• no clear answer for step 0 questions like where should I put by .py files relative to my pyproject.toml file
• a lack of willingness to draw a line in the sand and say “this is the way things are done” because “what if someone finds a better way in the future”
• we don’t pick one of two ~equivalent choices because “they’re both valid workflows”
• there is no agreed upon direction for the ecosystem

We don’t need more “generic” build-backends today

With setuptools gaining pyproject.toml configuration support and a standard structure for declaring metadata that is implemented by ~every build-backend, there isn’t a significant difference between the various tools for pure-Python packages.

There are only so many ways to construct a .zip file containing a bunch of pure-Python files + metadata. The only real difference is in the user experience, and the user experience is largely determined by the tooling that invokes the build-backend.

Building more build-backends that are intended to be extended (beyond what we already have, between hatchling, pdm-backend and setuptools) feels unnecessary to me; and we might already have one too many options for this.

On pip as a workflow tool

Donald Stufft has said on pip’s issue tracker:

All that being said, I think trying to follow the “unix philosophy” is a mistake and is actually a pretty poor UX. Yea a lot of nerds grok it because we’ve caused enough collective brain damage by being forced to use it over time and it works better for the typical unix tools because they generally just come preinstalled. I think it would just add additional complexity to an already confusing landscape of tools for our end users.

And, Paul Moore has said:

At some point, I do think that pip needs to make a firm decision on whether it’s a development workflow tool or just an installer

And, I think that now is the time to make that decision.

The recent strategy discussion is sprawling and has largely operated with the assumption that pip is not something that can become a “workflow tool”. I think that’s an incorrect assumption.

Personally, I’ve wanted pip to cover more aspects of the user’s workflow since before PEP 517 was implemented, which was around the time I started getting involved in Python¹⁵. Improving the “base” tool to cover more use cases is not a bad idea – it’s helpful for most user personas. Outside of blessing another tool, this is the path of least resistance that we can take to getting to a better state.

Do I think there’s a significant amount of work needed for making pip into a workflow tool?

Yes.

Do I think that the amount of energy that’s gone into Poetry/Hatch/Pipenv/PDM development, collectively, would have made more than a meaningful dent on this issue?

Yes.

Do I think that, at this point, blessing another tool is a good idea?

Maybe. We’ve built a competitive ecosystem, and I don’t think we can just “pick one” of the “new things” and expect that to be the end of it.

Did we ever have sufficient buy-in + capacity to do this with pip, along with contributor experiences that would facilitate this?

No. That’s been a part of the problem – we’ve made it fairly tractable to “build your own” in a sandbox that lets you ignore the need to support entire swaths of workflows, and that’s something you can’t compete with easily for contributor experience. And, when the alternative is “spend a few months trying to implement something in a ’legacy’ codebase, while catering to needs that you don’t have, also convince a bunch of people with limited availability that your idea is a good one and wait for them to review what you wrote”, it’s not surprising that we end up with a bunch of “new things” and have multiple groups building multiple workflow tools.

We still don’t have agreement that this is the direction that we, as a community, want pip to go.

On user-facing communication

There’s no single place where users can go to get information about the Python packaging ecosystem – either on how it’s evolving or what the functional best-practices are today. We either (a) don’t have them documented or (b) don’t have a good approach to communicating about this to end users.

There’s a cost to this.

LWN recently directed readers to a blog post¹⁶ that claims that the strategy discussion is evidence that “PyPA must be destroyed”, implies that there’s “ivory towers of packaging tool maintainers”, that “half of the discussion participants did not even bother reading what the people think” based on a misunderstanding of how the discussions have occurred¹⁷ and that the Python Packaging User Survey somehow happened in a vacuum (it was extensively discussed with lots of input from PyPA members).

That blog post has captured the current discourse around Python packaging and set the tone: one painting the volunteers who currently maintain the tooling as being “vs reality”.

This is exactly the sort of thing that happens when there’s no authoritative voice in the space: the vacuum will be filled by someone else on the internet, who will likely be making sensational claims that aren’t being validated before being repeated.

On formal UX analysis

This has been mentioned in multiple places and has come up in the past in other contexts around Python Packaging.

I think doing “full UX analysis” is going to be forbiddingly difficult. Don’t get me wrong: a complete UX review of the Python Packaging ecosystem would be awesome as part of a coordinated effort to make progress on the fundamental problems here.

Something like this is unlikely to happen because there’s a really motivated UX expert with a lot of volunteer time to donate. We’d basically need an enormous cheque for work that’s… “understand Python packaging really well and figure out a path to making it better”. That’s a difficult thing to tell a funder to throw money at.

Notably, there are a lot of stakeholders here: the easiest “persona” to identify is the maintainers of the tools themselves. After that, it starts to become fuzzy quickly. There’s redistributors, end users, Linux OS distros, Linux-specific non-OS package managers, cross-platform distributions, direct users, corporate users who have their own internal packaging systems, or like astronomers, students, statisticians, business analysts, and more.

Breaking the UX problem into a smaller piece, like a single point from the dimensions we could unify, makes this a much more meaningfully sized piece for seeking funding toward. Even then, it’ll still probably only be available to be funded by the larger wallets and likely need to be a part of a project that has some other deliverables.

Acknowledgements

I’d like to thank the following people for reviewing drafts of this post at various stages and providing valuable feedback: Donald Stufft, Kushal Das, and Pavithra Eswaramoorthy.

Also, I would’ve appreciated if the discourse on this wasn’t moving along at the speed that it’s been moving after I publicly committed to writing this. And, yes, I’m aware that some of the things I’ve said here are conclusions that been reached by the broader group on that thread.

Finally, I do have more thoughts; especially on how to get to a better place, but ~6k words is about as long as I want to go here.

The Python Packaging Authority

The Python Packaging Authority (PyPA) is a fairly loose group of projects that happen to be related to Python packaging. While the PyPA has a formal governance model, there’s no “oversight” or “enforcement” on projects that are part of the PyPA – they are only required to adopt the PSF Code of Conduct and to be accepted by the existing members.

Functionally, each PyPA project is free to do what it wants. The only real benefit of being a PyPA project is the ability to use the PSF as a fiscal sponsor and the ability to say that it’s a “PyPA project”.

Practically, the PyPA serves as “a body to hammer out & maintain interoperability specifications” for Python packaging. The PyPA also includes foundational tools that are used in the Python packaging ecosystem, like pip and setuptools.

Relationship with “Core Python”

(I’m using “Core Python” to refer to the Python language and standard library, as maintained by the CPython Core Developers, also known as “core team” for Python)

As of the time of writing, Core Python’s involvement in Python Packaging (in a post-distutils world) is fairly limited. The interpreter and Python standard library provide the following pieces, around the packaging ecosystem:

• Shipping ensurepip and venv in the standard library.

  • venv supports creating a virtual environment with pip and setuptools¹ installed in it.
  • ensurepip provides copies of pip and setuptools¹ that are used by venv, and allows bootstrapping the packaging tooling for Python.

• “site-packages” directories on the import path.

  • site adds site-packages (“global” and “user”) to the import path.
  • site respects the presence of a pyvenv.cfg file (used by venv) to mark a virtual environment, influencing which site-packages it adds to the import path.

• sysconfig provides relevant paths for placing files and build configuration for the interpreter.

• a stable ABI for the C API, for building extensions compatible across Python 3.x (x >= 2).

Delegation of Python packaging ecosystem decisions

The Python Steering Council delegates the decision making for the packaging ecosystem to the Python Packaging Authority (PyPA). This is done through standing delegations to specific PyPA members on specific aspects of Python Packaging. The PyPA is, like CPython Core Developers, a group of volunteers who maintain various bits and pieces of the Python packaging ecosystem.

The CPython Core Developers and the PyPA are not the same group of people, but there is meaningful overlap between the two groups. There are 85 “active” CPython Core Developers² and 50+ (public) PyPA members³. Off the top of my head, I can count 5 people who are active in both CPython and one or more PyPA projects.

A bit of history

From https://www.pypa.io/en/latest/history/:

2011-02-28: The Python Packaging Authority (PyPA) is created to take over the maintenance of pip and virtualenv from Ian Bicking, led by Carl Meyer, Brian Rosner and Jannis Leidel. Other proposed names were “ianb-ng”, “cabal”, “pack” and “Ministry of Installation”.

As “Ministry of Installation” likely implies, these names were chosen partly in jest. The name stuck and, those who started using Python after this name was picked, ended up treating the name at face value. PyPA projects essentially serves as all of the packaging infrastructure for Python making it the de facto authority.

Over time, “Authority” in PyPA did end up being backed by real authority: the standing delegations from the elected Python Steering Council, ceding control of certain kinds of Python Enhancement Proposals (PEPs) to specific PyPA members. There’s formal processes, formal interoperability specifications as Python PEPs and a formal governance model.

To quote Thomas Kluyver:

You don’t get much more authoritative than that without an army.

What about non-PyPA projects?

Since PyPA’s inception, there have been non-PyPA projects related to Python packaging (eg: buildout is older than pip). By and large, there isn’t any sort of antagonistic relationship between PyPA and non-PyPA projects. PyPA and non-PyPA project maintainers have often worked together on various things, and many non-PyPA projects are listed as “key projects” for Python packaging.

For example, at the time of writing, the two most popular non-PyPA projects are Conda and Poetry. Both of these rely on the interoperability specifications and/or tools that the PyPA works on (conda, poetry) to do what they do.

Since it was on the top of my mind¹, I decided try to nudge ChatGPT into the direction of giving me a useful answer about error message design for command line tools.²

Now, tell me what an excellent error message from a command line tool should contain.

An excellent error message from a command line tool should contain the following elements:

1. A clear and concise description of the error: The error message should explain what went wrong and why the command failed.

2. A suggestion for how to fix the error: If possible, the error message should provide a suggestion for how the user can fix the error and try the command again.

3. The error code: It can be helpful to include an error code or other identifier that can be used to look up more information about the error.

4. The context in which the error occurred: Providing information about the context in which the error occurred (e.g., the command that was being run, the arguments that were provided) can help the user understand the root cause of the error and how to fix it.

5. A link to additional resources: If there are additional resources that can help the user understand and fix the error (e.g., documentation, forums), it can be helpful to include a link to these resources in the error message.

I’ll admit this was mildly impressive on the first skim-read. Looking closer, it became clear that the response is actually flawed in multiple ways and the model made some assumptions that are not completely accurate.

Overall, this is amusing enough to me that I decided to document it here.

Further reading

• https://abnormalsecurity.com/blog/double-edged-sword-of-chatgpt
• https://stackoverflow.com/help/gpt-policy
• https://twitter.com/emilymbender/status/1603039301864157184
• https://mastodon.social/@Gargron/109469945447325370 ;)

On Dec 27th 2022 ~9pm UTC, I published an update to the “about” page on this website that (wrongly) claimed:

My work has been recognised by the Python Software Foundation, and I have been awarded the PSF Community Service Award in 2020.

I haven’t received the PSF Community Service Award. It wasn’t mentioned in the listing below this sentence, of the various awards I’ve received and the community work I’ve involved in.

This stayed up for roughly 4 days, until Dec 31st 2022 ~6pm UTC, when I noticed it and removed it¹.

How this happened

I recently updated various pages on this website. I also updated the about page, to use a nicer design, data model and layout.

This sentence was a part of a multi-sentence suggestion I had created using a large language model, similarly to how I’d come up with sentences for some of other other sections on the page. I had pasted this specific suggestion into the document with the intent of removing this sentence and using the rest of it. I, however, stepped away from the computer before doing so.

I came back to this a few days later and, without looking carefully, committed the whole thing and published it before closing the editor window.

What I’m doing about it

I don’t think anyone noticed this mistake except for me, so this might’ve just been a “victimless crime”. However, I tend to hold myself to a high standard when it comes representing my own work accurately, and this doesn’t hold up to that.

There isn’t much I can do about this now, except to apologise for the mistake and document the fact that I’d made this mistake – which is what this post is.

Packages with native code are a clearer win, because the wheel file will contain pre-compiled binaries for the platform you’re installing on. This means that you don’t need to have a compiler and non-Python build dependencies installed, and you don’t need to wait for the compiler to do its thing.

The quick answer

Installing from a wheel is analogous to unpacking a zip file. When installing from a source distribution, pip will actually build a wheel from the source distribution and then install using that wheel. It’s the same operation at the end of the day, but there’s a bunch of extra “work” involved in building a wheel from a source distribution (which contributes to making things slower).

The picture answer

The wordy answer

When installing from a wheel, pip will fetch the wheel file, and then unpack it. That’s it. There’s nothing else to do.

When installing from a source distribution, pip will fetch the source distribution, unpack it to a temporary directory, (potentially) create a build environment and install build-dependencies in that environment, make a subprocess call (or multiple) to the build-backend to get it to generate a wheel file. The build environment and unpacked sources are then deleted. We now have a wheel file which will then be handled like any other wheel file.

Further reading

• How pip builds packages from source
• Specification of the wheel format
• Installer’s installation logic code, if you’re inclined that way
• pypackaging-native: “a collection of content about key Python packaging topics and issues for projects using native code” (which is what made me write this post)

Software

• sphinx-theme-builder: 0.2.0a15 release.
• sphinx-basic-ng: 0.0.1a12 release.
• Furo: 2022.06.04 and 2022.06.21 releases.
• Documentation improvements to pip.
• Made progress on moving pip away from distutils usage, unblocking Python 3.12’s distutils removal.
• sphinx-theme-builder got a bunch of updates.
• sphinx-basic-ng got a bunch of updates.
• Improve sphinx-theme-builder usability for downstream redistributors.
• Significantly simplified the default configuration file template in Sphinx, which eventually got released in Sphinx 5.1.
• Worked on getting pip’s test isolation to behave correctly without virtualenv <= 16 (from ~2019).
• Worked on Sphinx 5 compatibility for various bits of the Sphinx ecosystem that I maintain.
• Spent a decent amount of time working on Lutra.
• Built CPython documentation with Lutra.

Community

• Getting the Packaging PEPs into a consistent state.
• Start moving Python Packaging standards to packaging.python.org.
• Engage in a discussion about Bokeh moving off of setup.py.
• Get PEP 668 over the line and… it’s accepted!
• Start poking at PEP 625, to get the ball rolling on that again.
• General contributions to executable books and Sphinx.
• Gather a first round of feedback, on CPython documentation built with Lutra.

General Notes

This was backfilled in Sept 2022.

Software

• Release pip 22.1!
• Lots of bugfixes for pip 22.1!
• Make some progress on improving pip’s build system logic.
• Make improvements to Furo, switching it to use sphinx-basic-ng.

Community

• Remotely attend PyCon US Sprints.
• Attend and engage in the Python Documentation Community meeting.
• Collaborate on setting up a “Packaging PEPs” pages on https://peps.python.org/.
• PEP 691 (adding a JSON-based API to Python Package Indexes) went through its initial round of discussions. I co-authored this PEP.
• More work on getting PEP 668 (externally managed environments) ready for a decision. I co-authored this PEP.

General notes

This month was a vacation month, and it shows! I got to meet family and friends, and had a generally relaxed few weeks.

Technical

• Release pip 22.1.b1
• Release Furo 2022.04.07
• Release release-helper 1.0.0
• Improve the presentation of upgrade prompts from pip.
• Some exploratory work for how much effort it would be, to remove setup.py install calls from pip’s codebase.
• Some exploratory work for adding a “Track: Packaging” to Python PEPs.
• Some exploratory work for automating adding maintainers/committers to PyPA, with the information being stored in a Git repository with history.
• Did an “open source responsibility audit”, to check what I’ve committed to responsibility for and for what things I should step down.

Communication

• PyCon US Packaging Summit planning.
• Poke a few more people about PEP 668 (externally managed environments).
• Started writing a PEP for “default” extras (i.e. optional dependencies that get installed by default).
• Interviewed as part of a post about Package Managers

General notes

My plans for attending PyCon US changed very close to the event date, since PyCon announced that there will be a co-located event does not have any published masking or vaccination requirements. I’m pretty bummed out about that. Thankfully, everyone involved in the trip planning/booking has been understanding.

PyCon US is wrapping up, as I write this. I imagine there’s gonna be a bunch of folks with a bunch of energy contributing to various projects, something I look forward to!

Things have been fairly bumpy over the last little while.

Dec 2021

Technical

• sphinx-basic-ng: release alpha 8
• sphinx-theme-builder: release alpha 12
• pip: Significant usability improvements
  • Much better cleaner error presentation style, providing additional context in build-time error messages.
  • Abort immediately on metadata generation failure instead of backtracking.
• installer: Preserve executable bits, bringing it closer to behaving appropriately.
• installer: Improve the documentation.
• pip: Start rewriting the build isolation logic.
• oldest-supported-numpy: Improve testing setup for dependency declarations.
• pydata-sphinx-theme: Rework the theme’s build system, to use sphinx-theme-builder.
• pyproject-hooks: Create and scaffold the project, as a migration path for pep517.
• Started working on a tool to make writing these easier for me.

Communication

• Initiated the conversation for starting the Python packaging summit at PyCon.
• Nudged Discourse to enable the footnotes plugin on all hosted instances.
• Reworked PyPI support’s limit request templates.
• Became a maintainer on Flit.

Jan 2022

Technical

• pip: Release 22.0.
• Furo: Immaculate Indigo release.
• sphinx-inline-tabs: Release beta 11.
• sphinx-basic-ng: Release alpha 9.
• sphinx-theme-builder: Release alpha 13.
• pip: Significantly improve how errors in subprocesses are presented.
• Sphinx: change “permalink to this headline” to “permalink to this heading”.
• pydata-sphinx-theme: Improve presentation and styling of the scaffolding.

Communication

• Release management and change management for pip 22.0.
• Discussion with the Packaging Project manager about rolling out changes.
• Moving externally managed environments PEP forward.

Feb 2022

Technical

• ✨ Furo: Jazzy Jasmine and Keen Kobi releases, with major improvements.
• pip: Release 22.0 bugfixes.
• installer: Release 0.5.0
• pip: Make more improvements to pip’s documentation.
• pip: Get the fix for Windows upgrade protection logic landed.
• sphinx-themes.org: Rework the “Kitchen Sink”.

Communication

• Poke at PEP 668, to check that there’s still interest.
• Get the ball rolling for the Packaging Summit at PyCon US 2022.
• Move installer into the PyPA umbrella.

Mar 2022

Technical

• sphinx-theme-builder: Release alpha 14.
• installer: Release 0.5.1.
• Python: Upgrade the pip version that’s bundled with latest bugfix releases-
• Lutra: Publicly release the source code under MIT.
• pip: Explore changing wheel installation logic to use installer.
• pip: Explore changing pyproject.toml build system to use build.
• pip: Make progress on the broader build logic refactor.
• pip: Make more improvements to pip’s documentation.
• pip: Improve the pip upgrade prompt presentation.
• dotfiles: Completely rework my dotfiles setup, allowing “merging” multiple repositories.

Communication

• Update the PyPA recommendations around pipenv, to better reflect the current state of the project.
• Make a public proposal about adopting the concept of “Teams” (from PEP 8015).
• Ask publicly about blockers for dropping pip’s legacy resolver.

What next?

There’s PyCon US close to the end of the month! That’s the first in-person event that I’ve bought tickets for since 2020!

There’s going to be a Python Packaging Summit, at PyCon US this year – I’m one of the organisers. I am also going to be at the event for sprints.

Other commitments

A full time job that pays my bills. :)

These last two months have been fairly dense!

Oct 2021

Technical

• pip: 21.3 release.
• packaging: 21.1 release.
• sphinx-basic-ng: alpha7 release.
• resolvelib: 0.8.0 release.
• installer: 0.4.0 release.
• installer: 0.3.0 release.
• pip: Some benchmarking to see get a better idea of how stuff performs.
• pip: Start working on adding diagnostic details to pip’s error messages.
• pip: Documentation improvements.
• vendoring: Rework the license fetching mechanism, to utilise wheels whenever possible.
• installer: Modernise the project, and add Python 3.10 support.
• sphinx-basic-ng: Make some tweaks and improvements.
• sphinx-themes.org: Some TLC for the site.
• sphinx-design: Contribute improvements to tab presentation.
• Lutra: settle down on a name for my second Sphinx theme.

Communication

• PEP 655: More discussions. :)
• Help psf/black settle down on a stability policy.
• Build consensus on adopting rich in pip, amongst maintainers and various community members.

November 2021

Technical

• sphinx-theme-builder is now a thing! 🎉
  • Furo and pydata-sphinx-theme have adopted it!
• pip: Adopt rich! 🎉
  • pip will now use rich’s progress bar!
• Lutra: More progress on the Sphinx theme.
• sphinx-basic-ng: Adapt as I learn things from implementing Lutra.
• Sphinx: Start working on dropping vendored jQuery and underscore.js.
• pyproject-hooks: New project!

Communication

• Be the PyDev of the week! 🎉
• PEP 655: More discussions. :)
• Ensure that pygments and rich are OK with being vendored in pip.
• Discuss and explore how pip should behave in environments without setuptools.
• Start moving away from packages using PEP numbers, as their name.

General notes

The work for improving error messages in pip is coming along really nicely! This has been a great mix of frustratingly-difficult and visually-pleasing work that I’m glad I’m able to do.

As a reminder, the next Volunteer Responsibility Amnesty Day is almost here! If you are someone who volunteers for things, consider participating and checking in with yourself on December 21, 2021.

What next?

Well, I’m gonna take a break from OSS during the holidays.

• pip: Continue working on the diagnostic errors effort.
• Sphinx: More progress on $theme-I-am-writing.
• Sphinx: Make some progress on the jQuery / UnderscoreJS rewrite.

Other commitments

A full time job that pays my bills. :)

Work I did

This has been a fairly productive few months, despite all the things that have been happening outside of my open-source-related responsibilities.

July 2021

Technical

• installer: 0.2.3 release!
• packaging: 21.0 release!
• sphinx-basic-ng: Alpha 6 release.
• pip: Make progress on the documentation revamp project.
• pip: Make progress on adopting black.
• Furo: More Sphinx 4 compatibility stuff.
• Furo: Refactored most of the stylesheets.
• Furo: Implemented and released dark mode toggle, which was a heavily requested feature.
• Furo: Improve accessibility by adding appropriate labels across the site, improving constrast ratios across the board and validating the keyboard-only navigation story.

Communication

• Lots of PR reviews across all of GitHub!
• Onboard a new contributor (@DiddiLeija) to pip.
• Collaborate on the lockfile standardisation effort (became PEP 655).
• Help with ComputeCanada’s setup for providing optimised wheels to their users.
• Sphinx: experiment with improving the documentation site.

August 2021

Technical

• Furo: First stable release!
• sphinx-inline-tabs: Release beta10!
• pip: More progress on the documentation revamp project.
• pip: Migrate automatic merge conflict label mechanisms.
• pip: Improve deprecation helpers.
• pip: Improve issue filing experience.
• Furo: Add support for sphinx-design elements.

Communication

• Poke at the PEP 655 discussion.
• Poke at the GPU-optimised wheels discussion.
• sphinx-design: Provide feedback on the initial iteration.
• TOML: newline normalisation discussion.
• pip: Drive discussion and resolve the black adoption story.
• Furo: user support, helping out with their usage and docs.

Sept 2021

Technical

• pip: Lots of PR reviews and issue tracker trimming.
• pip: More progress on documentation revamp.
• Furo: Rework presentation of API documentation.
• Furo: General polishing, based on issues reported by users.
• installer: Work on modernising the codebase.
• installer: Land the PRs that users have filed for this.
• vendoring: Improve behaviours, for patching and import rewriting.

Communication

• pip: Discussion and kickoff of the effort to improve error messages.
• pip: Pick up the release manager duties for 21.3 (Oct 2021).
• installer: Communicate around the support promises for EoL Python.
• Furo: user support, helping out with their usage and docs.
• Executable Books: Get involved in the governance discussions.

General notes

I’m really excited about the progress on the pip documentation revamp. I’m particularly happy with how the Build System Interface section has come together.

Volunteer Responsibility Amnesty Day is on the next solstice: December 21, 2021. If you volunteer for things, consider participating and checking in with yourself.

What next?

• PEP 655: Help out with the rewrite.
• pip: 21.3 release!
• pip: Wind down the documentation rewrite effort.
• pip: Make progress on the diagnostic errors effort.
• installer: Make last EoL Python release.
• Sphinx: More progress on $theme-I-am-writing.
• Poke and check what’s up with the Python Documentation Workgroup.

Other commitments

A full time job that pays my bills. :)

Work I did (1 May 2021 - 30 Jun 2021)

Technical

• Made significant progress on pip’s documentation rewrite.
• More progress on installer’s usability.
• Finally completed the restructuring of pip’s distribution via https://bootstrap.pypa.io/. Shoutout to @ewdurbin who did most of the magic here. :)
• A lot of issue tracker triage (and mostly saying no) for Furo.
• Sphinx 4 compatibility for Furo.
• Wrote my own VS Code editor UI theme, finally.
• Sphinx: More progress on $theme-I-am-writing.

Communication

• PyPA logistics for moving pipx and cibuildwheel.
• Passively involved in the editable installs standardisation process.

If you wanna talk to me on a weekend, I’m still doing open social weekends. :)

General notes

Nothing to say here this time.

What next?

Some more wrapping up of ongoing projects, since I actually didn’t really do that as much as I’d hoped for.

Technical

• Make further progress on pip’s documentation rewrite.
• installer: Finalise and make a stable release.
• Sphinx: Continue iterating on sphinx-basic-ng.
• Sphinx: Continue iterating on $theme-I-am-writing.
• Sphinx: Continue iterating on the PEP 517 backend.

Communication

• Spend time on the Python lockfile standardisation effort.
• Help with the July pip release.

Other commitments

A full time job, that pays my bills. :)

Work I did (1 Apr 2021 - 30 Apr 2021)

Technical

• Initial release of sphinx-basic-ng is out!
• Initial release of installer is out!
• Revamped sphinx-themes.org!
• Made progress on the “sister theme” for Furo. Picked a name for it too!
• Made progress on a PEP 517 build backend for Sphinx themes, and told a few relevant folks about it.

Communication

• My social weekends are still a thing, and have been great so far! If you wanna talk to me on a weekend, click that link. :)
  • Continuing to collaborate on designing a lockfile format for Python.
• Got involved in discussions about modernising the Python documentation theme.
• Got involved in discussions about new tutorial in the Sphinx documentation.
• Did some post-pip-release fire fighting. :)

General notes

Nothing to say here this time.

What next?

I’m going to try to wrap up the big things I started over the last few months.

Technical

• installer: Finalise and make a stable release.
• Sphinx: Continue iterating on sphinx-basic-ng.
• Sphinx: Continue iterating on $theme-I-am-writing.
• Sphinx: Continue iterating on the PEP 517 backend.

Communication

• Speak at a few local meetups, remotely.
• Spend time on the Python lockfile standardisation effort.
• Spend time on the Python documentation improvement effort.
• Spend time on the Sphinx tutorial and documentation improvement effort.

Other commitments

A full time job, that pays my bills. :)

I took the opportunity to refresh/revamp the site structure, which worked out pretty well too.

Some notes from doing this transition:

• Hugo’s separation of content vs templates is GREAT.
• Hugo’s documentation is exceedingly beginner unfriendly. :(
• Hugo’s support for {date}-{slug}.md files is… difficult to discover.
• redirect_from == aliases
• I couldn’t search Hugo’s changelog, which makes it difficult to figure out whether something you’re reading from a 2018 issue was removed.
• “Why is this like this” and other variants were spoken 17 times, in the span of 5 hours.

Work I did (1 Mar 2021 - 31 Mar 2021)

Technical

• Started writing a PEP 517 build backend for Sphinx themes.
• Started writing sphinx-basic-ng, an attempt at modernising the Sphinx theme ecosystem.
• Released new versions of sphinx-autobuild.
• More updates to Furo.
• Made progress on installer’s implementation.
• Made progress on pip’s documentation rewrite.
• Started work on rewriting ScriptTest, to pay down pip’s technical debt.

Communication

• Talking to relevant folks about toml on PyPI (and moving it to… a specific GitHub repository).
• Finally sat down and did an “open source responsibility audit”.
• My [social weekends] are still a thing, and has been great so far!
• Still collaborating on designing a lockfile format for Python.
• Added to The Executable Book Project’s GitHub organisation! ^>^
• Chatted with a few people about the state of Python and pip on Debian.

General notes

Looking back, I think I picked up a couple of new projects based on random brain waves I had! That’s perfectly timed, because I’ve decided to pivot away from my earlier approach of “yay, more responsibility!”.

What next?

This is getting increasingly harder to decide on, as my free time chunks are becoming smaller and I’m picking up bigger projects. :)

Technical

• Sphinx Theme PEP 517 stuff: Make the initial release.
• sphinx-basic-ng: Make the first usable release.
• pip: Clear some of the backlog on the pull request front.
• pip: More progress on the documentation rewrite.

Communication

• Spend more time looking into the Python lockfile standardisation effort.
• Write a blog post, on automated code formatting.
• Find more speaking opportunities, to talk about things that aren’t Python packaging!

Other commitments

A full time job, that pays my bills. :)

The plan is to get back to doing these roughly once a month again.

Work I did (1 Jan 2021 - 26 Feb 2021)

Technical

• Published the last version of pip that supports Python 2! 🎉
• Published a few releases of Furo – a Sphinx theme I wrote.
• Made some (unreleased) changes to sphinx-autobuild.
• Made some more progress on installer – a reusable library for Python (wheel) package installation.
• Rewrote get-pip.py and the generation pipeline for it.
• Started a rewrite of pip’s documentation. I’d love to get some feedback on this.

Communication

• I’m experimenting with a new thing: social weekends!
• I presented 2 talks at FOSDEM: in the Python devroom and Open Source Design devroom. Shout-out to Bernard Tyers, for all the help and the bazillion reminders to make sure I do all the things on time. :)
• Collaborating on designing a lockfile format for Python, that can hopefully be standardised for interoperability.

General notes

Onboarding in a new company, relocating internationally, settling into a new space has been… well, it’s all been a very interesting learning experience.

Given the fairly strict lockdown and the percentage of people wearing masks in my locality, I’ve spent a lots of time indoors. Looking forward to the social weekends experiment I’m doing.

What next?

Technical

• pip: Work on the documentation rewrite, hopefully to get it ready in time for the next release.
• pip: Clear some of the backlog on the pull request front.
• pip: General discussions for new features and enhancements.
• TOML: Work on writing that the compliance test suite.
• TOML: Bring toml for Python back from the dead.
• Furo: Make the first stable release.
• Start work on the other Sphinx theme I have in mind.

Communication

• Spend more time looking into the Python lockfile standardisation effort.
• Catch up on the Python-on-Debian saga, and see how I can contribute constructively.

Other commitments

Oh, I have a full time job at Bloomberg now. :)

I’m actually really good at procrastinating things when I’m uncomfortable.
- me, back in Feb 2020

Prelude

As you’re likely aware, the human world has been quite a different place since my last post here. My transition to this change wasn’t super smooth, but hey, it could’ve been a lot worse. My mental health has definitely had a really fun roller coaster ride. I have been fortunate enough to be able to stay safe and also work remotely with some wonderful folks over this time.

I wrapped up my 3 month GSoC project, after 3 years

As you might know, I’ve been working on pip’s dependency resolver since 2017, which was originally my Google Summer of Code project. There is now a public pip release that uses a new, written-from-the-ground-up dependency resolver, that replaces the old pseudo-dependency-resolver it had. 🎉

A large part of why this project was finally pushed over the line was that the Packaging-WG at the Python Software Foundation was able to secure funding toward this, from Mozilla (through its Mozilla Open Source Support Awards) and the Chan Zuckerberg Initiative.

Huge thanks to Bernard Tyers, Ernest W. Durbin III, Georgia Bullen, Nicole Harris, Paul Moore, Sumana Harihareshwara and Tzu-Ping Chung for being amazing colleagues (and friends!) as we worked on this together.

I have a college degree now

Hurray! I made it out of that place. As a faculty once told me, “you got to where you are despite this college, not because of it”.

My college has made me sign paperwork that prohibits me from speaking out against university management (something that should speak for itself). Here’s a link to a relevant section on Wikipedia.

Update: I had to change the link above to a permalink to a specific version of the document, because the linked section was edited out the day I put up this post.

I mentored a GSoC student

I mentored Raphael McSinyx, who worked on pip, exploring speedups to pip’s dependency resolution process. You can read more about their work in his final GSoC report.

I relocated for a job

I am now working at Bloomberg Engineering as, currently, a part of the Python Infrastructure team in London.

I’m officially no longer living in my parent’s home. Onward to new adventures, I guess.

I am doing Sphinx-y things now

I made a Sphinx documentation theme: Furo, modernised sphinx-themes.org and am collaborating with the amazing folks of the Executable Books project.

I went to San Francisco

CZI conducted an EOSS kickoff meeting, bringing lots of members of their first cohort of 32 EOSS grantees into one room. This included me!

It was a really well conducted event, and the room was filled with brilliant people. I felt like a squirrel in a room full of elephants.

This happened right before COVID-19 was deemed serious enough to care about it. It was the last trip I had in the before-times, and it was definitely a good one.

The motivation behind writing this, is to make sure all the developers working on this project are on the same page, and to have a written record about the state of affairs.

Architecture

The “legacy” resolver in pip, is implemented as part of pip’s codebase and has been a part of it for many years. It’s very tightly coupled with the existing code, isn’t easy to work with and has severe backward compatibility concerns with modifying directly – which is why we’re implementing a separate “new” resolver in this project, instead of trying to improve the existing one.

The “new” resolver that is under development, is not implemented as part of pip’s codebase; not completely anyway. We’re using an abstraction that separates all the metadata-generation-and-handling stuff vs the core algorithm. This allows us to work on the core algorithm logic (i.e. the NP-hard search problem) separately from pip-specific logic (eg. download, building etc). The abstraction and core algorithm are written/maintained in https://github.com/sarugaku/resolvelib right now. The pip-specific logic for implementing the “other side” of the abstraction is in https://github.com/pypa/pip/tree/master/src/pip/_internal/resolution/resolvelib.

Testing

In terms of testing, we have dependency-resolution-related tests in both resolvelib and pip.

resolvelib

The tests in resolvelib are intended more as “check if the algorithm does things correctly” and even contains tests that are agnostic to the Python ecosystem (eg. we’ve borrowed tests from Ruby, Swift etc). The goal here is to make sure that the core algorithm we implement is capable of generating correct answers (for example: not getting stuck in looping on the same “requirement”, not revisiting rejected nodes etc).

pip

The tests in pip is where I’ll start needing more words to explain what’s happening. :)

YAML-based tests

We have “YAML” tests which I’d written back in 2017, as a format to easily write tests for pip’s new resolver when we implement it. However, since we didn’t have a need for it to be working completely back then (there wasn’t a new resolver to test with it!), the “harness” for running these tests isn’t complete and would likely need some work to be as feature complete as we’d want it to be, for writing good tests.

YAML tests: https://github.com/pypa/pip/tree/master/tests/yaml
YAML test “harness”: https://github.com/pypa/pip/blob/master/tests/functional/test_yaml.py and https://github.com/pypa/pip/blob/master/tests/lib/yaml_helpers.py

“new” resolver tests

unit tests

We have some unit tests for the new resolver implementation. These cover very basic “sanity checks” to ensure it follows the “contract” of the abstraction, like “do the candidates returned by a requirement actually satisfy that requirement?”. These likely don’t need to be touched, since they’re fairly well scoped and test fairly low-level details (i.e. ideal for unit tests).

New resolver unit tests: https://github.com/pypa/pip/tree/master/tests/unit/resolution_resolvelib

functional tests

We also have “new resolver functional tests”, which are written as part of the current work. These exist since how-to-work-with-YAML-tests was not an easy question to answer and there needs to be work done (both on the YAML format, as well as the YAML test harness) to flag which tests should run with which resolver (both, only legacy, only new) and make it possible to put run these tests in CI easily.

New resolver functional tests: https://github.com/pypa/pip/blob/master/tests/functional/test_new_resolver.py

test_install*.py

These files test all the functionality of the install command (like: does it use the right build dependencies, does it download the correct files, does it write the correct metadata etc). There might be some dependency-resolution-related tests in test_install*.py files.

These files contain a lot of tests so, ideally, at some point, someone would go through and de-duplicate tests from this as well.

How can you help?

If you use pip, there are a multiple ways that you can help us!

• First and most fundamentally, please help us understand how you use pip by talking with our user experience researchers. You can do this right now! You can take a survey, or have a researcher interview you over a video call. Please sign up and spread the word to anyone who uses pip (even a little bit).

• Right now, even before we release the new resolver as a beta, you can help by running pip check on your current environment. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely that you will hit issues when the new resolver is released (and may address hidden problems in your current environment!). If you run pip check and run into stuff you can’t figure out, please ask for help in our issue tracker or chat.

Thanks to Paul Moore and Tzu-Ping for help in reviewing and writing this post, as well as Sumana Harihareswara for suggesting to put this up on my blog!

I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver.

Work I did (Jan 6 - Feb 5)

Technical

• Co-worked with another developer, in person, for 1 week, on pip!
• Triaged pip’s issue tracker (a lot).
• Spend some time improving pip’s test suite infrastructure.
• Investigated Python 2 usage, to identify anomalies.
• Helped with virtualenv 20.0 release (kinda!).
• Invested effort to improve pip’s test suite
• Helped aggregate test cases for pip’s next generation resolver.

Communication

• Managed the pip 20.0 release fiasco.
• Helped the UX folks get started with working on pip.

Additional notes on challenges

January has been a very productive month.

Most of the challenges have been the logistics around work, not the work as such.

My health has been pretty good and there’s a certain flow to my work that I’m enjoying now. Turns out, if you like what you’re doing, you tend to be pretty productive! :)

As long as I remember to push my blog posts to the repository, they’ll actually go live on the day they’re supposed to.

Goals for February 2020

Technical

• Internal Cleansing: AKA Technical debt down payment.
• Issue triage: Triage a fair number of issues on pip’s issue tracker.
• Technical Documentation: improving pip’s technical documentation, for contributors and developers

Communication

• Help all the other contractors to get up to “full speed” for working on pip
• Get PyPA to participate in GSoC 2020
• Python Packaging Summit at PyCon US 2020: help organization.
• Move forward on Python Packaging Governance

Other commitments

None.

Help us

How can you help us?

• provide test cases where the latest released version of pip (19.3.1, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us design and test the new resolver.
• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page lists fairly well-scoped projects that would happen much faster if we get funding to achieve them.
• Have an interview with our UX expert, who is working to improve usability of Python Packaging tooling.

I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver.

Work I did (Dec 11 - Jan 5)

Technical

• Wrore and adopted vendoring for use in pip.
  • Initiated conversation for pipenv to switch to it too.
• Made progress toward TOML 1.0, via useful clarifications in the specification
• Continued tackling pip’s build logic refactoring.
  • Refactoring of pip’s wheel building logic.
  • Designing a “package builder” library – figuring out what Python Packaging’s “build” step should look like to end users.
• Started reviewing virtualenv’s rewrite.
• PR Reviews: ~25, including improvements to pip’s installation logic, better wheel-related error messages and more.

Communication

• A few release manager-y things:
  • (I’ll be the release manager for pip 20.0)
  • Identified tasks that need to happen prior to pip’s release.
• Discussions on discuss.python.org:
  • Giving pip an API https://discuss.python.org/t/2833
  • Python Packaging in GSoC 2020 https://discuss.python.org/t/2830
  • PyPA governance PEP https://discuss.python.org/t/2619
• Discussions on issue trackers:
  • New Resolver: technical choices https://github.com/pypa/pip/issues/7406
  • Switch to a dedicated tool for vendoring pip’s dependencies https://github.com/pypa/pip/pull/7485
• Provided helper/starter for pip’s UX review work.
• Onboarded a new triager to pip: Welcome Stéphane Bidoul! :)

Additional notes on challenges

December was fairly productive, until the end of the month, when the holidays came in and I also wasn’t in the best of health.

I couldn’t come up with a good talk topic for PyCon US 2020. :/

Goals for January 2020

Technical

• Onboard and co-work with a developer, on pip.
• Triage pip’s issue tracker. A lot.
• Spend some time improving pip’s test suite infrastructure.
• Investigate Python 2 usage from PyPI, to identify anomalies.

Communication

• Ensure pip 20.0’s release goes well (I’m the release manager).
  • Yay automation that I worked on a little while back!
• I’ll be very active on PSF’s Zulip instance, as part of the grant-funded work on pip’s resolver.

Other commitments

None.

Help us

How can you help us?

• provide test cases where the latest released version of pip (19.3.1, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us design and test the new resolver.
• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page lists fairly well-scoped projects that would happen much faster if we get funding to achieve them.

This update has been a little late, since I have been traveling and recovering from an illness.

I also have good news: I will be working on open source projects related to Python Packaging (full time) starting early Dec 2019 till May/Jun 2020.

Work I did (Nov 6 - Dec 10)

Technical

• Worked to separate pip’s dependency vendoring logic from rest of pip
  • This would enable other projects that use similar mechanisms (eg. pipenv)
• Made some progress for preparing for TOML’s 1.0 release

Communication

• Finalized my internship, and got the necessary approvals at college
• PyConf Hyderabad 2019! (yay!)
  • I gave a talk: Rethinking Python Packaging – a thought experiment.
  • I ran a sprint! ~5 new contributors worked on pip!
• Initiated a [discussion for Python Packaging at PyCon US and in GSoC 2020.

Additional notes on challenges

As I’d anticipated, November has been low productivity. I didn’t have much time to put toward working on “Open Source stuff”, due to various other commitments – broadly: exam preparation, attending a family wedding and talk preparation for PyConf Hyderabad.

Goals for December

Technical

• Finish decoupling pip’s dependency vendoring logic to a dedicated tool
• Resume work on pip’s build logic refactor
  • Shout out to Chris Hunt (@chrahunt) for keeping things moving while I was away.
• Check for pip 20.0 preparedness (due Jan 2020)
• Do issue triage on pip’s issue tracker
• (stretch goal) Look into further reducing fragile manual steps in pip’s release process

Communication

• Catch up on November’s discussions
• Propose a talk at PyCon US 2020
• Make progress on discussions related to PyCon US and GSoC 2020
• Identify areas which (in my opinion) would be of interest in pip’s UX review

Other commitments

There is likely some travel in late December and early January, with family. And that’s about it in terms of planned commitments.

Help us

How can you help us?

• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page lists fairly well-scoped projects that would happen much faster if we get funding to achieve them.
• volunteer to help us build robust testing infrastructure
• provide test cases where the latest released version of pip (19.3.1, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us design and test the new resolver.
• help triage pip’s issue tracker!

Work I did (Oct 6 - Nov 5)

Technical

• Wrapped up refactoring metadata generation within pip. (yay!)
• Started refactoring wheel building logic within pip.
• A few minor fixes/cleanup within pip.
• Helped out with pip 19.3’s release.
• Significantly simplified pip’s release processes.
• Reviewed >50 PRs.
• Fixed a “major” regression in pip’s detection of virtual environments.
• Got some work done, progressing closer to TOML 1.0.0.

Communication

• Summarized the progress on the build logic refactoring.
• PyCon India 2019! (yay!)
  • I gave a talk!
  • I ran a sprint! ~20 contributors and just me as a mentor. (This was fun!)
• Figured out a lot of the details around my semster-long internship.
• Had a chat with b.p.o. maintainers, and published a blog post describing their plans going forward.
• My proposed governance model for PyPA, is becoming a PEP.

Additional notes on challenges

Oh boi, pip’s test suite doesn’t like venv. The initial draft for this commit message was about 4x as long, and a lot less… nice. I sunk quite a few hours across two days, into that one commit, trying to make things work before realizing that it’d be better to tackle fixing the test isolation separately.

A lot of my time over the last two weeks has gone into handling a big mess, in my hostel room caused by fungi. Even though I didn’t cause this mess, I have to deal with it (allergic bronchitis isn’t making it easier).

Other than that though, I think everything went as I had anticipated. During PyCon India + Diwali, I made the exact amount of progress on the “code” problems as I’d expected – None. Back in college, there was enough academic workload to drain me by the end of the day. :)

Basically, I think almost all of my “write code”, happened on 2 or 3 days total, in this past month.

Goals for November

Technical

• Figure out a plan for refactoring pip’s wheel building logic.
• Make more progress on the TOML’s compliance suite.

Communication

• Wrap up final-semester internship search.
• (stretch) Write a blog post using all the material that was trimmed out of my PyCon India talk – “A checkin on Python Packaging”.

Other commitments

I have my end-sem examinations all of November (there’s one tomorrow).

Overall, November is gonna be fairly “low” productivity. Hopefully, I would be able to make some progress in early December (I’ll be back home!).

Help us

How can you help us?

• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page now summarizes what bugfixes/features/projects within and outside PyPA are blocked on the new resolver, some background context, and how funding would help us get it finished faster.
• provide test cases where the latest released version of pip (19.2.3, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us design and test the new resolver.
• volunteer to help us build robust testing infrastructure
• triage pip bugs! Around 10% of pip’s issues have not been triaged yet and many others need of action/review.

PEP 581 proposes sunsetting bugs.python.org, in favor of GitHub issues. “Why not focus on improving Roundup / bpo?” section made me wonder: What is needed to improve Roundup / bpo? This blog post is my attempt at summarizing the state of bpo and how it could be improving in the future, based on a discussion with folks maintaining bpo.

I won’t go into whether PEP 581 or PEP 595 is a better idea – that’s something for the CPython core developers to decide on.

What is that tracker?

Broadly, bpo is an instance of Roundup, that is hosted by the PSF.

Roundup is a simple-to-use and -install issue-tracking system with command-line, web and e-mail interfaces. It is based on the winning design from Ka-Ping Yee in the Software Carpentry “Track” design competition.

By design, it is an extremely customizable platform. Roundup has a “core” that can run multiple issue trackers. Each of these issue trackers can contain customizations in the HTML/CSS and extensions for functionality that are specific to them.

When CPython moved to using Roundup, both projects were using subversion for version control. Since then, CPython moved to Mercurial (Roundup followed soon) and now CPython is on Git.

How is it all working?

The PSF-hosted Roundup, is actually a fork of Roundup maintained by a few folks. The fork was created for adding functionality that made it better for bpo, when the initial transition what made. There are still some functionality improvement patches today (more on this further below).

PSF-hosted Roundup, runs 3 issue trackers:

• https://bugs.python.org (or b.p.o)
• https://bugs.jython.org
• https://issues.roundup-tracker.org

All the relevant repositories of the PSF are in Mercurial, and hosted on BitBucket currently.

Where are we headed?

The maintainers of bpo want to switch to using upstream Roundup, instead of our PSF fork. To that end, most of the functionality improvement patches made in the fork have been ported upstream. There is only one major thing that’s not been integrated upstream (yet!) – GitHub integrations built as part of the GitHub migration of CPython source code.

There’s an upcoming Roundup 2 release (maybe Q1 2020) – adding a new REST API, Python 3 support, a responsive UI and more. These would address some of the major concerns with Roundup powering bpo today. There is a plan to host a read-only test instance of Roundup 2 alpha/beta/rcs “soon”, for exploring new functionality and usability.

Broadly, from what I can tell, the plan of action is:

1. Finalize the GitHub login patch for Roundup.
2. Port the GitHub integrations to upstream Roundup.
3. Set up the test tracker, for bugs.python.org on Roundup 2.
4. Wait for Roundup 2 release.
5. Start running bpo on a regular Roundup 2. (AKA get rid of the PSF’s fork)
6. Move the instance code and templates, to GitHub.

Steps 3 and 6 aren’t strictly blocked on the steps before them, but this is probably the easiest order if we’re doing things sequentially. Given that Step 6 has to happen before 1 June 2020 ¹, it is reasonable to assume this would all get done by then.

Once the Roundup 2 transition is completed, the documentation of the bpo instance would be improved (how it works, what the various files/folders in the repository do and so on). Along with Roundup 2’s REST API, this should significantly reduce the barrier to contribute to bpo or to build enhancements around it.

Many thanks to Ezio Melotti for indulging in a very insightful conversation and for helping me write this post.

Work I did (Sept 6 - Oct 5)

Technical

• Figured out a rough structure for approaching pip’s build logic refactor.
• Discussed and sketched a rough “goal” model for pip’s build logic, to adopt after initial cleanups and decoupling refactor work.
• Made significant progress on refactoring pip’s metadata handling and generation logic. (yay!)
• Identified a quirk in how pip’s current resolution logic, which makes it conceptually intertwined with pip’s index interaction logic (sigh).
  • out-of-scope for the ongoing “build logic refactor”, but we do need to resolve this at some point.
• Revamped pip’s linting pipeline to be… simpler (i.e. better).
• Multiple miscellaneous fixes/cleanup to pip.
• A lots of PRs reviewed (>80).
• Reviewed and merged ABNF-related PRs, for TOML.
• Sunk a good day to debugging 500s on GitHub (it was due to emojis). :(

Communication

• Reached out to quite a few folks regarding my semester-long internship.
• Discuss goals and approaches, for pip’s build logic refactor, with fellow pip maintainers across multiple issues on the issue tracker.
• Had a chat w/ a pipenv maintainer, discussing the resolver code reuse situation, and some design discussion about the ongoing build logic refactoring.
• Had multiple chats with fellow pip maintainers, discussing a rough plan and target, for the build logic refactor as well as our development workflow.
• Polished up my proposed “Status-Quo” PyPA governance model.
• Reviewed a proposal for better licensing metadata for Python Packages.

Additional notes on challenges

I had more exams! I’d hoped to get more work done but I hadn’t realize that I’d have exams over 2 weekends – Sunday, 29 Sept to Sunday, 6 Oct. (yay college)

In addition, I realized that my initial approach toward reworking pip’s “prepare distribution” logic was fatally flawed because I forgot about a usecase. My discussions with pipenv and pip maintainers were very fruitful and we pooled our combined knowledge of the tooling into text documents and diagrams. We’ve come up with a more feasible (and much better) plan than the one I had come up with on my own.

Recently, I read something on discuss.python.org, that just perfectly encapsulates how I feel about this stuff:

all of this stuff is really hard and there’s a lot of knowledge scattered around in people’s heads that could really do with being captured somewhere

Overall, I made time to work on pip this month, and made significant progress on cleaning up pip’s metadata generation and handling logic.

Goals for October

Technical

• Wrap up refactoring metadata generation and handling in pip.
• Work on a “blessed” TOML’s compliance suite, as we prepare for TOML 1.0.
• (stretch goal) Begin refactoring the next component – installation.

Communication

• Update the GitHub Project board for pip’s build logic refactor, based on the mental model built over the past few discussions.
• Attend PyCon India:
  • Give a talk about the state of Python Packaging.
  • Run a sprint on pip, packaging and pypa-bot (and more?).
• Figure out my internship situation and getting approval from my HoD for the internship I’d be doing.

Other commitments

I expect to have some time to work on pip in October, though not as much as September.

In mid-October, I’ll be speaking at PyCon India 2019 (noted above) and I expect from today until my talk, almost all of my free time would be going towards that talk preparation. After that, I am expecting that I’ll spend a few days going through the processes for approvals, at college, related to my final semester internship.

Help us

How can you help us?

• provide test cases where the latest released version of pip (19.2.3, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us design and test the new resolver.
• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page now summarizes what bugfixes/features/projects within and outside PyPA are blocked on the new resolver, some background context, and how funding would help us get it finished faster.
• triage pip bugs! Around 10% of pip’s issues have not been triaged yet and many others need of action/review.
• help improve our “how to triage pip bugs” guide.
• volunteer to help us build robust testing infrastructure

Work I did (Aug 6 - Sept 5)

Technical

• Began working on splitting the code for handling of the two kinds of source distributions.
• Wrap up pip 19.2, with pip 19.2.3 release, unblocking Python 3.8’s final beta. Hooray!
• Fix documentation builds for pip, on ReadTheDocs – this took way longer to figure out than I’d expected.
• Multiple miscellaneous fixes that I could do in gaps between classes.
• Lots of PRs reviewed (30+)

In August, I wanted to get further on separating “prepare distribution” logic from the resolution logic and on turning distribution-type-specific build code into dedicated modules within the new “distributions” sub-package in pip. I didn’t get there.

Communication

• Gave a keynote at PyCon Korea! (no video links yet – will probably be here)
• The details and requirements for my final semester internship have been clarified.
• OKed the MOSS funding proposal
• Contributed to several discussions on the Python discussion forum, including on better licensing metadata and PyPA governance.

During this month, I also proposed an improvement to twine check (motivated by confusing output when making a pip release), made some more progress toward TOML 1.0, and worked on some of my personal tooling which broke due to a VS Code update. :)

Additional notes on challenges

Lack of time. Outside of my talk at PyCon Korea and mid-semester examinations at college, I could not find much time to work on the goals I had set for myself. In the time that I could make, the urgency of the pip release, broken documentation build, and code review left very little time for me to make progress on foundational improvements to pip’s codebase.

Goals for September

Technical

• Wrap up the goals I’d set for August:
  • Finish splitting logic for handling on source distributions in the new “distributions” sub-package in pip.
  • Separate “prepare distribution” logic from the resolution logic.
• Develop a clearer mental model for pip’s handling of distribution metadata.
• Investigate how to adapt pip’s distribution preparation logic to provide distribution metadata as an independent object.

Communication

• (rolled over from August) Reach out to maintainers of pipenv, pip-tools and poetry, to have a conversation about code reuse.
• Break down tasks, and discuss the priorities for future items for work in pip’s build logic refactor.

Other commitments

I expect to have more time to work on pip in September. There will be less academic workload (read: no exams) and I’m not travelling this month. In mid-October, I’ll be speaking at PyCon India 2019 (yay!) on the state of Python packaging, so I’ll also be working on that talk this month.

Help us

I’m currently looking for 20+ week internships, for my final semester credits in college, starting in December 2019 or early January 2020. If you reckon I’d be a good addition to your team, please reach out.

And if you are reviewing funding requests for the Chan Zuckerberg Initiative or for Mozilla’s Open Source Support program, I hope you’ll approve our proposals. :)

Also, if you care about how Python Packaging is governed, please respond to my PyPA governance proposal.

Other than that, I have the same list as in June. Thanks!

Also, thanks to Sumana Harihareswara for help in editing.

Work I was planning to do

Technical

• Clearly determine what parts of the build system are intertwined with pip’s resolution logic.

Communication

• Get confirmation from my college about required dates/duration of internships in final semester.

Work I did (24 June - 6 Aug)

Technical

• Identified portion of build system intertwined with pip’s resolver:

  • “Get abstract distribution”:
    • building installable artifacts is intertwined with pip’s resolver.
    • Specifically, the resolver currently passes information about how it would handle a distribution, to the “preparer” that fetches and builds it.
  • Marker evaluation and handling is done by “RequirementSet” and not the resolver.

• Started moving code for pip’s handling of installable artifacts into a pip._internal.distributions package.

Communication

• Still working on getting requirements for my final semester internship. Other students have told me that it has to be longer than 15 weeks and should start in early December, but I am seeking official word on that. I’m working on options that would let me work on the pip resolver during the internship.

Additional notes on challenges

No technical challenges. But I didn’t have much time to work on this, because of college and visa paperwork (because PyCon Korea invited me to give a keynote!), and because of illness.

Goals for August

Technical

• Separate “prepare distribution” logic from the resolution logic.
• Split code for building various kinds of source distributions into dedicated modules within the new “distributions” sub-package in pip. (related cleanup)

Communication

• Reach out to maintainers of pipenv, pip-tools and poetry, to have a conversation about code reuse.
  • I know pipenv maintainers have work that can be reused.
  • Need to identify the overlapping area between the tools, since they may require a slightly different results after resolution than pip.
• Work with PSF’s Packaging Working Group on funding proposals to get more help on the resolver testing, UX and rollout.

Other commitments

I am travelling to South Korea this month, for keynoting at PyCon Korea. Immediately after that, I have college examinations from 18th-25th August, and right after that, project reviews for this semester’s courses. So my time is pretty limited in August, but should get freer in September.

Help us

I have the same list as last time. Thanks!

Also, thanks to Sumana Harihareswara for adapting this from a separate update I’d shared.

What’s up?

Over the past few months, I’ve worked on a prototype for a resolver in pip, spent some time planning what rolling out a new dependency resolver in pip might look like, and continue to work on pip’s build logic to decouple it from the dependency resolution logic.

The prototype has helped me better understand the relationships and contracts between various portions of the codebase. This understanding has led me to conclude that it is a worthwhile investment to better decouple the build logic from the resolution logic, then introduce the abstractions defined in resolvelib/zazo, and then proceed to adopting better dependency resolution.

What’s next?

The work on refactoring pip’s build logic is in progress and we’re tracking it on GitHub. Significant technical debt has been accumulated in this part of pip’s codebase over the years, so this is a big project that will likely take multiple months. This would/will also make it easier to provide more user-friendly error messaging, though that is not something that I will pursue in this project.

This investment will enable us to untangle pip’s internals from the resolver, enabling pip to share code for dependency resolution with other packaging tooling (like pipenv and hatch) and reuse the work they have done (e.g., work done on pipenv’s resolver). This also makes a softer rollout possible, one that allows us to iterate based on feedback from users and minimize disruption.

Help us

How you can help us?

• triage pip bugs! Around 20% of pip’s issues have not been triaged yet.

• draft a “how to triage pip bugs” guide.

• provide test cases where the latest released version of pip (19.1.1, at the time of writing) fails to resolve dependencies properly (on zazo’s issue tracker). They will help us test the new resolver.

• look at New Resolver: Rollout, Feedback Loops and Development Flow and help us decide: Is the feature flag approach a good idea? Is it a good idea to get rollout feedback via some mechanism other than the pip GitHub issues? Any other thoughts to help us smooth out the rollout?

• talk with your company about becoming a PSF sponsor. The Fundable Packaging Improvements page now summarizes what bugfixes/features/projects within and outside PyPA are blocked on the new resolver, some background context, and how funding would help us get it finished faster.

• volunteer to help us build robust testing infrastructure

Thanks to Sumana Harihareswara for (a lot of) help in writing this, and Dan Ryan for reviewing the content!

Seems like this is reaching you.

Hello there! I’m Pradyun. You probably knew that already. Thanks for spending the time to reach this random piece of writing by me.

Acknowledgements

This blog used to be built using Jekyll and the theme was based off Pixyll. The theme was heavily restyled to be more in line with how I wanted my site to look like. I ended up making changes to almost every element of the design, though the layout and colours were solid in Pixyll.