How I Setup A Private Local PyPI Server Using Docker And Ansible

  • The setup was not under config management, meaning we didn’t know how we would reconstitute it if it dies and like every software project there wasn’t much detailed documentation on the how-to.
  • The Python packages did not have any backups, so if something was to happen it would be bye-bye to old packages i.e. It would be tricky to tests old system releases.
  • The server needed to be restarted occasionally to forcefully refresh the packages as our package index had grown over the past few years.
  • Research and evaluating existing tools that the Python ecosystem had to offer, devpi and pypi-server being the most prominent ones.
  • Run the PyPI server in a container preferably Docker (current setup was running in a ProxMox LXC container.)
  • Ensure that deployments are deterministic and,
  • PyPI repository can be torn down and recreated ad hoc by a single command (preferably through Ansible).
  • Overall ensure that there isn’t any significant downtime between the change-over i.e. The client-side shouldn’t have to make any changes.

Prerequisite

sudo apt install docker.io
# The Docker service needs to be set up to run at startup.
sudo systemctl start docker
sudo systemctl enable docker
python3 -m pip install docker-compose

Containerization

Directory Structure

├── Makefile
├── pypi_server
│ ├── config.yml
│ ├── create_pypi_index.sh
│ ├── docker-compose-dev.yaml
│ ├── docker-compose-stable.yaml
│ ├── Dockerfile
│ ├── entrypoint.sh
│ └── README.md
└── README.md

Makefile

# Which will lint the Dockerfile, build, tag and push the image to our local registry
make push_pypi_server
cat >> Makefile << EOF 
SHELL := /bin/bash -eo pipefail
# Defined images here
.PHONY: $(IMAGES)
IMAGES := pypi_server
# Docker registry URL
REGISTRY :=
.DEFAULT_GOAL := help
define PRINT_HELP_PYSCRIPT
import re, sys
print("Please use `make <target>` where <target> is one of\n")
for line in sys.stdin:
match = re.match(r'(^([a-zA-Z-]+).*?)## (.*)$$', line)
if match:
target, _, help = match.groups()
if not any([target.startswith('--'), '%' in target, '$$' in target]):
target = target.replace(':','')
print(f'{target:40} {help}')
if '%' in target:
target = target.replace('_%:', '_{image_name}').split(' ')[0]
print(f'{target:40} {help}')
if '$$' in target:
target = target[:target.find(':')]
print(f'{target:40} {help}')
endef
export PRINT_HELP_PYSCRIPT
.PHONY: help
help:
@python3 -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
pre_build_%: IMAGE = $(subst pre_build_,,$@)
pre_build_%: ## Run Dockerfile linter (https://github.com/hadolint/hadolint)
docker run --rm -i hadolint/hadolint < $(IMAGE)/Dockerfile
build_cached_%: IMAGE = $(subst build_cached_,,$@)
build_cached_%: pre_build_% ## Build the docker image [Using cache when building].
docker build -t "$(IMAGE):latest" "${IMAGE}"
build_%: IMAGE = $(subst build_,,$@)
build_%: pre_build_% ## Build the docker image [Not using cache when building].
docker build --no-cache -t "$(IMAGE):latest" "${IMAGE}"
touch .$@
tag_%: IMAGE = $(subst tag_,,$@)
tag_%: pre_build_% ## Tag a container before pushing to cam registry.
if [ ! -f ".build_${IMAGE}" ]; then \
echo "Rebuilding the image: ${IMAGE}"; \
make build_$(IMAGE); \
fi;
docker tag "$(IMAGE):latest" "$(REGISTRY)/$(IMAGE):latest"
push_%: IMAGE = $(subst push_,,$@)
push_%: tag_% ## Push tagged container to cam registry.
docker push $(REGISTRY)/$(IMAGE):latest
rm -rf ".build_$(IMAGE)"
EOF

Dockerfile and scripts

cat >> Dockerfile << EOF 
FROM python:3.7
RUN pip install --no-cache-dir \
devpi-client==5.2.2 \
devpi-server==5.5.1 \
devpi-web==4.0.6
ENV PYPI_PASSWORD
EXPOSE 3141
WORKDIR /root
VOLUME /root/.devpi
COPY create_pypi_index.sh /data/create_pypi_index.sh
RUN chmod a+x /data/create_pypi_index.sh
COPY entrypoint.sh /data/entrypoint.sh
ENTRYPOINT ["bash", "/data/entrypoint.sh"]
COPY config.yml /data/config.yml
CMD ["devpi-server", "-c", "/data/config.yml"]
EOF
cat >> entrypoint.sh << EOF 
#!/usr/bin/env bash
if ! [ -f /root/.devpi/server ]; then
devpi-init
fi
exec "$@"
EOF
cat >> create_pypi_index.sh << EOF 
#!/usr/bin/env bash
# Creates PyPI user and an index for uploading packages to.devpi use http://localhost:3141
devpi login root --password=
devpi user -c pypi email= password=${PYPI_PASSWORD:-}
devpi user -l
devpi index -c pypi/stable bases=root/pypi volatile=True mirror_whitelist=*
EOF
PYPI_CONTAINER=$(docker ps --filter "name=pypi" --filter "status=running" --format "{{.Names}}")
docker exec -ti ${PYPI_CONTAINER} /bin/bash -c "/data/create_pypi_index.sh"

Devpi configuration

cat >> docker-compose-dev.yaml << EOF 
---
version: '3'
services:
devpi:
build:
context: .
dockerfile: ./Dockerfile
ports:
- "${DEVPI_PORT:-3141}:3141"
volumes:
- "${DEVPI_HOME:-./devpi}:/root/.devpi"
tty: true
stdin_open: true
EOF

Compose file(s)

env DEVPI_HOME="${HOME}/.devpi" docker-compose -f docker-compose-dev.yaml up --build -d
# or
# cat << EOF > .env
# DEVPI_HOME="${HOME}/.devpi"
# EOF
# docker-compose --env-file ./.env -f docker-compose-dev.yaml up --build -d
# --------------------------------------------------------------------------
# or native
# docker build -t pypi_server .
# docker run -d -ti -v "${HOME}/.devpi:/root/.devpi" -p 3141:3141 pypi_server

Garbage Collection

env DEVPI_HOME="${HOME}/.devpi" docker-compose -f docker-compose-dev.yaml down --volumes --rmi all

Client: Permanent index configuration for pip

mkdir -p ~/.pip
cat >> ~/.pip/pip.conf << EOF
[global]
no-cache-dir = false
timeout = 60
index-url = http://localhost:3141/root/pypi/stable
[search]
index = http://localhost:3141/root/pypi/
EOF
cat >> ~/.bashrc << EOF 
export PIP_INDEX_URL=http://localhost:3141/root/pypi/stable/
EOF

Automation

Conclusion

  • Uploading packages to the local PyPI server is beyond the scope of this post.
  • The purpose of this post was mainly to share the approach that worked well for us. You may use it to host your private package repository and index, adapting it to the cloud provider and web server of your choice.

Reference

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store