How I Setup A Private Local PyPI Server Using Docker And Ansible. [Continues]

TL;DR

  • Selects machines to execute against from inventory
  • Connects to those machines (or network devices, or other managed nodes), usually over SSH
  • Copies one or more modules to the remote machines and starts execution there

Containerization

Automation

Prerequisite

python3 -m pip install ansible paramiko
ansible-galaxy collection install \
ansible.posix \
community.docker

Directory Structure

├── ansible.cfg
├── ansible-requirements-freeze.txt
├── host_inventory
├── Makefile
├── README.md
├── roles
│ └── pypi_server
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ └── simple_test-1.0.zip
│ ├── tasks
│ │ └── main.yml
│ └── templates
│ └── nginx-pypi.conf.j2
└── up_pypi.yml

Ansible Configuration

cat >> ansible.cfg << EOF
[defaults]
inventory=host_inventory
# https://github.com/ansible/ansible/issues/14426
transport=paramiko
[ssh_connection]
pipelining=True
EOF
wget https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg

Selecting a machine to run your commands from inventory

cat >> host_inventory << EOF
vagrant ansible_host=192.168.50.4 ansible_user=root ansible_become=yes
[pypi_server]
vagrant
EOF
cat >> Vagrantfile << EOF
# -*- mode: ruby -*-
# vi: set ft=ruby :
# set up the default terminal
ENV["TERM"]="linux"
Vagrant.configure(2) do |config|
config.vm.box = "opensuse/Leap-15.2.x86_64"
config.ssh.username = 'root'
config.ssh.password = 'vagrant'
config.ssh.insert_key = 'true'
config.vm.network "private_network", ip: "192.168.50.4"
config.vm.network "forwarded_port", guest: 3141, host: 3141 # devpi Access
# consifure the parameters for VirtualBox provider
config.vm.provider "virtualbox" do |vb|
vb.memory = "1024"
vb.cpus = 1
vb.customize ["modifyvm", :id, "--ioapic", "on"]
end
config.vm.provision "shell", inline: <<-SHELL
zypper --non-interactive install python3 python3-setuptools python3-pip
zypper --non-interactive install docker
systemctl enable docker
usermod -G docker -a $USER
systemctl restart docker
SHELL
end
EOF
# password: vagrant
ssh-copy-id vagrant@192.168.50.4
ansible all -m ping

Ansible Roles

ansible-galaxy init pypi_server
  • defaults/main.yml: default variables for the role.
  • files/main.yml: files that the role deploys.
  • handlers/main.yml: handlers, which may be used within or outside this role.
  • meta/main.yml: metadata for the role, including role dependencies.
  • tasks/main.yml: the main list of tasks that the role executes.
  • templates/main.yml: templates that the role deploys.
  • vars/main.yml: other variables for the role.

Playbook

cat >> up_pypi.yml <<EOF
---
- name: configure and deploy a PyPI server
hosts: pypi_server
roles:
- role: pypi_server
vars:
fqdn: # Fully qualified domain name
fqdn_port: 80
host_ip: "{{ hostvars[groups['pypi_server'][0]].ansible_default_ipv4.address }}"
nginx_reverse_proxy: reverse_proxy
EOF

Plays

cat >> defaults/main.yml << EOF
---
container_name : pypi_server
base_image : << Your Docker Registry>>/pypi_server:latest
devpi_client_ver: '5.2.2'devpi_port: 3141
devpi_user: devpi
devpi_group: devpi
devpi_folder_home: ./.devpi
devpi_nginx: /var/data/nginx
EOF
  • Install apt and python packages.
  • Update apt-cache and install python3-pip.
  • Install ansible-docker dependencies.
  • Start devpi and configure nginx routings.
  • Start devpi server on docker container.
  • Pause for 30 seconds to ensure server is up.
  • Confirm if docker container is up.
  • Create a PyPI user and an index.
  • Template nginx reverse proxy config.
  • Check if nginx reverse proxy is up.
  • Reload nginx reverse proxy.
  • Check if PyPI server is running!
  • Install python dependencies locally in a virtual environment.
  • Check if devpi index is up and confirm nginx routing!
  • Login to devpi as PyPI user.
  • Find path to simple-test package.
  • Upload simple-test package to devpi.
  • Check if package was uploaded.
  • Install python package from PyPI server.
  • Garbage cleaning.
cat >> tasks/main.yml << EOF
---
- name: Install apt and python packages
block:
- name: update apt-cache and install python3-pip.
apt:
name: python3-pip
state: latest
update_cache: yes
- name: install ansible-docker dependencies.
pip:
name: docker-py
state: present
become: yes
tags: [devpi, packages]
- name: start devpi and configure Nginx routings
block:
- name: start devpi server on the docker container.
community.docker.docker_container:
name: "{{ container_name }}"
image: "{{ base_image }}"
volumes:
- "{{ devpi_folder_home }}:/root/.devpi"
ports:
- "{{ devpi_port }}:{{ devpi_port }}"
restart_policy: on-failure
restart_retries: 10
state: started
- name: pause for 30 seconds to ensure server is up.
pause:
seconds: 30
- name: "confirm if {{ container_name }} docker is up"
community.docker.docker_container:
name: "{{ container_name }}"
image: "{{ base_image }}"
state: present
- name: create pypi user and an index.
shell: "docker exec -ti {{ container_name }} /bin/bash -c '/data/create_pypi_index.sh'"
register: command_output
failed_when: "'Error' in command_output.stderr"
- name: template nginx reverse proxy config
template:
src: "nginx-pypi.conf.j2"
dest: "{{ devpi_nginx }}/{{ fqdn }}.conf"
- name: "check if {{ nginx_reverse_proxy }} is up"
community.docker.docker_container_info:
name: "{{ nginx_reverse_proxy }}"
register: result
- name: "reload {{ nginx_reverse_proxy }}: nginx service"
shell: "docker exec -ti {{ nginx_reverse_proxy }} bash -c 'service nginx reload'"
when: result.exists
- name: pause for 30 seconds to ensure nginx is reloaded.
pause:
seconds: 30
tags: [docker, nginx]- name: check if pypi server is running!
delegate_to: localhost
connection: local
block:
- name: install python dependencies locally in a virtual environment
pip:
name: devpi-client
version: "{{ devpi_client_ver }}"
virtualenv: /tmp/venv
virtualenv_python: python3
state: present
- name: "check if devpi's index is up and confirm nginx routing!"
shell: "/tmp/venv/bin/devpi use http://{{ fqdn }}/pypi/trusty"
- name: login to devpi as pypi user
shell: "/tmp/venv/bin/devpi login pypi --password="
- name: find path to simple-test package
find:
paths: "."
patterns: '*.zip'
recurse: yes
register: output
- name: upload simple-test package to devpi
shell: "/tmp/venv/bin/devpi upload {{ output.files[0]['path'] }}"
- name: check if package was uploaded
shell: "/tmp/venv/bin/devpi test simple-test"
- name: install python package from pypi server
pip:
name: pip
virtualenv: /tmp/venv
extra_args: >
--upgrade
-i http://{{ fqdn }}/pypi/trusty
--trusted-host {{ fqdn }}
- name: garbage cleaning
file:
path: "/tmp/venv"
state: absent
tags: [tests]
EOF
cat >> nginx-pypi.conf.j2 <<EOF
server {
server_name {{ fqdn }};
listen 80;

gzip on;
gzip_min_length 2000;
gzip_proxied any;
gzip_types application/json;

proxy_read_timeout 60s;
client_max_body_size 70M;

# set to where your devpi-server state is on the filesystem
root {{ devpi_folder_home }};

# try serving static files directly
location ~ /\+f/ {
# workaround to pass non-GET/HEAD requests through to the named location below
error_page 418 = @proxy_to_app;
if ($request_method !~ (GET)|(HEAD)) {
return 418;
}

expires max;
try_files /+files$uri @proxy_to_app;
}
# try serving docs directly
location ~ /\+doc/ {
# if the --documentation-path option of devpi-web is used,
# then the root must be set accordingly here
root {{ devpi_folder_home }};
try_files $uri @proxy_to_app;
}
location / {
# workaround to pass all requests to / through to the named location below
error_page 418 = @proxy_to_app;
return 418;
}
location @proxy_to_app {
proxy_pass http://{{ host_ip }}:{{ devpi_port }};
proxy_set_header X-outside-url $scheme://$http_host;
proxy_set_header X-Real-IP $remote_addr;
}
}
EOF

Makefile

make install_pkgs pypi
cat >> Makefile << EOF 
.DEFAULT_GOAL := help
define PRINT_HELP_PYSCRIPT
import re, sys
print("Please use `make <target>` where <target> is one of\n")
for line in sys.stdin:
match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line)
if match:
target, help = match.groups()
if not target.startswith('--'):
print(f"{target:20} - {help}")
endef
export PRINT_HELP_PYSCRIPThelp:
python3 -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
install_pkgs: ## Install Ansible dependencies locally.
python3 -m pip install -r ansible-requirements-freeze.txt
lint: *.yml ## Lint all yaml files
echo $^ | xargs ansible-playbook -i host_inventory --syntax-check
pypi: ## Setup and start PyPI server
ansible-playbook -i host_inventory -Kk up_pypi.yml
EOF

Final Testing

  • Stopped pypi_server container, delete pypi_server on server
  • Ran a CI job that builds and pushes Docker images to our local docker registry.
  • Started pypi_server container by executing make pypi whilst in the current working directory (ansible roles) on an ansible dedicated server.
  • Verified if pypi.domain FQDN is up (curl http://pypi.domain && dig pypi.domain)
  • In a virtual environment, installed a random Python package then rebuilt the wheel before pushing it to pypi.domain

Conclusion

Reference

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store