使用 pypiserver 快速搭建内网离线 pypi 仓库实践

pypiserver is a minimal PyPI compatible server for pip or easy_install. It is based on bottle and serves packages from regular directories. Wheels, bdists, eggs and accompanying PGP-signatures can be uploaded either with pip, setuptools, twine, pypi-uploader, or simply copied with scp.

pypiserver 服务端配置

如果你的 Linux 环境缺少 Python 2.7 可以参考我的文章直接离线升级至最新版本

Python 2.6 升级至 Python 2.7 的实践心得 https://wsgzao.github.io/post/python-2-6-update-to-2-7/

pypiserver > 1.2.x works with python 2.7 and 3.3+ or pypy. Older python-versions may still work, but they are not tested. For legacy python versions, use pypiserver-1.1.x series.

# 替换 pip 为阿里云，感概豆瓣的时代已经过去
tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host= mirrors.aliyun.com
EOF

# 直接在线安装 pypiserver
pip install pypiserver
# 离线下载 pypiserver
mkdir /tmp/pypiserver
pip install -d /tmp/pypiserver/ pypiserver

# Copy packages into this directory.
mkdir ~/packages
# Copy some packages into your ~/packages folder and then get your pypiserver up and running:
pypi-server -p 8080 ~/packages &

pypiserver 客户端配置

## Download and Install hosted packages.
pip install  --extra-index-url http://localhost:8080/simple/ ...

# or
pip install --extra-index-url http://localhost:8080

## Search hosted packages
pip search --index http://localhost:8080 ...

# 个人推荐的配置
tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url = http://172.28.70.126/simple
extra-index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host = 172.28.70.126
EOF

pypiserver 进阶配置

pypiserver Running as a systemd service

https://github.com/pypiserver/pypiserver#running-as-a-systemd-service

Adjusting the paths and adding this file as pypiserver.service into your systemd/system directory will allow management of the pypiserver process with systemctl, e.g. systemctl start pypiserver.

More useful information about systemd can be found at https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

# 安装需要的包
yum install nginx -y
pip install passlib pypiserver gunicorn

# 创建 pypiserver 服务方便服务启停管理
tee /usr/lib/systemd/system/pypiserver.service <<-'EOF'
[Unit]
Description=gunicorn daemon
After=network.target

[Service]
PIDFile=/run/pypiserver.pid
ExecStart=/usr/local/bin/gunicorn -w16 \
    --pid /run/pypiserver.pid \
    -b :10012 \
    'pypiserver:app(root="/var/www/pypi")'
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target 
EOF

# Warning: pypiserver.service changed on disk. Run 'systemctl daemon-reload' to reload units.
systemctl daemon-reload

# 启动 pypiserver 服务
systemctl enable pypiserver.service
systemctl start pypiserver.service
systemctl status pypiserver.service

# 停止 pypiserver 服务
systemctl disable pypiserver.service
systemctl stop pypiserver.service
systemctl status pypiserver.service

[root@centos7 run]# systemctl status pypiserver.service
● pypiserver.service - gunicorn daemon
   Loaded: loaded (/usr/lib/systemd/system/pypiserver.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-04-13 15:14:08 CST; 859ms ago
 Main PID: 22524 (gunicorn)
   CGroup: /system.slice/pypiserver.service
           ├─22524 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22530 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22531 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22532 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22533 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22534 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22535 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22536 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22537 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22538 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22539 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           └─22540 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")

Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22531] [INFO] Booting worker with pid: 22531
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22532] [INFO] Booting worker with pid: 22532
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22533] [INFO] Booting worker with pid: 22533
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22534] [INFO] Booting worker with pid: 22534
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22535] [INFO] Booting worker with pid: 22535
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22536] [INFO] Booting worker with pid: 22536
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22537] [INFO] Booting worker with pid: 22537
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22538] [INFO] Booting worker with pid: 22538
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22539] [INFO] Booting worker with pid: 22539
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22540] [INFO] Booting worker with pid: 22540
[root@centos7 run]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1517/dnsmasq        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      977/sshd            
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      978/cupsd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1383/master         
tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      19378/sshd: root@pt 
tcp        0      0 0.0.0.0:10012           0.0.0.0:*               LISTEN      22524/python        
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::22                   :::*                    LISTEN      977/sshd            
tcp6       0      0 ::1:631                 :::*                    LISTEN      978/cupsd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1383/master         
tcp6       0      0 ::1:6011                :::*                    LISTEN      19378/sshd: root@pt 

# 检查 pypiserver 服务
cd /var/www/pypi
# 向仓库中添加 python package
[root@centos7 pypi]# pip download pypiserver
Collecting pypiserver
  Downloading https://mirrors.aliyun.com/pypi/packages/d7/78/5772432dad2b9e754ab92f4d301fa507069b9decc8c943c1b18c2043ff4f/pypiserver-1.2.1-py2.py3-none-any.whl (83kB)
    100% |████████████████████████████████| 92kB 643kB/s 
  Saved ./pypiserver-1.2.1-py2.py3-none-any.whl
Successfully downloaded pypiserver

[root@centos7 pypi]# ll
total 84
-rw-r--r-- 1 root root 83529 Apr 13 14:55 pypiserver-1.2.1-py2.py3-none-any.whl

# 搜索刚才下载的 package
[root@centos7 pypi]# pip search -i http://127.0.0.1:10012 pypiserver
pypiserver (1.2.1)  - 1.2.1
  INSTALLED: 1.2.1 (latest)

# 配置 nginx 做反向代理
tee /etc/nginx/conf.d/pypi.conf <<-'EOF'
upstream pypiserver {
  server 127.0.0.1:10012;
}

server {
  listen 10087;

  # disable any limits to avoid HTTP 413 for large package uploads
  client_max_body_size 0;

  location / {
    proxy_pass http://pypiserver/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up pypiserver behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;
    proxy_request_buffering off;
  }

  location /packages/ {
    alias /var/www/pypi; # static file
  }
}
EOF

# 启动 nginx
systemctl enable nginx
systemctl start nginx
systemctl status nginx

# 检查 nginx 服务
pip search -i http://172.28.79.126:10087 pypiserver
pypiserver (1.2.1)  - 1.2.1
  INSTALLED: 1.2.1 (latest)

11 replies • 2018-04-18 11:09:56 +08:00

lfzyx

Apr 13, 2018

2018 年 04 月 12 日
Python 2.7

wellsc

Apr 13, 2018 via iPhone

Docker container 也很好用

z550665887

Apr 13, 2018

我记得当初测试 pypiserver 对于本地没有缓存的 pip 包的 pip install 请求会返回一个官网(默认)的源下载链接。对于无法访问公网的服务器来说这个 pip install 请求不会生效。。。
后来采用 docker 启的 devpi 服务完美解决的。

julyclyde

Apr 13, 2018

2.6 升级到 2.7 居然还能有心得？？
python 的安装做的有那么差吗？
python 还会完善 yum 的设置？越俎代庖还是你没搞清楚这俩东西没关联？

为什么那么爱好“离线包”、“上传到网盘”？
你制作的离线包，你会继续投入精力去持续更新吗？已经放出去的旧版本链接，你能召回吗？

wsgzao

Apr 13, 2018

@julyclyde #4 Blog 本身记录的是一种方法和解决问题的思路，很少有人可以持续更新技术类笔记，只是作为一个参考，我本来也觉得不是很复杂的东西但有时候就是会采坑，所以记录一点简单实用的，很复杂的技术自己放在笔记里和公司内部团队共享就好了

wsgzao

Apr 13, 2018

@z550665887 #3 pip search --index http://localhost:8080，我们内部用 GitLab+Jenkins 做团队开发的时候是通过 fabfile 写在脚本里执行的，等后续研究透彻 GitLab CI/CD + pipenv 的新方法后再做分享

julyclyde

Apr 13, 2018

@wsgzao 问题是，你的“因为 Python 2.7.13 以后版本会自动完善 yum 配置，所以不必参考以前的网上文章去修改其他地方”既没有正确的说明（你以为的） python 和 yum 的关系，也没说“以前网上文章”哪些内容是不再需要的，你这样的写法并不能起到记录的作用

wsgzao

Apr 15, 2018

@julyclyde #7 Sorry，我点我确实没有做好，因为在 Python 官方安装的方法中没有看到有 FQ 提示，而 Google 搜索到的大部分信息都是通过修改软连接修复，将#!/usr/bin/python 修改为 #!/usr/bin/python2.6，我也没想太多，就以实际情况为主，没有去思考深沉次的原因

julyclyde

Apr 15, 2018

@wsgzao 现在，新装的 2.7 可以在以前 2.6 的 site-packages 目录里成功找到 yum 的源码了吗？

wsgzao

Apr 16, 2018

@julyclyde #9 从路径上来看应该是独立的，官方倒是也没有说明优化了哪些安装的方式，我实际测试从 python2.6 升级至 python2.7.13 没有问题，然后最近又升级到 python2.7.14 也是直接编译安装，相当顺利，后续估计要开始测试 python2.7 和 python3.6 使用 pipenv 共存的问题，估计又要折腾好久了

``` bash
test101@JQ/root#whereis python
python: /usr/bin/python2.6 /usr/bin/python /usr/lib/python2.6 /usr/lib64/python2.6 /usr/local/bin/python2.7 /usr/local/bin/python2.7-config /usr/local/bin/python /usr/local/lib/python2.7 /usr/include/python2.6 /usr/share/man/man1/python.1.gz
test101@JQ/root#cat /usr/bin/yum
#!/usr/bin/python

test101@JQ/root#find / -name python
/usr/share/doc/m2crypto-0.20.2/demo/ZopeX3/install_dir/lib/python
/usr/share/doc/m2crypto-0.20.2/demo/Zope27/install_dir/lib/python
/usr/share/doc/m2crypto-0.20.2/demo/Zope/lib/python
/usr/share/gdb/python
/usr/local/bin/python
/usr/bin/python
/root/.local/share/virtualenvs/pipenv-qb5OO9ES/bin/python
/root/Python-2.7.14-pip/Python-2.7.14/python
test101@JQ/root#find / -name site-packages
/usr/lib/python2.6/site-packages
/usr/local/lib/python2.7/site-packages
/usr/lib64/python2.6/site-packages
/root/.local/share/virtualenvs/pipenv-qb5OO9ES/lib/python2.7/site-packages
/root/Python-2.7.14-pip/Python-2.7.14/Lib/site-packages

```

julyclyde

Apr 18, 2018

@wsgzao 所以你的文章是既没有提到原有的问题，也没有论证新版没有这个问题，就直接武断的下结论说没问题？然后还要作为经验推广？
从你用 whereis 和 find 的做法来看，我猜你大概不知道 python 的 import 是从什么路径里找文件的？