Setting up a private egg repository is easy!
Contents
The Python setuptools package system and its "egg" repository is a easy way of distributing software. Individual packages can list other packages they depend on. At install time these dependencies are resolved, recovered and installed along with your package. All you have to do is create a setuptools friendly "setup.py" and upload into PyPi.
But what if you don't want to make your package public? This is a common case if you are a closed commercial project and don't wish to make the source code available. The answer is to create a private egg repository hosted on your own web server. This has a number of advantages:
- Your packages are only available to a restricted set of people.
- Public packages can still be listed as dependencies and fetched from elsewhere.
- You can copy public eggs into the private repository to cache them locally.
Having local copies of public packages also helps if the public site is temporarily unavailable. The basic set up process is quite straight forward:
- Create a place on the file system to store eggs.
- Get a web server to share this out with basic authentication.
- Copy eggs to the location on disk.
To access the private repository you use the EasyInstall find links option (-f).
1 | easy_install -f http://<username:pass>@<my domain>/<egg basket> <my private package>
|
Create The Egg Store On the File System
I create a directory like "/opt/webdistrib" and change the owner and permissions so that Apache can access it.
1 2 3 4 5 | sudo mkdir -p /opt/webdistrib
# www-data is a command user/group that apache set's up when installed.
# It may be different for your system.
sudo chown www-data:www-data /opt/webdistrib
|
When you copy new egg files into this directory you will need to make sure Apache has permission to access it.
Apache Set Up
I use the Apache web server to host my own repository. Other web servers can be used however I'm not going to go into that here. You can add the following Alias and Directory commands into an existing configuration, modifying it for your needs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # Allow the download of apps and other things from a shared web folder (basic auth protected).
Alias /basket /opt/webdistrib
<Directory "/opt/webdistrib">
AuthUserFile /opt/<path to secure place>/access
AuthName "<My Company> - Distrib"
AuthType Basic
require valid-user
order deny,allow
allow from all
Order allow,deny
Allow from all
Options +Indexes
</Directory>
|
This will set a URL like "http://<my domain dot com>/basket" for example. When a user goes to this address a basic authentication box will pop up. However before anyone can access it we must create the password file.
1 2 3 4 5 | # Use -c only the first time the file is created!
htpasswd -c /opt/<path to secure place>/access webuser
# Add further users as follows:
htpasswd /opt/<path to secure place>/access webuser2
|
Now restart Apache once the configuration passes all checks. You should be able to login an see the contents of the basket.
VirtualHost Example
The following is an example of a complete VirtualHost entry I use on my home server.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | <VirtualHost 192.168.0.1:*>
DocumentRoot /opt/web/sites/www
ServerName banter
ServerAdmin mail@banter
<Directory /opt/web/sites/www>
Order allow,deny
allow from all
</Directory>
# Allow the download of apps and other things from a shared web
# folder (basic auth protected).
Alias /distrib /opt/webdistrib
<Directory "/opt/webdistrib">
AuthUserFile /opt/<path to secure place>/access
AuthName "<My Company> - Distrib"
AuthType Basic
require valid-user
order deny,allow
allow from all
Order allow,deny
Allow from all
Options +Indexes
</Directory>
</VirtualHost>
|
Easy Install
To use easy_install to recover your private package you will need to use the find links (-f) flag. If you have password protected the URL you will also need to pass the username and password in the URL.
1 | easy_install -f http://<username:pass>@<my domain>/<egg basket> <my private package>
|
Example setup.py
I thought it might be useful to also include a little section on making and developing python eggs. The following is a template setup.py file I use to make a package into an egg.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | """
Project's setuptool configuration.
This should eggify and in theory upload to pypi without problems.
I typically layout a package on the file system as follows:
egg package name
|
+-- lib
| |
| +-- package name
| |
| +-- tests - unit test files
|
+-- scripts - command line tool
Oisin Mulvihill
2009-06-10
"""
from setuptools import setup, find_packages
Name='<egg package name>'
ProjecUrl="" # I don't use this
Version='<package version number e.g. 1.0.0>'
Author='<Your Name>'
AuthorEmail='<You at some place dot domain>'
Maintainer=' <Your Name>'
Summary='<A library to do...>'
License='<Up to you really. I sometimes use: Commercial, All rights reserved.>'
ShortDescription=Summary
Description=Summary
TestSuite = '<package name.tests unit tests>'
needed = [
# Example of third-party dependancy listing:
#'stomper',
#'mako',
#'configobj',
# You can also list packages in your private repository here as well.
]
EagerResources = [
# If you need to include all content inside your package uncomment an modify this.
# All directories to be include in your project will need a __init__.py file of they
# won't be packaged.
#'<your package name>',
]
ProjectScripts = [
# I use this in addition to EntryPoints for occasions where I'm doing straigh distutils install.
#'scripts/<command line script>',
]
PackageData = {
# Include every file type in the egg file:
'': ['*.*'],
}
# Make executable versions of the scripts:
EntryPoints = {
#'console_scripts': [
# '<script name> = <python path to main function inside you package e.e=g <package name>.main:main',
# ]
}
setup(
url=ProjecUrl,
zip_safe=False,
name=Name,
version=Version,
author=Author,
author_email=AuthorEmail,
description=ShortDescription,
long_description=Description,
license=License,
test_suite=TestSuite,
scripts=ProjectScripts,
install_requires=needed,
packages=find_packages('lib'),
package_data=PackageData,
package_dir = {'': 'lib'},
eager_resources = EagerResources,
entry_points = EntryPoints,
)
|
Build and egg
Building an egg from the example setup.py is a straight forward task. You make the following call from the command line.
1 | python setup.py bdist_egg
|
Developing your egg
If you wish to work on your egg, without having to build and install it each time, then you can use the following command.
1 | python setup.py develop -f <auth details and address to your internal dependancies from>
|
This will 'install' your package into the site-packages using a special link file. Any third party or private dependencies will also be installed.