Setting up a private egg repository
Posted 2009-06-10 18:15:52 by .

Related tags: python pipy easyinstall apache setuptools easy_install

 

 

The Python setuptools package system and its "egg" repository is a easy way of distributing software. Individual packages can list other packages they depend on. At install time these dependencies are resolved, recovered and installed along with your package. All you have to do is create a setuptools friendly "setup.py" and upload into PyPi.

But what if you don't want to make your package public? This is a common case if you are a closed commercial project and don't wish to make the source code available. The answer is to create a private egg repository hosted on your own web server. This has a number of advantages:

  • Your packages are only available to a restricted set of people.
  • Public packages can still be listed as dependencies and fetched from elsewhere.
  • You can copy public eggs into the private repository to cache them locally.

 

Having local copies of public packages also helps if the public site is temporarily unavailable. The basic set up process is quite straight forward:

       1.  Create a place on the file system to store eggs.
       2.  Get a web server to share this out with basic authentication.
       3.  Copy eggs to the location on disk.

 

To access the private repository you use the EasyInstall find links option (-f).

1
 
easy_install -f http://<username:pass>@<my domain>/<egg basket>  <my private package>

 

Create The Egg Store On the File System

I create a directory like "/opt/webdistrib" and change the owner and permissions so that Apache can access it.

1
2
3
4
5
 
sudo mkdir -p /opt/webdistrib

# www-data is a command user/group that apache set's up when installed.
# It may be different for your system.
sudo chown www-data:www-data /opt/webdistrib

When you copy new egg files into this directory you will need to make sure Apache has permission to access it.

 

Apache Set Up

I use the Apache web server to host my own repository. Other web servers can be used however I'm not going to go into that here. You can add the following Alias and Directory commands into an existing configuration, modifying it for your needs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 
# Allow the download of apps and other things from a shared web folder (basic auth protected).
Alias /basket /opt/webdistrib
<Directory "/opt/webdistrib">
    AuthUserFile /opt/<path to secure place>/access
    AuthName "<My Company> - Distrib"
    AuthType Basic
    require valid-user
    order deny,allow
    allow from all

    Order allow,deny
    Allow from all
    Options +Indexes
</Directory>

This will set a URL like "http://<my domain dot com>/basket" for example. When a user goes to this address a basic authentication box will pop up. However before anyone can access it we must create the password file.

1
2
3
4
5
 
# Use -c only the first time the file is created!
htpasswd -c /opt/<path to secure place>/access webuser

# Add further users as follows:
htpasswd /opt/<path to secure place>/access webuser2

Now restart Apache once the configuration passes all checks. You should be able to login an see the contents of the basket.

 

VirtualHost Example

The following is an example of a complete VirtualHost entry I use on my home server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
<VirtualHost 192.168.0.1:*>

    DocumentRoot /opt/web/sites/www
    ServerName banter
    ServerAdmin mail@banter

    <Directory  /opt/web/sites/www>
        Order allow,deny
        allow from all
    </Directory>

    # Allow the download of apps and other things from a shared web
    # folder (basic auth protected).
    Alias /distrib /opt/webdistrib
    <Directory "/opt/webdistrib">
        AuthUserFile /opt/<path to secure place>/access
        AuthName "<My Company> - Distrib"
        AuthType Basic
        require valid-user
        order deny,allow
        allow from all

        Order allow,deny
        Allow from all
        Options +Indexes
    </Directory>

</VirtualHost>

 

Easy Install

To use easy_install to recover your private package you will need to use the find links (-f) flag. If you have password protected the URL you will also need to pass the username and password in the URL.

1
 
easy_install -f http://<username:pass>@<my domain>/<egg basket>  <my private package>

 

"Gotcha"

Easy install will fail silently if it cannot log in using the given username and password. You will need to check the Apache logs to see what error is occuring.

 

Example setup.py

I thought it might be useful to also include a little section on making and developing python eggs. The following is a template setup.py file I use to make a package into an egg.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
"""
Project's setuptool configuration.

This should eggify and in theory upload to pypi without problems.

I typically layout a package on the file system as follows:

egg package name
       |
       +-- lib
       |   |
       |   +--  package name
       |        |
       |        +-- tests - unit test files
       |
       +-- scripts - command line tool


Oisin Mulvihill
2009-06-10

"""
from setuptools import setup, find_packages

Name='<egg package name>'
ProjecUrl="" # I don't use this
Version='<package version number e.g. 1.0.0>'
Author='<Your Name>'
AuthorEmail='<You at some place dot domain>'
Maintainer=' <Your Name>'
Summary='<A library to do...>'
License='<Up to you really. I sometimes use: Commercial, All rights reserved.>'
ShortDescription=Summary
Description=Summary

TestSuite = '<package name.tests unit tests>'

needed = [
    # Example of third-party dependancy listing:
    #'stomper',
    #'mako',
    #'configobj',

    # You can also list packages in your private repository here as well.
]

EagerResources = [
    # If you need to include all content inside your package uncomment an modify this.
    # All directories to be include in your project will need a __init__.py file of they
    # won't be packaged.
    #'<your package name>',
]

ProjectScripts = [
    # I use this in addition to EntryPoints for occasions where I'm doing straigh distutils install.
    #'scripts/<command line script>',
]

PackageData = {
    # Include every file type in the egg file:
    '': ['*.*'],
}

# Make executable versions of the scripts:
EntryPoints = {
    #'console_scripts': [
    #    '<script name> = <python path to main function inside you package e.e=g <package name>.main:main',
    # ]
}

setup(
    url=ProjecUrl,
    zip_safe=False,
    name=Name,
    version=Version,
    author=Author,
    author_email=AuthorEmail,
    description=ShortDescription,
    long_description=Description,
    license=License,
    test_suite=TestSuite,
    scripts=ProjectScripts,
    install_requires=needed,
    packages=find_packages('lib'),
    package_data=PackageData,
    package_dir = {'': 'lib'},
    eager_resources = EagerResources,
    entry_points = EntryPoints,
)

 

Build and egg

Building an egg from the example setup.py is a straight forward task. You make the following call from the command line.

1
 
python setup.py bdist_egg

 

Developing your egg

If you wish to work on your egg, without having to build and install it each time, then you can use the following command.

1
 
python setup.py develop -f <auth details and address to your internal dependancies from>

This will 'install' your package into the site-packages using a special link file. Any third party or private dependencies will also be installed.

 

 

Comments

No Comments. Be the first to add one.

New Comment
Your email is never published nor shared.
Required fields are marked *
*
*