pypi-mirror team mailing list archive

[Daniel Kraft <dk-launchpad@xxxxxx>]

Hi,

I was just thinking about the problem that many packages only provide a
download URL instead of the files on pypi. We can sync some of them
(easily if they provide direct download links and ugly/heavily if they
provide a download-list-page-URL). This code is ready but *very* slow,
because there are no hints like md5 checksums on foreign sites. In
addition those sites are sometimes *extremely* slow or even down.

We have two options:

1. Mirror those packages
This means, we will have the packages right now in the mirrors (if we
put around 2-3 hours in coding exceptions), but we might lose a lot of
uptime and reliability because it is very difficult to decide what to do
if there are no download links  on the download page, because we don't
know if it's a temporary or permanent loss - and a lot more unknowns
;)). The sync-time will also increase from currently 5 minutes to about
4..10 hours.
We would then officially support the usecase where package maintainers
only provide a link to pypi.

2. Ignore those packages
This means, we have to try to convince them to upload their packages to
pypi because otherwise the packages are not mirrored and people who try
to use the package will get error messages. This happens currently if
you try
  easy_install -i http://pypi.d9t.de/ z3c.pypimirror
because of the unmet dependency on BeautifulSoup, which isn't hosted by
pypi.

I'd prefer 2., as it is MUCH easier to maintain. I even think that
meeting 1. is even easier manually than automatically.
But for 2. we need some people really using our mirrors ;)


Daniel