debian-multimirror

Yet another mirroring program? YES because it does what I need!
Mirrors of this page: mirror1, mirror2, mirror3
Artistic logo
more than 200 downloads! - 0 open bugs
Latest release is version: 2.5

News:

8/4/2006 - Version 2.5 is out! there's now support for signing release files. It's done with the gpg_key tag, like:
<gpg_key sign="on" key="pedro@larroy.com" />

To use it nicely: first, you need a working gpg key pair:

gpg --gen-key

Then, export the public key:

gpg -a --export [key] > key.asc

Now in clients, add the key:

cat key.asc | apt-key add -

Alternatively:

gpg --keyserver pgpkeys.mit.edu --recv-key 2D230C5F
gpg -a --export 2D230C5F | apt-key add -

14/10/2005 - Version 2.00 is out! debian-multimirror now handles sources.

What is it?

Multithreaded debian mirroring tool, written in perl with great flexibility for mirroring a custom subset of the debian OS. Features:

A tool for generating mirroring information from sources.list files.
Excluding, including and including the union of one or several list of packages. (For example the output of dpkg --get-selections from several machines). (completed!!)
Uses parallel programming so can exploit redundancy between mirrors and download from several mirrors at the same time for best performance and less mirror load. (Uses only one thread per mirror in order not to saturate individual mirrors).
Uses XML format for configuration file. (completed!! ).
Package information data is big, so memory consumption is around 80 Mb in the common case, and 30 Mb minimum (for one distribution and one host).
Uses a custom algorithm to select a consistent set of packages and currently chooses the most replicated package set.
It can still benefit from out of sync or even broken mirrors.
Checks SHA1 and MD5 package information for assuring data integrity.
Handles missing files on a mirror, and tries to get it from other

This was done to satisfy the following needs:

Partial mirroring with easy input of package lists, supporting basic set operations over them.
Excluding sections, such as games, kde or gnome for example, so more disk efficient.
Increased speed, usually I don't get good speed from any nearby mirror. Can use partial or incomplete mirrors.
Doesn't put your mirror offline while upgrading it.
Clean configuration file.

License & copyright:

This software is copyrighted. You can use, distribute and modify it according to the terms of the GPL license.

Software distribution:

You can download debian-multimirror software here.

Subversion access (SVN):

Get the latest development version with svn doing this:

svn co svn://larroy.com/deb_mm

And for updating your previous source:

svn up svn://larroy.com/deb_mm

Simplified architecture

Sample configuration file

<?xml version="1.0" encoding="UTF-8"?>

<config>
    <destdir>/home/ftp/debian-marillat</destdir>
    <tmpdir>/home/piotr/tmp/dm-marillat</tmpdir>
    <retries>5</retries>
    <work_unit_size>8</work_unit_size>
    <clean>1</clean>
    <url>ftp://ftp.nerim.net/debian-marillat</url>
    <distribution codename="etch">
        <component name="main" source="yes"></component>
        <type>binary-i386</type>
        <include>
			<all>1</all>
        </include>
        <exclude>
        </exclude>
    </distribution>
</config>

Sample output

localhost: get_packages_info()
ftp.at.debian.org: get_packages_info()
ftp.de.debian.org: get_packages_info()

Testing metadata of sid:
        localhost VS ftp.at.debian.org: python2.1-doc versions doesn't match
        localhost VS ftp.de.debian.org: python2.1-doc versions doesn't match
        ftp.at.debian.org VS ftp.de.debian.org: equal
Testing metadata of sarge:
        localhost VS ftp.at.debian.org: python2.1-doc versions doesn't match
        localhost VS ftp.de.debian.org: python2.1-doc versions doesn't match
        ftp.at.debian.org VS ftp.de.debian.org: equal

Package status:
        localhost, distribution:
                sid:
                        15122 packages
                        Score: 0
                sarge:
                        14247 packages
                        Score: 0
        ftp.at.debian.org, distribution:
                sid:
                        15622 packages
                        Score: 1
                sarge:
                        14817 packages
                        Score: 1
        ftp.de.debian.org, distribution:
                sid:
                        15622 packages
                        Score: 1
                sarge:
                        14817 packages
                        Score: 1
Choosed ftp.at.debian.org sid as master distribution metadata. (Most replicated)
Choosed ftp.at.debian.org sarge as master distribution metadata. (Most replicated)

Computing the list of files we need to get: generate_work()

We already have: 1680 MB
Need to get: 8685 MB
Redundant across other hosts: 25299 MB
Raw redundancy available for paralellism: 291 %
launch_workers()
Starting ftp.at.debian.org
Starting ftp.de.debian.org
Starting localhost

...and the party starts.

Other mirroring tools:

Comparison table

features	debian-multimirror	debmirror
parallel	y	n
redundant	y	n
writes Package files	y	n
does excludes	y	y
does includes	y	y
manages sections	y	y
partial mirroring	y	n²
supports package lists	y	n¹
written in	perl	perl
ram usage	medium-high	medium-low
included in debian	no	y

1: Not without additional scripting
2: Packages not mirrored are still on local Packages.gz causing brokeness

strawberrymilk