Mirror

Mirror free and open-source projects you like with minimal effort.

Use this project as a service to keep a local or hosted copy of the projects source code or their assets.

Status

This project is in pre-alpha and under active development. Expect rapid and breaking changes.

Planned improvements and known bugs are listed on the project's todo branch.

Requirements

The following applications are required on your system.

Golang >= 1.22.4
git
rsync

Verification

Verify the project and its assets using the GPG Key included in the project.

gpg --import GPG-KEY
git commit verify-commit HEAD

To verify the projects vendored dependencies:

go mod verify

Building and Installation

Slackware

You can create a package using the Slackbuild script provided with the project.

cd contrib/slackbuild
sudo sh *.SlackBuild

Other Unix-like systems

Standard practice is to install under /usr/local/sbin

go build ./cmd/...
sudo mv mirror /usr/local/sbin
sudo chmod +x /usr/local/sbin/mirror

You will then need to integrate it into your system's service management to your liking.

Configuration

Configuration is provided via TOML file format.

Refer to the following example. Mirror will mirror each project at a random interval between 1 and 24 hours.

[global]
# The minimum time to wait before mirroring.
min_interval = "1h"
# The maximum time to wait before mirroring.
max_interval = "24h"

[[mirrors]]
method = "rsync"
from = "rsync://mirrors.kernel.org/slackware/slackware64-current"
to = "/mirror/slackware/slackware64-current"

[[mirrors]]
method = "git"
from = "https://github.com/ytdl-org/youtube-dl"
to = "/srv/git/slackcoder/youtube-dl"
description = "Command-line program to download videos from YouTube.com and other video sites"

[[mirrors]]
method = "github-assets"
from = "https://github.com/ytdl-org/youtube-dl"
to = "/mirror/youtube-dl"

Configuration may also be split across files in a directory. By default loads configuration from /etc/mirror/mirror.toml and the /etc/mirror/conf.d directory.

Staging and Verification (Experimental)

You can ensure mirror integrity by verifying the project in a staging directory before saving it.

Use the 'verify' parameter to define Bash executed shell commands. A non-zero exit value is considered a verification failure. The standard error output is then written to the log.

Use the 'staging-method' and 'staging-path' parameters to customize how the incoming mirror version is stored. Both these values can be defined globally or specific to a mirror.

The possible 'staging-method' values are:

none: Save the data directly to the mirror destination.
temporary: Save the data in a directory to be deleted after verification.
persistent: Use a second location to store incoming data without clearing it after.

The 'staging-path' defines the parent directory for where staging occurs, by default it is '/tmp/mirror'.

[[mirrors]]
method = "rsync"
from = "rsync://mirrors.kernel.org/slackware/slackware64-15.0"
to = "/mirror/slackware/slackware64-15.0"
staging-path = "/tmp/mirror"
staging = "persistent"
verify = """
  (gpg --verify-files *.asc && tail +13 CHECKSUMS.md5 | md5sum -c --quiet -) && \
  (find slackware64 -print0 -name '*.asc' | xargs -0L1 gpg2 --verify-files)
"""