mirror

Mirror free and open-source projects you like with minimal effort
git clone git://git.server.ky/slackcoder/mirror
Log | Files | Refs | README

README.md (3428B)

      1 # Mirror
      2 
      3 Mirror free and open-source projects you like with minimal effort.
      4 
      5 Use this project as a service to keep a local or hosted copy of the projects
      6 source code or their assets.
      7 
      8 ## Status
      9 
     10 This project is in pre-alpha and under active development.  Expect rapid and
     11 breaking changes.
     12 
     13 Planned improvements and known bugs are listed on the project's
     14 [todo](https://git.server.ky/slackcoder/mirror/tree?h=todo) branch.
     15 
     16 ## Requirements
     17 
     18 The following applications are required on your system.
     19 
     20  - Golang >= 1.22.4
     21  - git
     22  - rsync
     23 
     24 ## Verification
     25 
     26 Verify the project and its assets using the GPG Key included in the project.
     27 
     28 ```
     29 gpg --import GPG-KEY
     30 git commit verify-commit HEAD
     31 ```
     32 
     33 To verify the projects vendored dependencies:
     34 
     35 ```
     36 go mod verify
     37 ```
     38 
     39 ## Building and Installation
     40 
     41 ### Slackware
     42 
     43 You can create a package using the Slackbuild script provided with the project.
     44 
     45 ```
     46 cd contrib/slackbuild
     47 sudo sh *.SlackBuild
     48 ```
     49 
     50 ### Other Unix-like systems
     51 
     52 Standard practice is to install under /usr/local/sbin
     53 
     54 ```
     55 go build ./cmd/...
     56 sudo mv mirror /usr/local/sbin
     57 sudo chmod +x /usr/local/sbin/mirror
     58 ```
     59 
     60 You will then need to integrate it into your system's service management to
     61 your liking.
     62 
     63 ## Configuration
     64 
     65 Configuration is provided via [TOML](https://toml.io) file format.
     66 
     67 Refer to the following example.  Mirror will mirror each project at a random
     68 interval between 1 and 24 hours.
     69 
     70 ```
     71 [global]
     72 # The minimum time to wait before mirroring.
     73 min_interval = "1h"
     74 # The maximum time to wait before mirroring.
     75 max_interval = "24h"
     76 
     77 [[mirrors]]
     78 method = "rsync"
     79 from = "rsync://mirrors.kernel.org/slackware/slackware64-current"
     80 to = "/mirror/slackware/slackware64-current"
     81 
     82 [[mirrors]]
     83 method = "git"
     84 from = "https://github.com/ytdl-org/youtube-dl"
     85 to = "/srv/git/slackcoder/youtube-dl"
     86 description = "Command-line program to download videos from YouTube.com and other video sites"
     87 
     88 [[mirrors]]
     89 method = "github-assets"
     90 from = "https://github.com/ytdl-org/youtube-dl"
     91 to = "/mirror/youtube-dl"
     92 ```
     93 
     94 Configuration may also be split across files in a directory.  By default
     95 loads configuration from /etc/mirror/mirror.toml and the /etc/mirror/conf.d
     96 directory.
     97 
     98 ## Staging and Verification (Experimental)
     99 
    100 You can ensure mirror integrity by verifying the project in a staging directory
    101 before saving it.
    102 
    103 Use the 'verify' parameter to define Bash executed shell commands.  A non-zero
    104 exit value is considered a verification failure.  The standard error output is
    105 then written to the log.
    106 
    107 Use the 'staging-method' and 'staging-path' parameters to customize how the
    108 incoming mirror version is stored.  Both these values can be defined globally
    109 or specific to a mirror.
    110 
    111 The possible 'staging-method' values are:
    112 
    113  - none: Save the data directly to the mirror destination.
    114  - temporary: Save the data in a directory to be deleted after verification.
    115  - persistent: Use a second location to store incoming data without clearing it after.
    116 
    117 The 'staging-path' defines the parent directory for where staging occurs, by default it is '/tmp/mirror'.
    118 
    119 ```
    120 [[mirrors]]
    121 method = "rsync"
    122 from = "rsync://mirrors.kernel.org/slackware/slackware64-15.0"
    123 to = "/mirror/slackware/slackware64-15.0"
    124 staging-path = "/tmp/mirror"
    125 staging = "persistent"
    126 verify = """
    127   (gpg --verify-files *.asc && tail +13 CHECKSUMS.md5 | md5sum -c --quiet -) && \
    128   (find slackware64 -print0 -name '*.asc' | xargs -0L1 gpg2 --verify-files)
    129 """
    130 ```