What does HackerNews think of General?

Julia and Rust all work with using versions tied to git tags. Package management isn't tied to Github but git: packages can live on Gitlab or BitBucket, though they generally don't and that's the choice of package developers. Because of that there are tie-ins to make Github really nice to use, but for example with Julia the only piece that is truly Github based is the fact that the General registry lives in a Github repo (https://github.com/JuliaRegistries/General), but could easily migrate to another git platform if it needed to.

Admittedly not a huge dataset but the TTFP is significantly shorter. :P

       _       _ _(_)_     |  Documentation: https://docs.julialang.org
      (_)     | (_) (_)    |
       _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
      | | | | | | |/ _` |  |
      | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
     _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
    |__/                   |
    
    (@v1.6) pkg> add Plots
        Updating registry at `~/.julia/registries/General`
        Updating git-repo `https://github.com/JuliaRegistries/General.git`
       Resolving package versions...
       Installed Qt_jll ─ v5.15.2+3
      Downloaded artifact: Qt
        Updating `~/.julia/environments/v1.6/Project.toml`
      [91a5bcdd] + Plots v1.11.0
        Updating `~/.julia/environments/v1.6/Manifest.toml`
      [ede63266] ↑ Qt_jll v5.15.2+2 ⇒ v5.15.2+3
      Progress [========================================>]  246/246
    246 dependencies successfully precompiled in 140 seconds
    
    julia> @time using Plots
      3.689727 seconds (6.58 M allocations: 472.965 MiB, 7.49% gc time, 0.13% compilation time)
    
    julia> @time begin
           using Plots
           x = 1:10; y = rand(10); # These are the plotting data
           plot(x, y)
           end
      3.050765 seconds (3.63 M allocations: 218.824 MiB, 4.87% gc time, 59.07% compilation time)
    
    julia> @time begin
           using Plots
           x = 1:10; y = rand(10); # These are the plotting data
           plot(x, y)
           end
      0.001435 seconds (2.61 k allocations: 161.836 KiB)

> (@v1.6) pkg> add Plots Installing known registries into `~\.julia` Cloning registry from "https://github.com/JuliaRegistries/General.git" Fetching: [==> ] 3.4 %

I gave up because it's just too slow (more than 15 min to get to 3%).

Maybe plotting is indeed faster, but I don't think I'll actually get to that point.

The PkgServer is an open source caching server, available here: https://github.com/JuliaPackaging/PkgServer.jl

The PkgServer protocol serves content-addressed chunks of data to Julia Pkg clients, so to download a certain version of a package, the Pkg client will request things like `GET /package/${pkg_uuid}/${content_hash}`. If you can pre-fill a PkgServer with all of the packages that you want, then it can serve a client just fine.

The full design is that there are a small number of "Storage Servers" that continually explore the global registry of packages (called `General`, located here: https://github.com/JuliaRegistries/General), downloading and storing tarballs for every version of every package that is available. These storage servers store everything forever, while Pkg servers contain an LRU cache to allow them to be deployed close to whatever compute resources will be requesting packages. For an airgapped solution, you could generate a static snapshot of some selection of the packages you want to serve, then serve them with nginx and point to that "static storage server" with the opensource Pkg Server, and it would all "just work".

An example of how to generate a static storage server is here: https://github.com/JuliaPackaging/PkgServer.jl/blob/master/b..., you would serve the resultant directory structure with something like nginx, then point the Pkg Server to that server as the upstream storage server.

I will note that Julia Computing offers an enterprise solution for dealing with secure/restrictive environments called JuliaTeam which provides this in a convenient, managed bundle, along with many other useful features.

EDIT: Ah, I forgot to mention, Stefan and I gave a talk at JuliaCon 2020 that touches on some of this, here's a link to the timestamp of the relevant section: https://youtu.be/xPhnJCAkI4k?t=350

And for more info, here's the original planning issue (note some things have changed as we've implemented it over the last year, but the bones are the same): https://github.com/JuliaLang/Pkg.jl/issues/1377#issue-492482...