What does HackerNews think of cosmopolitan?

Whenever a community project goes commercial, its interests are usually no longer aligned with the community. For example, llama.com makes frequent backwards-incompatible changes to its file format. I maintain a fork of ggml in the cosmopolitan monorepo which maintains support for old file formats. You can build and use it as follows:

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan

    # cross-compile on x86-64-linux for x86-64 linux+windows+macos+freebsd+openbsd+netbsd
    make -j8 o//third_party/ggml/llama.com
    o//third_party/ggml/llama.com --help

    # cross-compile on x86-64-linux for aarch64-linux
    make -j8 m=aarch64 o/aarch64/third_party/ggml/llama.com
    # note: creates .elf file that runs on RasPi, etc.

    # compile loader shim to run on arm64 macos
    cc -o ape ape/ape-m1.c   # use xcode
    ./ape ./llama.com --help # use elf aarch64 binary above

It goes the same speed as upstream for CPU inference. This is useful if you can't/won't recreate your weights files, or want to download old GGML weights off HuggingFace, since llama.com has support for every generation of the ggjt file format.

Cosmopolitan[0] is indeed worth mentioning here. It's an amazing project.

[0] https://github.com/jart/cosmopolitan

Blink can absolutely emulate CPython. https://twitter.com/JustineTunney/status/1611264272637775873

I recommend obtaining Python from the Comsopolitan mono repo:

    git clone https://github.com/jart/cosmopolitan
    make -j8 o//third_party/python/python.com

That'll easily give you a static Python 3.6 binary that can be run under Blink.

Indeed! If your dependencies are able to be command line programs that are shell scripted together, then you can in fact have an access policy on a per-dependency basis, using the pledge.com program linked on my website. So shell scripters rejoice.

But it gets better. If you build Python in the Cosmopolitan Libc repository:

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan
    build/bootstrap/make.com -j8 o//third_party/python/python.com

Then you can use cosmo.pledge() directly from Python.

    $ o//third_party/python/python.com
    Python 3.6.14+ (Actually Portable Python) [GCC 9.2.0] on cosmo
    Type "help", "copyright", "credits" or "license" for more information.
    >>: import cosmo, socket
    >>: cosmo.pledge('stdio rpath wpath tty', None)
    >>: print('hi')
    hi
    >>: socket.socket()
    Traceback (most recent call last):
      File "", line 1, in 
      File "/zip/.python/socket.py", line 144, in __init__
        _socket.socket.__init__(self, family, type, proto, fileno)
    PermissionError: [Errno 1] EPERM/1/Operation not permitted

Since we didn't put "inet" in the pledge, you can now be certain your PyPi deps aren't spying on you or uploading your bitcoin wallet to the cloud. You can even use os.fork() to rapidly put each dependency in its own process, then call cosmo.pledge() afterwards to grant each component of your app its own maximally restrictive policy.

Cosmopolitan Python also ports OpenBSD's unveil() system call to Linux too. For example, to disallow all file system access, just call cosmo.unveil(None, None). You need a very recent version of Linux though. For instance, I use unveil() in production on GCE but I had to apt install linux-image-5.18.0-0.deb11.4-cloud-amd64 in order for Landlock LSM to be available to use unveil().

At 7:44 they go into the “magic numbers” used to define kernel syscalls like how Cosmopolitan [0] uses for building things like RedBean [1].

0. https://github.com/jart/cosmopolitan

1. https://redbean.dev/

They didn't build six executables that run on six operating systems. Rather, it's a single nativity compiled executable that runs on six different operating systems unmodified.

Cosmopolitan is an incredibly cool project that does more than you think.

https://github.com/jart/cosmopolitan

>This is just a hack in the respective executable file format headers (a genius hack nonetheless from Justine)

Its a lot more then that. Look at the code. https://github.com/jart/cosmopolitan You compile an executable with flags --nostdlib and --nostdinc because it includes cross platform standard libraries mapped to the OS specific magic numbers for system calls.

Think about a desktop computer with a graphics card. You can install either windows or linux on it. In both cases stuff gets rendered to the screen using the graphics cards. That is done through the driver, which maps higher level commands in libraries to instructions that graphics cards understands. While library format, is OS specific, the driver is essentially doing either ioctls or memory mapped io, both of which are the same x64 instructions.

So compiling everything with APE, means that library entry points are platform agnostic, and everything up the chain all the way to the game engine.

And, yes cosmopolitan only works on very basic things, as would be expected by the small number of people developing it, thats why I said IF it was adopted widely and developed further it would be possible.

That sounds like a good fit for redbean. How often are those industrial computers updated? I ask because backwards compatibility has always been a focus of the project. redbean runs on RHEL5+ (c. 2007) and Windows Vista+ (c. 2006) by default. See https://github.com/jart/cosmopolitan/#support-vector Lua is real nice and I offer support services too. See my email at https://github.com/jart/

Try this:

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan
    make -j8 o//third_party/python/python.com

Then put Django in the zip under the .python directory. You can also use the `.args` file just like redbean to have `-m module` run by default.

Author here. We have 364 separate test programs in the Cosmopolitan repository. However most of our testing comes from running the tests of huge existing projects that use libc, e.g. Python. Try running this on Linux:

    git clone https://github.com/jart/cosmopolitan.git
    cd cosmopolitan
    make -j16 o//third_party/python

That command will build Python and all its dependencies from scratch within the hermetic monorepo in addition to running its unit tests. On my $1,000 Core i9-9900 PC this takes 31.078 seconds.

Spoiler alert

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan
    make -j8 o//third_party/quickjs/qjs.com

Build it on Linux but it runs anywhere. https://github.com/jart/cosmopolitan/tree/master/third_party... Here's a prebuilt binary https://justine.lol/qjs.com

Got you covered. https://github.com/jart/cosmopolitan/blob/master/examples/ru...

    $ git clone https://github.com/jart/cosmopolitan && cd cosmopolitan
    $ make -j8 o//examples/rusage.com o//examples/hello.com
    $ o//examples/rusage.com o//examples/hello.com
    hello world 123
    RL: took 90µs wall time
    RL: ballooned to 108kb in size
    RL: needed 106µs cpu (0% kernel)
    RL: caused 13 page faults (100% memcpy)

This one has a tiny bit more overhead than the practical minimum you saw earlier in the benchmark. Note that APE binaries start off as shell scripts so the first run is going to be slower. It's something I'm working towards improving in a variety of ways. Here's the executive summary of the upcoming changes:

    int ws, pid;
    CHECK_NE(-1, (pid = fork()));
    if (!pid) {
      execve("ape.com", (char *const[]){"ape.com", 0}, environ);
      perror("execve");
      _Exit(127);
    }
    CHECK_EQ(pid, wait(&ws));
    CHECK_TRUE(WIFEXITED(ws));
    CHECK_EQ(0, WEXITSTATUS(ws));

Here's the latency of the various execution strategies:

    FORK+EXEC+EXIT+WAIT      │       APE │ APE-LOADER │ BINFMT_MISC
    ───────────────────────────────────────────────────────────────
    fork() + execve()        │      55µs │       66µs │        56µs
    vfork() + execve()       │      25µs │      446µs │        35µs
    /bin/sh -c ./ape (1st)   │     485µs │      457µs │       170µs
    /bin/sh -c ./ape (avg)   │     148µs │      466µs │       159µs

I think Cosmopolitan[1] is a beautiful example for self-modifying executables.

> Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable.

> Please note that your APE binary will assimilate itself as a conventional resident of your platform after the first run, so it can be fast and efficient for subsequent executions.

[1] https://github.com/jart/cosmopolitan

> You of course have to choose your target platform for this (...)

https://github.com/jart/cosmopolitan

(but, yeah, you probably want one version per target, even if hacks are cool)

Karmic Koala is within the support vector of Actually Portable Executable. Since if you can get it to run on that, it'll most likely run on all the other distros too. Plus these binaries run on Windows 7, Mac, and BSDs too. They even run without an operating system. No need for recompilation. No virtual machine required. They aren't bloated either since the binary footprint starts at 12kb. See https://justine.lol/ape.html and https://github.com/jart/cosmopolitan Best part is you can still use the compiler that comes with your Linux. All it does is reconfigures the compiler so that, rather than producing a binary that can only run on one version of one distro of one operating system, your program magically becomes capable of running on all of them.

> What I'm advocating for here requires that your program be compiled into a native executable binary file, and it must embed all its library dependencies (aka static linking).

the super example of this is justine's project

https://github.com/jart/cosmopolitan

it's pr fucking cool

> Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable.

Check out Cosmopolitan which vendors all its dependencies. https://github.com/jart/cosmopolitan You can't use Node with it, yet, but you can use QuickJS if you `make o//third_party/quickjs/qjs.com` and `o/third_party/quickjs/qjs.com -m third_party/quickjs/repl.js`. Cosmopolitan goes so far with vendoring dependencies that it actually has its own C library implementation that prevents functions like socket(), connect(), and getaddrinfo() from talking to the Internet if the software is running under GNU Make. Since sometimes we've found when porting software it'll have unit tests that do sneak leaks to public servers. It goes without saying that in order to make it even possible to vendor all dependencies, down to the binary level, there's a very long list of topologically ordered things that need to happen before you can even talk to the Internet. https://github.com/jart/cosmopolitan/blob/50937be7524197f23a... It's all cross platform and native at the same time for CLI/TUI. It'd surely make a more trustworthy tooling stack for a password manager than something like Electron since the temptation for people to incorporate things like FullStory into such GUIs is too great.

This is one of the reasons why I've been building Cosmopolitan Libc. https://github.com/jart/cosmopolitan Alpine and Musl aren't radical enough on the bloat-free angle (since they don't fix root problem which is dynamic shared objects) and their definition of portability is supporting architectures like sega saturn but do nothing to support intel/amd microarchitectural dispatching, which is likely why you encountered performance issues. It's like, I love Musl so much, but I just didn't see the value in being able to port my apps to Sega Saturn if my C Library only runs on Linux and has suboptimal performance on x86.

[side note] I seem to recall there being something about the the way the musl header files are written that basically throws a wrench into any scientific computing code that's compiled with it. Might have something to do with security like those checks intended to prevent people from overlapping memcpy so someone should totally fix that.

The fact that the IMF thinks your choice of operating system should impact your creditworthiness is something I've been warning people about for months because it shows why we need cosmopolitan software. Since it's nice to have the ability to pick up and move our apps elsewhere should we discover we're using the "wrong" one. Because political winds change, but good engineering endures. https://github.com/jart/cosmopolitan

And I would love to encourage you to do so! If you're working for Cosmopolitan then I'll be working hard to support you. There's a long history of depth to metal. Many people start off learning it by having fun with the old skool ibm pc boot process. Here's a gentle introduction: https://justine.lol/blinkenlights/realmode.html and https://github.com/jart/sectorlisp and http://reddit.com/r/osdev

Those concepts are implemented in the Cosmopolitan codebase here: https://github.com/jart/cosmopolitan/blob/master/ape/ape.S

That file runs a few hundred lines of old school assembly in order to bring us into the modern era. Towards the end of the file you'll notice it starts calling functions that are written in C which configure memory: https://github.com/jart/cosmopolitan/blob/master/libc/runtim...

After it configures memory, the ape.S file is able to call _start(). It also sets a bit in __hostos so that functions like write() know to use the serial port instead of issuing system calls: https://github.com/jart/cosmopolitan/blob/master/libc/calls/... That's the simplest possible example of a driver with spin locks which uses the x86 IN/OUT instructions.

If you want to take a dive off the deep end right now with Cosmopolitan on metal -- simulated -- then try the following build commands on your Linux terminal:

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan
    make -j8 o//examples/hello2.com
    qemu-system-x86_64 -m 16 -nographic -fda o//examples/hello2.com
    sed -i -e 's/USE_SYMBOL_HACK 0/USE_SYMBOL_HACK 1/' ape/ape.S
    make -j8 o//examples/hello2.com o//tool/build/blinkenlights.com
    o//tool/build/blinkenlights.com -rt o//examples/hello2.com

Author here. Absolutely. I used to love Turbo Pascal and Delphi when I was younger. If Free Pascal uses GNU LD.BFD or LLVM LLD when it links programs, then all you'd need to do is is configure it to use cosmopolitan.a when linking system call functions like read(), write(), etc. See https://github.com/jart/cosmopolitan Another option is if Free Pascal wants to write all the system call support from scratch, then doing that now is going to be a whole lot easier since the Cosmopolitan codebase does a really good job documenting all the magic numbers you'll need. See files like https://github.com/jart/cosmopolitan/blob/master/libc/sysv/s... and https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c... I've been working on a tiny C11 compiler called chibicc which has most GNU extensions and I managed to get it working as an actually portable executable with an integrated assembler: https://github.com/jart/cosmopolitan/blob/master/third_party... I also got Antirez's KILO text editor working as an APE binary. https://github.com/jart/cosmopolitan/blob/master/examples/ki... If we can build a linker too then we can get a freestanding single file toolchain + ide that's able to be a modern version of Turbo C.

Here's something you can paste into your Linux terminal to get started:

    git clone https://github.com/jart/cosmopolitan
    cd cosmopolitan
    make -j8
    o/third_party/chibicc/chibicc.com -include libc/integral/normalize.inc -o hello.com.dbg examples/hello.c o/cosmopolitan.a
    o/third_party/gcc/bin/x86_64-linux-musl-objcopy -SO binary hello.com.dbg hello.com
    ./hello.com

I used lots of code from Musl, OpenBSD, and FreeBSD. I used Marco Paland's printf. I used Doug Lea's malloc. I used LLVM compiler_rt. I used David Gay's floating point conversion utilities. The list goes on. Then I stitched it all together so it goes faster and runs on all operating systems rather than just Linux. See https://justine.lol/cosmopolitan/index.html and https://github.com/jart/cosmopolitan

Trapping (SYSCALL/INT) is a loose binding. The kernel can evolve all it wants internally. It can introduce new ABIs. Processes are also a loose binding. I think communicating with other tools via pipes and sockets is a fantastic model of cooperation. Same goes for vendoring with static linking. Does that mean I'm going to voluntarily load Linux distro DSOs into my address space? Never again. Programs that do that, won't have a future outside Docker containers.

Also, my executables are VM images. They can boot on metal too. Read https://justine.lol/ape.html and https://github.com/jart/cosmopolitan/issues/20#issuecomment-... Except unlike a Docker distro container, my exes are more on the order of 16kb in size. That's how fat an executable needs to be, in order to run on six different operating systems and boot from bios too.

Reading your words, it looks like you have made a kind of make replacement that just works. Following your link, it looks like... what ? I don't know, some tool that checks some build configuration, augments C and postprocesses executable files?

(Looking at https://github.com/jart/cosmopolitan it's an interesting hack. Might be useful in some cases, for example command-line tools like git.)

Have you made something that looks like a better make? If so, it would be nice to know more. Thanks.

I created the Occupy Wall Street website nine years ago. Believe me when I say the lengths people will go, to try and control community projects, is downright traumatizing. I never could have imagined that same kind of nastiness would impact open source.

If you don't feel comfortable engaging with the new toxic culture, you can use my underground liferaft. My liferaft isn't an operating system, but rather an attempt to help us not depend on them as much. I have no idea who's controlling GNU/Linux these days and Occupy was enough drama for one lifetime. My code has the same focus on clarity that Antirez put into his Redis codebase. My liferaft also empowers you to build and distribute tiny native portable programs (like Kilo!) using hermetically sealed tools. See https://github.com/jart/cosmopolitan

Static linking (https://github.com/jart/cosmopolitan)

    jart@debian:~/cosmo$ make -j12 CPPFLAGS+=-DIM_FEELING_NAUGHTY MODE=tiny o/tiny/examples/hello.com
    jart@debian:~/cosmo$ ls -sh o/tiny/examples/hello.com
    20K o/tiny/examples/hello.com

Note: Output binary runs on Windows, Mac, and BSD too.