What does HackerNews think of crater?
Run experiments across parts of the Rust ecosystem!
it's probably impossible to automate an entire ecosystem, and there is value to enabling a tighter integration within a project ecosystem (a subset of the language ecosystem).
> new proposed C++ changes - are checked against only easily and "well-known" accessible package.
Now that I have, so to say, shown you mine, lets see yours. Where is the tool to perform these checks in C++?
https://rustc-dev-guide.rust-lang.org/tests/crater.html
https://github.com/rust-lang/crater
Even then such a bug as GHC's can conceivably not be detected.
> Linux is really hurt here by the total lack of any unit testing or UI scripting standards.
> standards
I've been very impressed reading how the Rust developers handle this. They have a tool called crater[1], which runs regression tests for the compiler against all Rust code ever released on crates.io or GitHub. Every front-facing change that is even slightly risky must pass a crater run.
https://github.com/rust-lang/crater
Surely Microsoft has internal tools for Windows that do the same thing: run a battery of tests across popular apps and make sure changes in the OS don't break any user apps.
Where's the similar test harness for Linux you can run that tests hundreds of popular apps across Wayland/X11 and Gnome/KDE/XFCE and makes sure everything still works?
Arguably the nearest thing to a specification for the Rust language at present is "what crater would be happy with".
That seems more useful than thinking of rustc as a reference implementation, because when I read the release notes every now and again they say things to the effect of "this change is technically backwards-incompatible but crater reassures us that it won't cause anyone any trouble".
This is generally how dependencies work across all languages and frameworks, and I think the status quo is extremely inefficient. When a library maintainer releases updates with breaking changes, we waste thousands of developer-hours on upgrades and fixing all these changes.
I've been thinking about a new kind of package manager and repository that strictly enforces a certain level of test coverage, and also forces library maintainers to codify the upgrade process. For example, if you decide that you want to rename a `get` method in `v0.1.0` to `fetch` in `v1.0.0`, then your `v1.0.0` release MUST include an automated code-mod script that automatically runs when a developer updates the library version. This wouldn't work so well for dynamically typed languages, but this rename example should work flawlessly in a statically-typed language.
The code-mod script should also include lots of test cases, and be tested on lots of other real-world libraries. Similar to how Rust developers detect regressions in the Rust compiler by building a large number of crates.
There's probably a lot of reasons why this wouldn't work in the real world, but one can dream! There was a lot of controversies when Elm made this kind of decision about their package ecosystem and removed native modules [2]. Deno is working on a high quality Standard Library [4] with no external dependencies, and where all code is reviewed by the Deno core team (and presumably with a high level of test coverage.)
[1] https://github.com/rust-lang/crater
[2] https://dev.to/kspeakman/elm-019-broke-us--khn
Rust has Crater which allows new rustc builds to be run against the entire public crates.io dgraph.
The rust devs recompile a huge chunk of the entire rust ecosystem to make sure new compiler versions don't break things.
I can't imagine something like this being even close to feasible in the C++ ecosystem; there is too little standardization of build tooling, too much brittle/broken code.
Still it's not cheap and Amazon is paying, bless their hearts and excess compute capacity when it's not Christmas.
I think crater also deserves some credit here - given how much Rust source is in Cargo, it's very useful to be able to run a refactored compiler against all those packages and see which don't compile or start failing their unit tests.