I think there is a great potential for highly interactive predictive suggestions like shown under section 2 "discoverability" in the wider Unix ecosystem, however discoverability is the key issue, and a massive problem space. Command line programs are black boxes (they could be pure bash, a compiled C++ binary, an npm package, etc), which means there is no programmatic standard for extracting the command set.
The most backwards compatible format to achieving this would be a interface like abstraction on top of regular binaries (similar to the concept of TS interfaces adding type completion to JS) which describe a list of parameters/flags and their valid values/data types with an associated description. I think a system similar to this gives an intuitive and extensible suggestion system without requiring inconceivable redesigns to existing systems whilst maintaining the familiarity and ease of use of the regular command line.
You say that but it's actually an easier problem to solve than it first appears. Both my shell, murex[0], and Fish[1] parse man pages to provide automatic autocompletions for flags.
I also have a secondary parser for going through `--help` (et al) output for programs that don't have a man page. This isn't something that is enabled by default (because of the risk of unintended side effects) but it does work very effectively.
They way I look at it though, is you want to cover as many things automatically as you can but there will always be instances where you might want to manually fine tune the completions. For that, I've also defined a vastly simplified autocompletion description format. It's basically a JSON schema that covers 99% use cases out of the box and supports shell code to be included to cover the last 1% of edge cases.
This means you can write autocompletions to cover common usecases and have the shell automatically fill in any gaps you might have missed.