It's a nice exercise, but I don't see t going anywhere.
1st. I don't see that many bugs in the TCP/IP, filesystems, etc. specially those exploitable by black hat hackers, specially remote exploits. We usually see wide-spread vulnerabilities in the middleware like OpenSSH, which is in the userspace, putting everything in the app is not going to solve that, quite the contrary, we need to wait for the library maintainer to fix the bug, for the applications to integrate with the newer version of the library, and then, update all the applications.
2nd. Side-channel attacks. I'm not really confident in letting applications have that low level of hardware access. I mean, it's really easy for one application to steal the resources from another. We need authority if we are sharing peripherals. Well, we have one peripheral already that is shared by processes, that's memory and look at the mess that is a modern MMU and all the attacks on those that we have seen throughout the decades.
3rd. If we are going to implement that as a shared library, the result is exactly the same as today's micro and hybrid kernels, if we are not going to save memory and implement that as a static library, it's exactly the save as virtualization.
Correct me if I'm wrong.
I would (in theory), love a unikernel for the work I do. Our bare metal boxes run about 20 Unix processes -- cron, syslog, sshd, etc plus one server daemon. It's nice to have a familiar environment, but we could run on fewer boxes without a general purpose kernel and general purpose memory protection that we don't strictly need, especially for nodes that could easily be tweaked to forgo local disk access (because filesystems are tricky, I'd rather leave nonephemeral storage with something proven)
That said, we're not moving in the direction of unikernels, but it has a clear application in my mind. It's a fair bit of work to migrate, and I've yet to see a practical comparison, so it's unclear what the actual benefits over conceptual benefits would be.
What is the problem domain?
Chat service, in Erlang. I'd love to just boot to Erlang, and have as much of the system as possible be hotloadable, including the tcp stack. But I may be suffering from too much exposure to hot loading. :)
Sounds like a great problem to work on! When you have such a well defined problem it can be a great ticket to do good work since in some ways you can control the stack top to bottom. Have fun!
I'm not actually working on it, unfortunately. There's too much risk, not enough time, and we're in the middle of migrating to a different environment where it would be an even weirder thing. But I think it's a good example of a place it could work, and it would be a lot of fun.