Yeah, if you have a high res image, you can get color info at super low-res and then regenerate the colors at high res with another model. (though this isn't an efficient approach at all)
https://github.com/TencentARC/T2I-Adapter
i've also seen a controlnet do this.
You can do this with spatial palette t2i or controlnet. Give a super lores spatial palette as conditioning like this:
https://camo.githubusercontent.com/8e488996fd309165fb065b0cd...
It's the T2I adapter rather than control net(https://github.com/TencentARC/T2I-Adapter). Other than that, the big thing is that it uses SD XL
This already exists, check out the Spatial Palette (Color) Adapter here:
https://github.com/TencentARC/T2I-Adapter
It works with the Mikubill ControlNet plugin for A1111.