I had thought it was about the color profile because with hdr disabled from system settings, enabling the built in color profile desaturates colors quite a bit and does some kind of perceived brightness to luminosity mapping that desaturates bright / dark hdr content even more. Although I don't think that's the cause of my problems anymore.

Thanks to your tip about kscreen-doctor, I could try different combinations of hdr / wcg / edid and see how the colors look with different combinations:

I think there must be something wrong with my screen since the hdr reduces saturation more than anything else. Anyways, thanks for the good work

Edit: Tried this with an amd gpu. hdr+wcg works as expected without muted colors. hdr without wcg still significantly desaturates colors, so I guess that's a monitor bug. Now to figure out gpu passthrough.. (Edit 2: It seems to just work??)

Side note, when I turn off hdr only from kscreendoctor the display stays in hdr mode until it turns off and on again, that didn't happen with nvidia

Edit 3: Found something weirder... Hdr colors are muted on nvidia gpu and seems vibrant with the amd igpu. If I plug the monitor to the motherboard (amd), enable hdr, then unplug and plug it into the nvidia gpu, the colors are still vibrant??? I can disable and enable hdr again and again and they aren't affected. They're even fine when hdr is enabled without wcg???? But if I fully turn off the monitor and back on they once again become muted with hdr. Weird ass behavior