This has been the case for quite a while now and I cannot for the life of my figure out why that is. I never explicitly made it this way.
It's only in Firefox and a machine with pretty much the exact same #NixOS config does not exhibit this issue.
I tried clearing unused fonts out of \~/.local/share/fonts/ (no emoji fonts remain) and updated the fc-cache but it didn't help. (Obviously I restarted Firefox.)