After years of looking at this and working with x11 forwarding off and on. Honestly, just setup a VNC server and use the plethora of VNC clients for android. In my environment it performs better, and significantly easier to maintain. For my desktop I even find myself just using Steam Remoteplay if I need actual GPU performance over the internet.
Virtualgl +VNC is excellent if you get it working.
Yeah I might still go that route, but there's a lot of stuff where I just need to run a single app and the rest is all terminal so a full VNC session is overboard.
Also, it's for a server so the less open ports and daemons running, the better
Totally fair, but you don't have to open a port and could do ssh port forwarding if you are already going the ssh X forwarding route. I would try to measure the performance difference between X forwarding and a something like tigervnc. Depending on the specs of your server/clients it might not be very noticeable.
install an x server app in your phone (e.g. XSDL) and start it
install Termux from f-droid (the one in play store hasn't been updated due to changes in play store policy that prevents Termux package manager from working)
in Termux, run something like DISPLAY=127.0.0.1:0 ssh -Y user@hostname
Source code is available at GitHub but it require you to have your phone by yourside e.g. connected via usb or in the same network so you can connect via IP