Is there any way I can use a proxy on Android on a per app basis instead of system wide? Just like for some who don't want absolutely everything to be routed through a VPN, some VPNs offer the option to include/exclude applications. That's exactly what I'd like to do, just with a proxy on my phone.
Multiple VPN apps on Android allows you to set per-app rules including forwarding to different VPN servers
The Cloudflare app allows setting exclusions
However you can't have 2+ active VPN connections simultaneously on Android without root, so while for example OpenVPN allows you to set multiple VPN profiles with different app exclusions (binding chosen apps to a certain VPN connection) you're forced to pair this with a firewall to keep different groups of apps offline while their VPN profile is inactive, then switch which apps are online by switching VPN profile.
Alternatively - set up a single Wireguard VPN to your own server somewhere (it may be a rented VPS) and then set multiple outgoing VPN connections on that server, and then set forwarding rules based on which domains/IP each app communicates with (beware that this may make a mess with browsers and such if for example a single web page gets split over different VPN sessions due to content being hosted across different servers on the same page)