but without nix it’s a pita to maintain through restores/rebuilds.

No it isn’t. You can even define those routing polices in your systemd network unit alongside the network interface config and it will manage it all for you.

If you aren’t comfortable with systemd, you can also use simple “ip” and “route” commands to accomplish that, add everything to a startup script and done.

major benefit to using a contained VPN or gluetun is that you can be selective on what apps use the VPN.

Systemd can do that for you as well, you can tell that a certain service only has access to the wg network interface while others can use eth0 or wtv.

More classic ip/route can also be used for that, you can create a routing table for programs that you want to force to be on the VPN and other for the ones you want to use your LAN directly. Set those to bind to the respective interface and the routing tables will take place and send the traffic to the right place.

You’re using docker or similar, to make things simpler you can also create a network bridge for containers that you want to restrict to the VPN and another for everything else. Then you set the container to use one or the other bridge.

There are multiple ways to get this done, throwing more containers, like gluetun and dragging xyz dependencies and opinionated configurations from somewhere isn’t the only one, nor the most performant for sure. Linux is designed to handle this cases.

In terms of homelab stuff, I know a lot of people appreciate the containerized approach.

What I said applies to containerized setups as well. Same logic, just managed in a slightly different way.

By “set up wireguard to route through the VPS” you mean having wireguard forward a port from the VPS to a port on the homeserver at its wireguard IP address?

Yes, he means that.

qBittorrent will still need to publish the right IP address to peers though, right? So I will need to configure the proxy VPS’s IP address in qBittorrent…

No. For most things qBittorrent does public IP detection. For the rest your VPS will be doing NAT between the WG interface and the public internet. This means your qBittorrent client sends outgoing packets with the source address of your WG private IP and then the VPS will change those to it’s public IP address.

The thing you must be careful about is that you need to restrict qBittorrent to only send and receive traffic on the WG interface, otherwise it will be using both. You can do it in the settings, but the safest way is to do it at the container setup or systemd service level and completely hide any interface that isn’t the WG one from it.

You can force all outgoing traffic to use the VPN interface via iptables/routes (meaning if it doesn’t exist or doesn’t work nothing will be able to access the internet) OR use systemd globally hide the non-VPN network interface from all services except for the VPN client.

All of that can be achieved with simple systemd or iptables/routes tweaks. You can force all outgoing traffic to use the VPN interface via routes (meaning if it doesn’t exist or doesn’t work nothing will be able to access the internet) OR use systemd globally hide the non-VPN network interface from all software except for the VPN client.