Every 12-18 hours, internet stops working with Pihole running - DNS error

I'm trying to figure out what's going on. Every 12-18 hours, all the devices on the network start to fail with DNS errors (except video streaming for some weird reason). I'm able to log into pi-hole admin just fine, and everything "seems" fine.

I restart the routers, modem and then get it working again. The DNS server IP doesn't change so its not getting reset or anything.

I tried "disabling" the pi-hole in config, but the same issues happen. The only way I can solve permanently is to stop using the pi-hole DNS.

Pi-Hole running on Raspberry pi zero w, with Nest Wifi router and fiber optic altafiber modem.