Software: First and foremost: must be unix-like, must be able to communicate in both ways with an open-wrt router firmware distro and the devices on the local network (android, windows, linux, ipadOS systems). Must be very secure, like enterprise-grade or almost like that. Must be free and open-source. Must be somewhat fault-tolerant (so no Arch or gentoo or anything like that, i don’t feel like recompiling the server’s system daily). Must have these in base repos or easily installed in other methods: secure ssh client (like openSSH or such caliber), a software that enables me to securely control and see the gui of the server from android (Rustdesk? or such), (optionally i2p, dnscrypt, vpn clients, not needed if the router has them, just in case of emergency), ip camera management software, high-security intrusion-detection system, https server with css and js support (preferably command-line). Window manager: must support a very easy to use and lightweight tiling window manager (like i3wm) or if not, its installation and configuration needs to be possible and documented.
Hardware: affordable, x86_64 architecture, should be able to handle all of these at the same time, without freezing or overheating (i live in Hungary, so should be able to handle up to 40°C air temperature with stock fans or there should be space for more fans. liquid cooling is no-go).
I have considered these operating systems. Are any of these bad ideas? What you recommend that is not here?
AlmaLinux Alpine Linux Ubuntu Server Rhino Linux (unofficial ubuntu rolling) Debian Testing Void Linux FreeBSD
In that case, and if you do need a gpu (such as jellyfin, Plex or another reason) look at the GPU transcoding link in my previous comment. You can flash Nvidia consumer cards or price compare with Intel A series GPUs. This means a bit of tinkering but if you need transcoding and cheap Quadros aren’t available to you, it’s an option.
You can always go for a used PC with integrated graphics like Intel and see if that works for your use case. Follow the recommendation for a big case with lots of space, look at any of the dell Optiplex or similar office PCs. If you have specific applications Google them + minimum or recommended requirements. An SSD as a boot drive is absolutely worth it over an HDD.
Your camera setup probably won’t need an external graphics card but if it does you can always upgrade later.