After reviewing the entire thread, I have to say that this is quite an interesting question. In a departure from most other people’s threat models, your LAN is not considered trusted. In addition, you’re seeking a solution that minimizes subscription costs, yet you already have a VPN provider, one which has a – IMO, illogical – paid tier to allow LAN access. In my book, paying more money for a basic feature is akin to hostage-taking. But I digress.
The hard requirement to avoid self-signed certificates is understandable, although I would be of the opinion that Jellyfin clients that use pinned root certificates are faulty, if they do not have an option to manage those pinned certificates to add a new one. Such certificate pinning only makes sense when the client knows that it would only connect to a known, finite list of domains, and thus is out-of-place for Jellyfin, as it might have to connect to new servers in future. For the most part, the OS root certificates can generally be relied upon, unless even the OS is not trusted.
A domain name is highly advised, even for internal use, as you can always issue subdomains for different logical network groupings. Or maybe even ask a friend for a subdomain delegation off of their domain. As you’ve found, without a domain, TLS certificates can’t be issued and that closes off the easy way to enable HTTPS for use on your untrusted LAN.
But supposing you absolutely do not want to tack on additional costs, then the only solution I see that remains is to set up a private VPN network, one which only connects your trusted devices. This would be secure when on your untrusted LAN, but would be unavailable when awat from home. So when you’re out and about, you might still need a commercial VPN provider. What I wouldn’t recommend is to nest your private VPN inside of the commercial VPN; the performance is likely abysmal.
I previously proffered some information in the first thread.
But there’s something I wish to clarify about self-signed certificates, for the benefit of everyone. Irrespective of whichever certificate store that an app uses – either its own or the one maintained by the OS – the CA Browser Forum, which maintains the standards for public certificates, prohibits issuance of TLS certificates for reserved IPv4 or IPv6 addresses. See Section 4.2.2.
This is because those addresses will resolve to different machines on different networks. Whereas a certificate for a global-scope IP address is fine because it should resolve to the same destination. If certificate authorities won’t issue certs for private IP addresses, there’s a good chance that apps won’t tolerate such certs either. Nor should they, for precisely the reason given above.
A proper self-signed cert – either for a domain name or a global-scope IP address – does not create any MITM issues as long as the certificate was manually confirmed the first time and added to the trust store, either in-app or in the OS. Thereafter, only a bona fide MITM attack would raise an alarm, the same as if a MITM attacker tries to impersonate any other domain name. SSH is the most similar, where trust-on-first-connection is the norm, not the outlier.
There are safe ways to use self-signed certificate. People should not discard that option so wontonly.