25
[solved] Randomly getting ECH errors on self-hosted services. (lemmy.ca)
submitted 3 months ago* (last edited 3 months ago) by Darkassassin07@lemmy.ca to c/selfhosted@lemmy.world
16 comments fedilink hide all child comments

In the last couple of weeks, I've started getting this error ~1/5 times when I try to open one of my own locally hosted services.

I've never used ECH, and have always explicitly restricted nginx to TLS1.2 which doesn't support it. Why am I suddenly getting this, why is it randomly erroring, then working just fine again 2min later, and how can I prevent it altogether? Is anyone else experiencing this?

I'm primarily noticing it with Ombi. I'm also mainly using Chrome Android for this. But, checking just now; DuckDuckGo loads the page just fine everytime, and Firefox is flat out refusing to load it at all.

Firefox refuses to show the cert it claims is invalid, and 'accept and continue' just re-loads this error page. Chrome will show the cert; and it's the correct, valid cert from LE.

There's 20+ services going through the same nginx proxy, all using the same wildcard cert and identical ssl configurations; but Ombi is the only one suddenly giving me this issue regularly.

The vast majority of my services are accessed via lan/vpn; I don't need or want ECH, though I'd like to keep a basic https setup at least.

Solution: replace local A/AAAA records with a CNAME record pointing to a local only domain with its own local A/AAAA records. See below comments for clarification.

top 16 comments
sorted by: hot top controversial new old
[-] bobslaede@feddit.dk 8 points 3 months ago

Any chance you are both accessing your services locally with a local DNS, and publicly with something like Cloudflare?

[-] Darkassassin07@lemmy.ca 4 points 3 months ago* (last edited 3 months ago)

I do have external acces to Ombi via cloudflare; but the device I'm seeing this problem on is permanently connected to a VPN hosted from the same server machine as ombi/nginx with 'block all connections without VPN' enabled. And this testing has been done from within the same LAN.

It should never see/reach cloudflare for this service.

/edit; I've also disabled 'use secure DNS' in chrome. I host a local DNS within that lan/vpn network.

[-] bobslaede@feddit.dk 5 points 3 months ago* (last edited 3 months ago)

Try with nslookup and see if you're resolving the domain to both your local ipv4 address, and the Cloudflare ipv6 at the same time. I am using pihole for my local DNS, and it would give me both my local address, and also the Cloudflare ipv6 address.

Edit
My pihole will ask upstream even if the domain was listed locally. It doesn't ask Upstream for cname.

[-] Darkassassin07@lemmy.ca 5 points 3 months ago

Crap, looks like that's exactly what it is.

Now how to fix that...

[-] Darkassassin07@lemmy.ca 3 points 3 months ago

Added an AAAA record to pihole:

ombi.mydomain.example 0000:0000::0000:0000

Now nslookup returns the correct ipv4 address, and '::' as the ipv6.

We'll see if that works.

[-] Darkassassin07@lemmy.ca 2 points 3 months ago* (last edited 3 months ago)

That unfortunately did not work. I am only getting the ipv4 address now, but I still get the same ECH error in chrome 1/5 tries.

Firefox now changed errors from 'invalid certificate' to 'connection is insecure but this site has HSTS' (true). Still wont show the cert or provide any further info. (forgot to grab a screenshot before the below 'solution')

I'm really annoyed at this point and have just disabled cloudflare proxying for this service. That seems to have sorted it for all browsers. I may look further later, I may just say fuck it and leave it like this. Gotta walk away for a bit.

[-] bobslaede@feddit.dk 3 points 3 months ago

You should change to use cname in pihole. I will write up on my computer later for you.

[-] solrize@lemmy.world 2 points 3 months ago

Can you verify with wireshark that the traffic is only going through your lan? I'm not hip enough for nginx but I used to have to run apache under gdb all the time to trace random errors from the server. That would be next, if the traffic is really local.

[-] Darkassassin07@lemmy.ca 1 points 3 months ago

I'll look into that next if what I've done doesn't work. (see other comments)

[-] sk@hub.utsukta.org 2 points 3 months ago

this issue is an ongoing discussin over at NPM too, very mysterious

https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3982

[-] Darkassassin07@lemmy.ca 3 points 3 months ago* (last edited 3 months ago)

Thanks. That seems to be a similar, but slightly different error. I think the below may apply though.

I believe I've tracked down more of my issue, but fixing it is going to be a hassle:

When cloudflare proxying is enabled, there are 3 DNS records involved; A record with cloudflares ipv4, AAAA record with cloudflares IPV6, and the key to this puzzle: an HTTPS record with cloudflares ech/https config.

With pihole I can set DNS records for A/AAAA, but I have no way of blocking/setting the HTTPS record so it gets through from cloudflare.

The LAN A/AAAA records don't match the HTTPS record from cloudflare, so browsers freak out.

Once I disabled cloudflares proxying, I no longer get HTTPS records returned and all works as intended.

I'll either have to keep cloudflare proxying disabled, or switch pihole out for a more comprehensive DNS solution so I can set/block HTTPS records :(

Thank you @bobslaede@feddit.dk for pointing me in the right direction.

[-] bobslaede@feddit.dk 4 points 3 months ago* (last edited 3 months ago)

I've fixed the same issue for me.
Originally I had this in my Local DNS settings in my Pi-Hole:

- service1.domain	10.0.0.4
- service2.domain	10.0.0.4
- service3.domain	10.0.0.5

I changed that to this:

- host1.domain		10.0.0.4
- host2.domain		10.0.0.4

And then I added CNAME Records to the services like this:

- service1.domain	host1.domain
- service2.domain	host1.domain
- service3.domain	host2.domain

This fixed the whole thing for me :)

Edit: Gonna add some more info

The trick that makes this work, and probably will for you too, and allow you to keep your HTTPS queries, is that Pi Hole will just not ask upstream, if it has the DNS name in the CNAME records. Those CNAME records will have to point to a domain, that Cloudflare doesn't know about. That way there is no other records upstream that will confuse the DNS server and your browser.
The hostname you have in your local DNS records that your CNAME points to, will be something only known locally for you.

[-] Darkassassin07@lemmy.ca 2 points 3 months ago

@bobslaede@feddit.dk I could kiss you. You've been invaluable my friend, thank you!

Just gave this a test: CNAME ombi.domain -> local.domain with cloudflares proxy re-enabled.

Now the HTTPS, A, and AAAA requests all receive the CNAME response and browsers are happy. I didn't even have to modify ngnix to recognize local.domain like I thought I might.

[-] bobslaede@feddit.dk 2 points 3 months ago

Awesome! I'm glad that it worked. It took me a while to figure out, when it happened to me. Glad that I could make your life easier :)

[-] Moonrise2473@feddit.it 2 points 3 months ago

I had a similar problem when using nginx proxy manager. In the end I just gave up and directly used cloudflare tunnels+cloudflare ssl

[-] Darkassassin07@lemmy.ca 3 points 3 months ago

I think I've found the problem:

It seems my issue is pihole being unable to block/modify dns requests for HTTPS records, which don't match the LAN IPs pihole handed out in A/AAAA records.

I've disabled cloudflare proxying so they don't have HTTPS records to serve, but I'll have to replace pihole with a better lan DNS solution if I want to turn that back on.

this post was submitted on 28 Sep 2024
25 points (100.0% liked)

Selfhosted

40716 readers
404 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz