I think this is a problem with specifically how Lemmy is designed, how ActivityPub works in the browser, and how Lemmy is deployed. The issue should be entirely contained within Lemmy's nginx layer unless your second LB is doing some sort of mangling to the request (changing the request's Accept header, method, or path).

The design of Lemmy is that there is a "lemmy-ui" component that is responsible for generating HTML pages, and a "lemmy" component that is responsible for everything else. "lemmy-ui" communicates with "lemmy" to make API requests so it can build the necessary pages. Separating these things out makes a certain level of sense, and allows them to be developed independently and even replaced with alternate implementations if necessary.

Based on sniffing (Accept header) the request needs to either route to "lemmy" or "lemmy-ui". The path and method based requests are fairly easy to handle as there isn't ambiguity, if it is for a child of a certain path or it is a certain method (e.g POST), it always goes to "lemmy". Things are more complex for GET requests for a comment/post because users will load that URL in their browser expecting HTML (which "lemmy-ui" is responsible for). Bots/federation/API clients will request that same URL with basically only the Accept header being different and expect JSON-LD (which "lemmy" is responsible for).

The problem comes when you deploy such a system, you need some way to conditionally route certain requests to one service and certain requests to another. Lemmy has chosen to use nginx with manually maintained nginx.conf files.

The options as I see it are:

1. Build a system to generate the right nginx configuration when the proxy container starts. Something like it taking optional environment variables to define the hostname/port of the "lemmy" and "lemmy-ui, and it spits out basically what is in the repo right now but customized for you. If you need something more custom you're on your own with that still being able to serve as a starting point ,but you probably not even using that container.
2. Remove nginx entirely and set up "lemmy" to forward requests it thinks are by web browsers to a URL for the UI you can specify. (This may be problematic as it technically makes "lemmy-ui" and "lemmy" co-dependent, though whether "lemmy-ui" should expect to communicate with "lemmy" over anything but your instance's public address is another issue entirely...)
3. Remove nginx entirely and set up "lemmy-ui" to forward requests that look like backend requests to "lemmy".

I like option 2 best, as "lemmy" does something similar for "pictrs" already, though it is a little counterintuitive for what I consider to be the "backend" to forward requests to the "frontend".