Hi Rich,

A possible root cause is that some LB and web servers will drop headers with underscores in them. There's some obscure technical reason behind it (some CGI compatibility problems it seems).

You can either change your LB configuration to stop it from dropping headers with underscores (if that's an option), or you can upgrade to PPM 24.1 where we've renoved underscores from our CSRF_X_TOKEN header.

No idea why it would only happen with HTTPS and not HTTP though, so it's worth looking further in the LS configuration.

It's actually not dropping the token from the request header, but from the payload. 

We reviewed the network request in the chrome browser developer tool. That's where we saw the payload missing a value for the token, but the same token and correct value is in the header. This makes me wonder are we seeing the network request as it was formed in the browser or as it's returned to the browser from the server. This is a reason to be a little bit skeptical it's the LB. 

We also did a test with one node configured for direct access to https://server_name:port. Using the HTTPS_PORT and could not experience the problem. So that makes a case it's the LB. For comparison sake, when using the LB we also only had one node up.

One other test we did was to repeatedly send the same network request using curl so that we could be positive the network request started out intact. We never saw the LB strip these requests. 

Summary:

403 error only when EXTERNAL_WEB_PROTOCOL=HTTPS and network request sent from PPM through LB. No error when protocol is HTTP.

No error when  using direct access HTTPS or HTTP.

No error when EXTERNAL_WEB_PROTOCOL=HTTPS and network request sent using curl through LB.

If the problem only occurs with HTTPS, then maybe browser tries to load some resource from HTTP, which will be blocked by the browser on an HTTPS web site. 

Is there anything suspicious in the Web Browser console, or in the previous HTTP(S) requests leading up to loading the ACL popup?

As you wrote before, it works fine when using directly PPM HTTPS, without going though LB, thus my assumption that there's a problem in the LB config.
BASE_URL is the URL of LB, right?

Yes, the BASE_URL is correct for LB scenario. I'm about 90% sure it's an LB issue.

At this point you may want to open a support ticket/incident to get R&D involved in understanding what happens exactly and what gets dropped and when. It'll be hard to get more pointers here without access to the system.