APM 9.3 establishing secure connection https for VIPs
I have a bit strange probelm that I encountered earlier and I am not quite sure what to do at this point.
I have APM 9.3 installed in a distributed environment with 3 GWs and 2 DPS as below: (Windows Server 2012 R2)
I have configured to use the load balancing with VIPs (1 User VIP and 1 Data VIP).
The certs are generated correctly and establish trust to the certificate authority correctly. In addition, the firewall is configured to open port 443 as well as website binding. Lastly, the generated certs are imported into browsers such as Firefox and IE10. (Of course, I enabled TLS support on IIS by regedit in this particular case)
When I try accessing APM through userVIP.abc.xyz , it is not working indicating that "this page can't be displayed" for IE and "secure connection failed" for Firefox. HOWEVER, when I try accesing APM through mentioned GWs above such as gw1.abc.xyz, its accessible. I can reach APM portal and everything works like a charm.
At this point, I am scratching my head why this is happening. The certs are genereated with correct SANs and establish trust to CA, website binding is configured as well as firewall setting and the certs are imported in the browers.
If anybody can provide me a guidance or hints on what's missing or any suggestation, I am more than happy to listen and try it out. I've been struggling with this problem for a quite sometimes and I am at a point where I am not so sure what else left to try.
Thanks in advance.
We are not doing any reverse proxy at this stage..
I forgot to mention I was able to access APM through userVIP.abc.xyz in http without any issue. However, accessing userVIP.abc.xyz in https is an issue.
The thing that I don't quite understand is why it's giving me an error "secure connection failed" or "page can't be displayed" when all of SANs do not have problem accessing except User VIP. User VIP itself is included in SANs in the certs.
I checked with the security team and all of SANs are correct and I also checked keystore by myself and they are all there. Am I missing something here?
These are all working fine but https://userVIP.abc.xyz isn't working as described earlier.
There are 2 certs such as 1 User VIP and 1 Data VIP.
1 User VIP is apm.abc.xyz and 1 Data VIP is apmdata.abc.xyz
User VIP there are 4 SANs such as gw1.abc.xyz, gw2.abc.xyz, gw3.abc.xyz and userVIP.abc.xyz
Data VIP there are 4 SANs such as gw1.abc.xyz, gw2.abc.xyz, gw3.abc.xyz and dataVIP.abc.xyz
Please let me know what additional actions or steps I can try to resolve this issue. (Also it would be very nice if someone can explain why this is happening and what causes it to experience this behavior).
In addition to above, a couple of more things to add comments.
Looked through APM installation guide page 85 for configuring LB, closely followed the instructions.
Looked through APM hardening guide page10 for hardening workflow, closely followed the instructions.
The information that I received from LB team, when the cert is attached to VIP in client side, it is working fine. However, when the cert is attached to VIP in server side, it is not working.
Can anybody has any ideas? At this point, I ran out of all ideas and any ideas or guidance or suggesation would be helpful.
Thanks in advance,
Here is an additional piece of information.
http://userVIP -> works
https://userVIP -> does not work
Does anyone has an idea on this issue described above threads as well as this post?
Please let me know.
Thanks in advance!
Is the 443 VIP just a pass-through to pool members? TLS actually terminates on the APM servers?
Do the data and user VIPs land on the same backend servers?
Basically, our setup has always been to use one cert for data and user VIPs, with that cert named for the user VIP DNS name, with SANs for the data VIP DNS and the name of each gateway. So our cert looks like this (with sample names):
Name: apm.corp.network.name, SANs: apmdata.corp.network.name, lmnapg01.corp.network.name, lmnapg02.corp.network.name, lmnapg03.corp.network.name, lmnapg04.corp.network.name.
The LB just passes 443/TCP through with basic NAT to the GWs. Users and data providers see the names they call ("apm" or "apmdata") and the browser is happy. Admin direct tests to each gw on https also come up green coz of the individual servers in the SAN list.
We change the default tomcat home page to have a redirect to the https URL/topaz and force TCP/80 to redirect to TCP/443. Users never get :80 to the app at all.
Scot, thanks for the info.
To answer your question:
- SSL/TLS termination is done on the F5 (LB) for this configuration (when the certs are applied to the VIP).
- The F5 routes the VIP IP to the pool memebers.
Two types of traffic will need to be load balanced for the application: user traffic and data traffic. Because there are two types of traffic, there need to be 2 types of Virtual IP addresses (VIPs) defined.
What throw me off is that only user VIP isn't working but the rest of SANs are working fine. If certs is the issue then such an event would impact all of the SANs equally, but this is not the case.
The current set up is to use one cert for data VIP and one cert for usre VIP as follow with SANs:
Cert1 SAN1 is the FQDN of the user VIP
Cert1 SAN2 is the FQDN of Gateway Server1
Cert1 SAN3 is the FQDN of Gateway Server2
Cert1 SAN4 is the FQDN of Gateway Server3
Cert2 SAN1 is the FQDN of the data VIP
Cert2 SAN2 is the FQDN of Gateway Server1
Cert2 SAN3 is the FQDN of Gateway Server2
Cert2 SAN4 is the FQDN of Gateway Server3
Again, initially I thought it's something related to the port 443 since I was able to access user VIP with port 80. I've checked all the ports/firewall setting/binding and nothing is blocking port 443.
Any thoughts and ideas would be appreciated. This issue has been going on sometimes and at this point it became a bottleneck at this stage.
Thanks in advance,