Respected Contributor.. Andy___H1 Respected Contributor..
Respected Contributor..
620 views

DP 9.09 (12:3038) IPC No more available ports

Hi,

We upgraded to DP 9.09 yesterday from 9.06 on our primary site cell manager yesterday in the afternoon. Since then we've had a number of jobs, both backup jobs, but mostly object copy jobs (catalyst replications), fail with the following errors:

Cannot register TCP/IP listen socker (12:3038) IPC No more available ports in the range specified by OB2PORTRANGESPEC or OB2PORTRANGE

System error [98]  Address already in use

Now when we did this in our smaller test environment, we had similar problems, but they were only at our secondary remote replication site when we were putting data to tape. So our thinking is with 9.09 it is using more ports than 9.06 by default, using default settings of backups/object copies, which means our central cell manager has simply run out.

So we need to throttle the ports to ensure we don't run out. Our OB2PORTRANGESPEC is set to a range of 100 ports in OMNIRC on the cell manager, which can't be changed easily as it's governed by lots of firewalls!

This is how it currently works:

1) Cell manager kicks in many many host backups, all ours are Catalyst backups to Storeonce, with each host being the media server for it's own backups. We have backups running on schedules pretty much 24x7, but given the hosts are the media servers, and we normally only have approx 30 max jobs being run at any one time. 

2) Cell manager also kicks in object copy replications, but instead of using each original backup host as the replication media source/destination, it uses the central cell manager as the media server for the replications, using up yet more ports. Each replication job has no throttles on them, with each device set to no limits on "Max Number of Parallel Streams"

We think it's point two, that now running under 9.09, is messing us up. The replications are kicking in, using up all the ports as we are using it for source and destination gateway, and both replication jobs and backup jobs are running, causing one or other to fail due to lack of ports. 

So the plan to fix this temporarily, set replication device gateways to have Max Number of Parallel Streams to 4, so they don't use up many ports. Also, add in two other "media" servers in to the system so the replication load is spread across three servers, not just the cell manager, sharing the ports around. The reason why we use the cell manager as the sole replication source/destination was to take the replication load away from the original hosts that we backup, keeping it at a backup management layer only. Our thinking is by adding more media servers to act as replication source destination will solve our ports problems on our cell manager. 

Unless you tell us otherwise? Anyone else hit these problems with ports when moving over to 9.09 and beyond, especially with catalyst replications? Any other suggestions which could help us?

Thanks,

Andy

0 Likes
5 Replies
Micro Focus Expert
Micro Focus Expert

Re: DP 9.09 (12:3038) IPC No more available ports

Hello @Andy___H1

I think the best idea is to expand the range of ports. Anyway, DP 9.09 does not use this variable. Also, make sure that Inet port is not in the range. 

Regards, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a LIKE by clicking on the bottom at the left of the post and show your appreciation.
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: DP 9.09 (12:3038) IPC No more available ports

Hello

DP 9.09 use new port comunnication model and just three ports are opened.

Before DP 9.09
Data Protector used lots of ports depending on the number of integrations used in parallel. The omnirc
parameter OB2PORTRANGE(SPEC) defined a range of ports to be used by DP Services. This must be
matched with Firewall/DMZ setting of acustomer. In very large environments DP might have used up
hundreds of ports.
From DP 9.09 onwards
The maximum number of ports has been reduced to 3 making it much easier to manage and also improves
security because of much less open ports in a Firewall. Chances for mis-configuration are minimized.
5555 General Communication (Apps, MA, DA ...)
7112 DP Internal DataBase
7116 DP Application Server
This is for DP internal communication only. Be aware that some other
external communication still needs further ports. Examples:
NDMP, StorOnce Software Store, Hyper-V ... check the Admin Guide!

Therefore please remove this variable from cell and clients and try to run backups agains you will be able to close mroe than 95 ports in your firewall.

Best Regards

0 Likes
Jenni_S Outstanding Contributor.
Outstanding Contributor.

Re: DP 9.09 (12:3038) IPC No more available ports

Hi Andy,

As already stated by Andres & Jose DP9.09 changes how DP uses ports to simplify the firewall rules. Check out the Admin guide (Chapter2:Config Tasks -> FW support) for full details but basically DP will always use 5555 (5565 DP10.x) to initiate requests. The data will still flow using dynamic ports but a firewall doesn't block using other ports, just initiating the request on those ports., therefore you no longer need to restrict the range

So you can get rid of the OB2PORTRANGESPEC / OB2PORTRANGE and just make sure you have open in both directions port 5555 (+5565 if you want to future proof and intend to change the default port post DP10 upgrade) between all of your clients & Cell Manager AND between all of your clients and the Media Agent they will use AND between Media Agents that you are replicating/copying between

Also like Jose already stated some of the DP Integ / Options have other port requirements but details for those are in the admin guide in the section 2 FW support.

regards,
jenni

--------------------------------------------------------------------------------
If my post was useful, please click on KUDOS!
0 Likes
Respected Contributor.. Andy___H1 Respected Contributor..
Respected Contributor..

Re: DP 9.09 (12:3038) IPC No more available ports

Hi all,

Thanks for the help and advice, we've got a little further now, but are still struggling. We are unable to make the omnirc file changes yet as we are still mid-upgrade as not all the client hosts are upgraded yet! So we still have a mix of DP 9.09 and 9.06 hosts out there.

So for replications, we ended up moving them off the linux cell manager server, and they use two other linux boxes now, spreading the load between them. So far they are been a lot more reliable in running their processes with no "no available ports" errors. Note that the two other linux boxes do not have an .omnirc file restricting ports.

On our DP Cell we have Exchange, MSQL, Oracle, and basic filesystem backups. We run Windows and Redhat Linux. The only failures we are getting now, since we have moved the replications "off" to other servers, are the Oracle ones. The Oracle backup linux hosts are the only other hosts to have an .omnirc file set, with OB2PORTRANGESPEC hardcoded to the 100 range that our cell manager is set too. We put this on Oracle at the time back under DP 8 as this was the preferred option to use, from what we can remember.

Now a colleague of mine, whilst I was away on holiday, did end up changing the .omnirc on our 9.09 cell manager to remove OB2PORTRANGESPEC. But then all the SQL backups failed as none of the disk agents completed, Oracle ones failed as BAR backup session started but no client connected. We are assuming this was because we have/had a mismatch of 9.09 on Windows SQL clients and cell manager, and Oracle boxes all had the .omnirc set.

So to get our Oracle backups running more cleanly, and SQL ones working, I was going with this suggestion:

- Get all SQL hosts updated to 9.09

- Remove OB2PORTRANGESPEC parameter from cell manager and all Oracle boxes

- Try all backups again

Does this sound like a good plan?

Thanks, Andy

0 Likes
Jenni_S Outstanding Contributor.
Outstanding Contributor.

Re: DP 9.09 (12:3038) IPC No more available ports

Hi Andy,

Sounds like a solid plan to me. Remember that you need to open 5555/5565 between the clients and the media agent if there is a firewall in place between them.

regards,
jenni

--------------------------------------------------------------------------------
If my post was useful, please click on KUDOS!
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.