Do you have feedback on our new interface?
Do you have feedback on our new interface? Let us know HERE
Frequent Visitor.. Pucca
Frequent Visitor..
143 views

data protector assigns dynamic ports greater than 65535

I would appreciate if anyone could share an experience or advise with me regarding this issue, with I am stuck.
We have set up a Data Protector environment in a new datacenter and we random and sporadically get following error causing the backups to fail:
[Critical] From: BDA-NET@computer1 "source" Time: d-m-y
Cannot connect to Media Agent on system computer2, port 83543 (IPC Cannot Connect
System error: [110] Connection timed out
) => aborting.
We have 1 Windows 2016 Cell Manager, 4 Windows 2016 Data Movers as Media Agents, and 3 Oracle Linux 6.9 Clients.
We have DP 9.09 build 115 right now.
This issue seems to be an environmental one, I think. Sometimes the connection is working, other times not.
I have tried many many things already and have opened two tickets at support.
Although it would not be necessary to set port ranges, after months, based on the request from support, I have set following port ranges on all of these servers (CM, MA-s, DA-s).
I have played around with editing host files, checking telnet and nslookup, DNS entries, local and central firewall settings, routes, static routes, routing tables, local port ranges, netstat, client security in DP, ipv4 vs ipv6, tcpdumps and Wireshark pcaps. I am stuck and tired.
We have our live SAP systems on these clients. Debugging makes the jobs running 20x longer and causing issues. I could not generate all debug logs till now what the support was asking for.
I have spent months with catching the issue and find reason and solution.
We have central firewall - Cisco ACI-s. According to the network team, port 5555 is enabled, bidirectional in the whole cell. Our separately defined OB2 port ranges are enabled as well.
But I am somehow sure that it has something with networking.
The last thing I was thinking today was, how could I catch this extremely high port 83543.
I have already asked the support how is it possible to assign not existing ports but have never received an answer.
What I think, is the following:
the client allocates a high port connecting to the media agent (MA). This port 83543 is more than 16 bit long, so the TCP header cannot handle it. A part of it will be automatically cut-off by the OS (can be Windows or Linux), the rest 16 bit (16 bits from the right side) converted to decimal is port 18007 which is in our xMA port range, but either because it is not in the dynamic allocated port range, or because the process is interrupted by this cut-off-thing - maybe the process is expecting the same port 83543 on the destination (MA) which has been picked up on the source (DA) - , the connection will be failed.
I think it is something like this:
https://www.sysadminstories.com/2013/07/tcp-ports-greater-than-65535.html
What is your experience, please?
How can I force Data Protector to pick a dynamic port only from the existing\normal\set dynamic port range?
All participants have these omnirc settings regarding the port ranges:
OB2PORTRANGE = 59300-59499
OB2PORTRANGESPEC = xMA-NET:18000-18199
We were\are using these ranges in our old DP environment, in the old data centre, with DP 8.10, so I have taken them over and have applied in the new environment as well.
Since we have these, the situation is better, now I get port errors always from an 83xxx port range. Before this, when these ranges were missing from our configuration, I got port errors from 111-112xxx ports ranges.
This is set on Windows CM and MA:
PS C:\Temp> netsh int ipv4 show dynamicport tcp
Protocol tcp Dynamic Port Range
---------------------------------
Start Port : 49152
Number of Ports : 16384
PS C:\Temp> netsh int ipv4 show dynamicport udp
Protocol udp Dynamic Port Range
---------------------------------
Start Port : 49152
Number of Ports : 16384
PS C:\Temp> netsh int ipv6 show dynamicport tcp
Protocol tcp Dynamic Port Range
---------------------------------
Start Port : 49152
Number of Ports : 16384
PS C:\Temp> netsh int ipv6 show dynamicport udp
Protocol udp Dynamic Port Range
---------------------------------
Start Port : 49152
Number of Ports : 16384
Should I extend the dynamic port range maybe?
On Windows, I see ipv4 and ipv6 enabled - was thinking of, if disabling ipv6 could help, but as that takes 16 bit TCP headers as well, I have left it as it is.
On linux we have:
cat /proc/sys/net/ipv4/ip_local_port_range
9000 65500
and ipv6 is not enabled.
Please share your experiences with me. Thank you very much!

0 Likes
2 Replies
Super Contributor.. Gamut Super Contributor..
Super Contributor..

Re: data protector assigns dynamic ports greater than 65535

Part of the answer to your question can be found in this thread:

https://community.softwaregrp.com/t5/Data-Protector-User-Discussions/DP9-IPC-Cannot-Connect-10060-on-a-port-higher-than-65535/td-p/1646093

In short: DP 9.09 uses a firewall friendly approach: only port 5555/tcp has to be open (bidrectionally).

You already had a fair hunce with your 18007/tcp, but 18007 is tunneled via 5555/tcp. So, it seems therefore wise to only check port 5555/tcp for errors. There are of course multiple ways to Rome, so I will give you my 2 cents.

If you already have debug files, check them yourselves. I often solve my own problems by analyzing the debug files carefully (I like range 1-500), using a wide range of CLI tools like grep, awk, sed, cut, perl, diff.

And since you are running DP 9.09, you might want to investigate this problem with tcpdump/wireshark and listen only for the three way tcp handshake. You don't have to capture all packets, just the beginning. How to do that: you favourite search engine will provide you with tcpdump/wireshark filter on tcp flags like SYN, SYN+ACK. If you don't see them on the receiving side while the sending side does transmit them, you are bound to have congestion or other network related problems.

One last way: since I also get spurious timeouts, I wrote restart scripting to simply retry the backups (via restart failed objects). Although definately not a solution (it is a workaround), it does lead to succeeded backups (although in 2 or 3 tries).

YMMV, as always :-/.

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: data protector assigns dynamic ports greater than 65535

Hi @Pucca,

As @Gamut mentioned all ports previously forced with omnirc options OB2PORTRANGE and OB2PORTRANGESPEC are now collapsed into the Data Protector INET port (e.g. 5555). The difference is that new tunnled approach requires that the INET port is open in both directions. If this is not the case the communication will fail and report some random high port number to fail.

Remove both omnirc options and make sure the firewall allows the INET port in both directions between the clients involved.

Regards,
Sebastian Koehler

---
Please assign a KUDO to this post, if you find it useful.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.