Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..
1857 views

Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer - CoFC

I am backing up my exchange server over Fiber Channel. I get the following error every so often (once a week or so)

[61:3003] Lost connection to OB2BAR Backup DA named "ERROR"
on host mail.domain.com.
Ipc subsystem reports: "IPC Read Error
System error: [10054] Connection reset by peer

This ticket has been open for weeks now. I put some additional lines of code in the option file as per the techs dirction.

Are there HBA settings required for using Fiber Channel to backup? Any ideas?

0 Likes
23 Replies
Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hi @T. Cooper,

Well, with this amount of information I won't be able to give you any good advise. In general CoFC is just working our of the box. For Windows increase the login count in the Fibre Channel tab on the StoreOnce UI. Please check my posts...

https://community.softwaregrp.com/t5/Data-Protector-Practitioners/Data-Protector-9-08-Configuring-HP-StoreOnce-5100-Catalyst-to/m-p/1640331.

https://community.softwaregrp.com/t5/Data-Protector-Practitioners/DP-9-09-amp-storeonce-6500-catalyst-store-over-FC/m-p/1634923

Please share information on used versions of Data Protector, Exchange, StoreOnce as well as the configuration (which servers see the CoFC dievces, how many streams are backed up in parallel, is this a Source-side backup, is the Exchange server running out of memory during backup...)

It seems you suffer from crashing processes that can be caused by a problematic Data Protector agent, resource shortage or communication issues. Have you tried to send this backup via the network (StoreOnce Catalyst over IP) instead?

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Data Protector is version 10.02

Exchange is 2013

Storeonce is at 3.16.5-1746.4

I have alread increased Number of Devices from 4 to 8 per port.

 

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hello @T. Cooper,

Any progress on this issue? Have you checked debug_d2d.log on the Media Agent? If you need more details, here is a summary of how to enable CoFC debugging in Data Protector.

MF_Universe2018_skoehler_v3.png

An agent crash will usually be logged to the Windows Eventlogs or dmesg on Linux. Crash dumps on Windows are created in C:\ProgramData\Microsoft\Windows\WER\ReportQueue.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

No progress. It usually works after a -clear and -free_cell_resources. ill take a look at those logs. Ive made a couple tickets about this issue, but apparently Microfocus techs dont look at those logs either as the ticket can be ope for days or weeks and they dont remote in or do any real tech work. 

0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hi @T. Cooper,

Can you try the following omnirc settings on the Exchange client? Make sure there is no vssbar.exe and e2010_bar.exe running on the Exchange client after backups have been completed/aborted.

OB2IPCNOSENDRECVLOOP=1
OB2SHMIPC=0
OB2INETTIMEOUT=10
OB2_DNSTIMEOUT=2
OB2_IPC_PROTOCOLINFOTMEOUT=60

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
Tags (1)
0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Did all that. Added it the file (forget the name, the one with no extension).

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hi T.Cooper,

How long does it take to fail with the error message?

This appear to be a known issue in 10.02 using secure communication. So as a workaround can you add an exception for the Exchange nodes and DAG hostname..

omnicc -secure_comm -configure_exception <hostname>

Regards

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hello T. Cooper ,

For this one, you've mentioned "This ticket has been open for weeks now. ". I would assume you mean that you have it logged with DP Support.

Can you share to me the case ID and let me check. I may also alert the case owner who is working with you on this one.

The error you've mentioned could be due to a lot of things and if you have a open support case, I would expect that all related logs have been checked.

Rgds,

Sherman

DP Support

 

0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

 

I emailed the case owner. It has been reopened once before. The error is intermittent, typially once a week or so, but it can be as long as two week. I dont feel conformtable sharing it online.

I think I may be expecting too much from DP.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

Hello  T. Cooper,

First comment - if this is very much intermittent, then this is something more to be external from DP.

But again, since I don't have details, then I can't make the above statement conclusive. It is best to investigate this further.

My email address is sherman.senin@microfocus.com. You can forward to me that email you've sent to the case owner and if I have the chance to review the case notes + files/logs collected from you, I can provide my technical analysis + comments and recommendations.

Hope this helps

Rgds,

Sherman

DP Support

0 Likes
antaln Outstanding Contributor.
Outstanding Contributor.

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

If this is reported by BSM, then means it has lost connection to either Exchange agent or the VSS agent (which Exchange agent delegates most heavy lifting to) - this can be interpreted from "OB2BAR Backup DA" part of the error message. This means the connection has performed a suitable handshake (and is thus unlikely to be caused by secure communication settings.

The 'named "ERRROR"' part of the message means that this connection has no association with any object in BSM - at any time, integration agents may have multiple connections to BSM (e.g. to monitor for abort, or to send progress messages) and not all connections will be associated with objects being backed up.

The 'connection reset by peer' means a TCP reset was received - this could mean the peer process crashed (but that should make a little more noise in the session), or it could mean that something external dropped the connection (router/firewall dropping TCP connections due to idleness timeout, DPI firewall tripping on a payload, or transient network drop that lasts longer than the TCP max retransmission period).

Are there any other errors in the session? If not, then it's likely either connection to VSS's abort thread or to barscript (initial Exchange agent started by BSM). These connections are not associated with any object, so they fit the error message. And these connections tend to be idle most of the time - the VSS abort thread is listening for abort notification from BSM (so it doesn't send anything by default), and barscript is sending outputs from the Exchange agent (e.g. session messages from E2010_BAR, which may be sent rarely if the backup takes long). Unfortunately, none of these are subject to reconnects ('reconnect broken connections' session option). However, TCP keepalives are on by default since 9.06, which should prevent connection to be dropped due to idleness - defaults are 10 minutes for both idle time and the interval between probes. When was the last message from E2010_BAR reported before this error was printed? If it was less than 10 minutes, you may need to decrease keepalive time and interval to a lower value (OB2IPCKEEPALIVETIME and OB2IPCKEEPALIVEINTERVAL omnirc variables).

0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

I added these values to omnirc per the tech.

OB2INETTIMEOUT=60
OB2SHMIPC=0
OB2IPCKEEPALIVE=1
OB2IPKEEPALIVETIME=900
OB2IPCKEEPALIVEINTERVAL=60
OB2RECONNECT_RETRY=1
OB2IPCSECGRACEPERIOD=120

0 Likes
antaln Outstanding Contributor.
Outstanding Contributor.

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

OB2IPCKEEPALIVETIME=900 is actually higher than the default value (600). If there is something dropping connections that are idle longer than the default time (10 min), then increasing this value is unlikely to prevent that mechanism. You could try reducing it to 180 or 120.

0 Likes
Respected Contributor.. T. Cooper Respected Contributor..
Respected Contributor..

Re: Ipc subsystem reports: "IPC Read Error and System error: [10054] Connection reset by peer -

That is what the microfocus tech game me. I will ask him about it.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.