492 views

Max File size for Flex connector File Reader ?

Jump to solution

Hello,

Does any one know about the Max size file reader support for Flex File reader ?

I have a log file with 10GB size, does it make the connector file reader thread freeze or stuck for parsing it? 

 

Thanks.

 

Miran.

0 Likes
1 Solution

Accepted Solutions
Fleet Admiral
Fleet Admiral

Hi Miran,

I don't think it should be an issue but you have to properly configure your connector to be sure that each time you process the file, it does not restart from the beginning this is persistate and also processing the logs at the end because each time you will restart the connector, if it has to read all lines, you might have a delay in the logs reception.

But it is clear that if you could cut this file to smaller files like 1GB max, it will be better.
there are tools to do that.

The logs rotation will take also some times.
When the file is processed, you rename it or you delete it directly?

A last point, which is not negligible, if you have multiple files, the connector is able to process multiple files at the same time with multi threading thus it could be another reason to cut it.

Thanks
Regards

Michael

View solution in original post

3 Replies
Captain
Captain

Hello,

it should be tested 😉

Install connector, set 2 GB heap size, set "Cache Size" to 50 GB (not shure about it, but you can do it easily from console), point connector to this file and wait.... Simultaneously watch a tails of agent and wrapper logs.

Hope you have where to test it. If you are working directly on production server, don't forget to update your CV and LinkedIn (like me :D), if any.....

Fleet Admiral
Fleet Admiral

Hi Miran,

I don't think it should be an issue but you have to properly configure your connector to be sure that each time you process the file, it does not restart from the beginning this is persistate and also processing the logs at the end because each time you will restart the connector, if it has to read all lines, you might have a delay in the logs reception.

But it is clear that if you could cut this file to smaller files like 1GB max, it will be better.
there are tools to do that.

The logs rotation will take also some times.
When the file is processed, you rename it or you delete it directly?

A last point, which is not negligible, if you have multiple files, the connector is able to process multiple files at the same time with multi threading thus it could be another reason to cut it.

Thanks
Regards

Michael

View solution in original post

Hello  Michael,

 

Thank you for your instruction,  i will try and i will set the connector parameters to maximum. 

-When the file processed i will delete it.

 

Thanks,

Regards,

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.