Logger config backup is very big, it is running to long

I managed few loggers appliances in my life, and config backups were usually small, few megabytes. Few times I saw some 2-3GB config backup. That was also running fine.

But now I have logger which generate config backup so big that it filled whole logger filesystem. When you trigger config backup it start generate /opt/arcsight/logger/tmp/configs/configs.tar and this had around 330GB (this is size before compression) when it fill whole filesystem. Backup job clearly failed.  I made workaround, just mounted there NFS to /opt/arcsight/logger/tmp/configs mountpoint. Now configs are generated, but it takes too long.~8hours.

I also try to run it manually as "/opt/arcsight/logger/bin/arcsight configbackup config", there was no error, however I noticed that biggest part of config.tar are files backup-ed from location:

/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636473][632][1659515736232].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636474][627][1659515736442].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636475][642][1659515737201].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636476][631][1659515737533].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636477][640][1659515738048].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636478][644][1659515738333].ser
/opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316/Uncompressed[1561105012316][6636479][629][1659515738695].ser
...

there are thousands of them.

Any idea what are these files? Why there is so many of them

  • Suggested Answer

    I believe these are uncompressed chunks stored before written to postgres compressed or where uncompressed chunks go when they fail to compress. Might be a concern having so many. One of my appliances I looked at, only had one of these files. 

  • hmm good hint. there is almost 20000 of these filese since jul 30....
    # ll  /opt/arcsight/userdata/logger/user/logger/cache/receivers/1579870794187  |wc -l
    19114
    # ll /opt/arcsight/userdata/logger/user/logger/cache/receivers/1579870794187  | head
    total 19724616
    -rw-r--r--. 1 arcsight arcsight 1054899 Jul 30 08:50 Uncompressed[1579870794187][516350][50][1659163844203].ser
    -rw-r--r--. 1 arcsight arcsight 1054676 Jul 30 08:51 Uncompressed[1579870794187][516351][14][1659163904188].ser
    -rw-r--r--. 1 arcsight arcsight 1054676 Jul 30 08:52 Uncompressed[1579870794187][516352][16][1659163964189].ser

    Hmm, but that to do....maybe restart

  • Check logs for any errors inserting into sql before restarting to see what might be going on. Is this an appliance or software? 

  • It is appliance L7600. we want to migrate it to software version to be supported.

    I think i will not restart before weekend.

  • turn out, that restarting services deletes the cache, this was in logs:


    10:13:05.655 [main] INFO  default.com.arcsight.logger.LoggerReceivers@1b1473ab][recoverFromLastShutDown - Uncompressed[1607699735724][8715890][680][1659008184417].ser is expired. No recover on this file! Deleted!
    10:13:05.655 [main] INFO  default.com.arcsight.logger.LoggerReceivers@1b1473ab][recoverFromLastShutDown - Uncompressed[1607699735724][8715891][681][1659008186114].ser is expired. No recover on this file! Deleted!
    10:13:05.655 [main] INFO  default.com.arcsight.logger.LoggerReceivers@1b1473ab][recoverFromLastShutDown - Uncompressed[1607699735724][8715892][620][1659008188531].ser is expired. No recover on this file! Deleted!

    Now loggers is working fine now (cache files are not being created), but we lost the data which was cached  around  1TB of events :(

    We have exactly same behavior on another logger, not sure what to do.

  • can you check content of those files?

    Is it in CEF format?

    Each row in my *.ser files looks similar to CEF, it is starting like:

    ▒ECEF:1|Unix|Unix||arcsight:0:165|kernel event|Low|

    So it looks like CEF, only instead of date there is that wierd character ▒, maybe that could be reason why it is not processed?

  • Mine looks similar to what you describe. 

  • Are you files processed? I noticed on some loggers that those file are there but  there is not lot of them, but they have old date. Events in these files were clearly not processed by logger, just sits there in cache.

    To my original issue, i noticed that restart of receiver, helps a little, events which being send after restart are processed, although those  which were already cached, these are still unprocessed.

    Also i noticed that number in filepath /opt/arcsight/userdata/logger/user/logger/cache/receivers/1561105012316 refers to reciever, you can see that number in webui in receiver list it is part of url if you click on particular receiver.