Large number of Quick Finder pending jobs

Hi,

I've been having a problem with a user losing missing some of his folders, and stumbled upon another problem with Quick Finder pending jobs.

Running a GWCheck Analyze/Fix Databases, I have a number of users on this post office with lines such as :

STRUCTURAL VERIFICATION of database /var/opt/novell/groupwise/mail/tek/ofuser/userydz.db
Verifying data
- Database is structurally consistent
- Space that could be recovered: 0 bytes
Quickfinder: Pending jobs = 2009538 Files = 539caf3d.idx, 58d81e29.inc

This is one of the worst cases, but over half of the users have at least 1000 pending jobs.

I've following the steps in https://www.novell.com/support/kb/doc.php?id=7012997, which did solve a couple of users. However, the above user resulted in this in the post office log :

15:51:14 EECD Starting QuickFinder index compression
15:51:15 EECD Scanning Post Office for Orphaned Files
15:51:18 EECD Finished Post Office Scan for Orphaned Files
15:51:22 EECD Compressing QuickFinder index: userydz.db (2009538)
15:51:39 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:10 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:49 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:49 EECD The database function 87 reported error [F03E] on userydz.db
15:52:49 EECD Error updating QuickFinder indexes: [F03E]
15:52:49 EECD Updating QuickFinder index: userydz.db (2009538)
15:53:02 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:53:02 EECD The database function 87 reported error [F03E] on userydz.db
15:53:02 EECD Error updating QuickFinder indexes: [F03E]
15:53:04 EECD QuickFinder indexing thread finished

The server has 6GB of RAM, so not huge, but doesn't seem to have any memory issues - swap is barely used for example.

Groupwise system is 14.2.2, recently upgraded from 2012 SP4. All servers are OES2015 SP1

Any help would be appreciated,
Thanks
Kevin

  • Hi.

    Some basics first:

    How is your quickfinder configured, e.g what's the start time, and how
    often does it run?

    Second, that Mailbox you're having issues with is *ENORMOUS*. 2 Million
    unindexed objects (aka mails, appointments, you name it) is a league of
    it's own. What's up with that?

    Third, to fix it, after you have set your quickfinder to run often
    enough and index enough items per run, do a structural rebuild on that
    user database. That will nuke the current index for the mailbox annd
    start over.

    Set the post office logs to verbose, and check it for a while if the
    number of items left to index decreases (and no errors are shown) when
    QF runs.



    Am 15.05.2017 um 17:10 schrieb Kevin White:
    > Hi,
    >
    > I've been having a problem with a user losing missing some of his
    > folders, and stumbled upon another problem with Quick Finder pending jobs.
    >
    > Running a GWCheck Analyze/Fix Databases, I have a number of users on
    > this post office with lines such as :
    >
    > STRUCTURAL VERIFICATION of database
    > /var/opt/novell/groupwise/mail/tek/ofuser/userydz.db
    > Verifying data
    > - Database is structurally consistent
    > - Space that could be recovered: 0 bytes
    > Quickfinder: Pending jobs = 2009538 Files = 539caf3d.idx,
    > 58d81e29.inc
    >
    > This is one of the worst cases, but over half of the users have at least
    > 1000 pending jobs.
    >
    > I've following the steps in
    > https://www.novell.com/support/kb/doc.php?id=7012997, which did solve a
    > couple of users. However, the above user resulted in this in the post
    > office log :
    >
    > 15:51:14 EECD Starting QuickFinder index compression
    > 15:51:15 EECD Scanning Post Office for Orphaned Files
    > 15:51:18 EECD Finished Post Office Scan for Orphaned Files
    > 15:51:22 EECD Compressing QuickFinder index: userydz.db (2009538)
    > 15:51:39 EECD Error: Memory Allocation error [F03E] in
    > Squeeze.QFSqzIndex.2 ()
    > 15:52:10 EECD Error: Memory Allocation error [F03E] in
    > Squeeze.QFSqzIndex.2 ()
    > 15:52:49 EECD Error: Memory Allocation error [F03E] in
    > Squeeze.QFSqzIndex.2 ()
    > 15:52:49 EECD The database function 87 reported error [F03E] on userydz.db
    > 15:52:49 EECD Error updating QuickFinder indexes: [F03E]
    > 15:52:49 EECD Updating QuickFinder index: userydz.db (2009538)
    > 15:53:02 EECD Error: Memory Allocation error [F03E] in
    > Squeeze.QFSqzIndex.2 ()
    > 15:53:02 EECD The database function 87 reported error [F03E] on userydz.db
    > 15:53:02 EECD Error updating QuickFinder indexes: [F03E]
    > 15:53:04 EECD QuickFinder indexing thread finished
    >
    > The server has 6GB of RAM, so not huge, but doesn't seem to have any
    > memory issues - swap is barely used for example.
    >
    > Groupwise system is 14.2.2, recently upgraded from 2012 SP4. All servers
    > are OES2015 SP1
    >
    > Any help would be appreciated,
    > Thanks
    > Kevin
    >



    --
    Massimo Rosen
    Micro Focus Knowledge Partner
    No emails please!
    http://www.cfc-it.de
  • Hi Kevin,

    Your Quick Finder Indexing is horribly behind in indexing your user databases. The "Squeeze" error is very indicative of this as is the number of un-indexed items per mail box.

    To reindex this entire Post Office in one go is going to cripple your system. The TID you followed is a good start but I'm going to recommend some more information as to what steps to take on the users so incredibly out of date. After we have dealt with such users we are going to have to fix indexing on your entire Post Office in order to get the entire system up to date, and then to ensure QF Indexing is configured is such a way that you never get to this situation again.

    So starting with individual users:

    Go to the Web Console of your post office.
    Login as a user with full rights to the post office or your admin user.
    Go to the Configuration tab.
    Find Log Settings and click on that to open the configuration page for logging.
    Select Diagnostic as the log level.
    Click on Submit.
    Without Logging being in Diagnostic level several of the next options regarding quick finder indexing may not be visible.
    Return to the Configuration page.
    Find Quick Finder Indexing and click on that link.
    Under Indexing Options do the following:

    - Set the Indexing Level to Unlimited
    - Then there are four tick boxes. Three of them should already be ticked, leave them ticked.
    - Put a tick in the fourth box to delete the old qfinder files
    - then place the user FID into both the Beginning and End box.

    Click Submit.

    Considering that there are more than 2 million messages for this user alone requiring indexing this may take a very long time i.e. several hours. Only do one user at a time and let it complete before attempting another one.

    Try that, let us know if this user database is indexed. If you still get "Squeeze" errors then further intervention will be required.

    Looking forward to hearing back from you.

    Cheers,
  • Laura,

    Am 16.05.2017 um 07:54 schrieb laurabuckley:
    >
    > Go to the Configuration tab.
    > Find Log Settings and click on that to open the configuration page for
    > logging.
    > Select Diagnostic as the log level.
    > Click on Submit.
    > Without Logging being in Diagnostic level several of the next options
    > regarding quick finder indexing may not be visible.


    Diagnostics is overkill here. Verbose will do.

    > - Set the Indexing Level to Unlimited
    > - Then there are four tick boxes. Three of them should already be
    > ticked, leave them ticked.
    > - *Put a tick in the fourth box to delete the old qfinder files*
    > - then place the user FID into both the Beginning and End box.
    >
    > Click Submit.


    With his error, that will unfortunately not change a thing. Been there,
    done that. The only way to get QF going again for that mailbox is to do
    a structural rebuild (not because of the rebuild itself, but because a
    structural rebuild will *really* trigger a nuke/replace of his existing
    index, which the POA console will *not* do.

    > Considering that there are more than 2 million messages for this user
    > alone requiring indexing this may take a very long time i.e. several
    > hours.


    Hours? You mean days...


    CU,
    --
    Massimo Rosen
    Micro Focus Knowledge Partner
    No emails please!
    http://www.cfc-it.de
  • Thank you for your input Massimo. Greatly appreciated.

    Only yesterday I saw the same "squeeze" error with large number of unindexed items. No GWCheck was necessary to get the process going.

    Cheers,
  • Am 16.05.2017 um 12:34 schrieb laurabuckley:
    >
    > Thank you for your input Massimo. Greatly appreciated.
    >
    > Only yesterday I saw the same "squeeze" error with large number of
    > unindexed items. No GWCheck was necessary to get the process going.


    You must have been incredibly lucky. I've been doing this routine quite
    a bit recently (it seems every "real" system updated from earlier
    version to 2104R2 seems to suffer from this error on at least some
    meailboxes), and the http console way has never done anything good.

    Nevertheless, I strongly believe the first step here should be to check
    why this mailbox has *minimum* 2 million items. This isn't anything
    that's remotely feasible.

    CU,
    --
    Massimo Rosen
    Micro Focus Knowledge Partner
    No emails please!
    http://www.cfc-it.de
  • Thank you for sharing your experiences Massimo.
  • Thanks Laura and Massimo for your help here.

    I did a structural rebuild on this user, and then followed the steps to completely rebuild the QF index for this user. So far, it's done 1.8 million records in about 20 hours.

    As to why this user has so many emails - we had a large number of staff leave and moved all of their emails onto one 'Archive' account.
    This 'Archive' account has around 200GB of email that we need access to.

    The overnight QF updates are on the default settings, and this works ok for all of our other post offices.

    We don't actually need QF indexes on this 'Archive' account. Is it possible to disable QF indexing on a per user basis, or is it just by post office?
    If it's by PO, I could create another PO with QF disabled, and then move this 'Archive' onto that.

    2 million items does seem excessive though... I've just ran gwcheck mailbox stats for this user, and it has a total on 113905 mails / appointments / notes / tasks.
    Am I misunderstanding what the 2 million items are?

    Thanks again
    Kevin

    On 16/05/2017 12:54, Massimo Rosen wrote:
    > Am 16.05.2017 um 12:34 schrieb laurabuckley:
    >>
    >> Thank you for your input Massimo. Greatly appreciated.
    >>
    >> Only yesterday I saw the same "squeeze" error with large number of
    >> unindexed items. No GWCheck was necessary to get the process going.

    >
    > You must have been incredibly lucky. I've been doing this routine quite a bit recently (it seems every "real" system updated from earlier version to 2104R2 seems to suffer from this error on at least some meailboxes), and the http console way has never done anything good.
    >
    > Nevertheless, I strongly believe the first step here should be to check why this mailbox has *minimum* 2 million items. This isn't anything that's remotely feasible.
    >
    > CU,


  • Hi.

    Am 16.05.2017 um 16:22 schrieb Kevin White:
    > Thanks Laura and Massimo for your help here.
    >
    > We don't actually need QF indexes on this 'Archive' account. Is it
    > possible to disable QF indexing on a per user basis, or is it just by
    > post office?
    > If it's by PO, I could create another PO with QF disabled, and then move
    > this 'Archive' onto that.


    It's by PO, but don't bother. As I assume this account doesn't really
    chnage much, QF won't be a problem once everything is indexed
    succesfully. From then, it'll only do deltas.

    > 2 million items does seem excessive though... I've just ran gwcheck
    > mailbox stats for this user, and it has a total on 113905 mails /
    > appointments / notes / tasks.
    > Am I misunderstanding what the 2 million items are?


    No, you aren't. There must be something else. An Alarm of an appointment
    for instance is an item too. But I would really run in-depth gwchecks on
    that account to see what's up there.

    CU,
    --
    Massimo Rosen
    Micro Focus Knowledge Partner
    No emails please!
    http://www.cfc-it.de
  • Well, it took a while, but this huge user in now fully indexed.

    Looking at the history on this account, it looks like a large number of emails were moved into it, all at the same time.
    I guess this would have massively increased the pending quick finder jobs, leading to a backlog with other users on this post office.

    If we have another batch of staff leave, there is a good chance this will happen again. So I'm going to create a new post office with QF indexing disabled, and move this account there.

    Thanks for your help
  • Hi.

    Am 17.05.2017 um 11:17 schrieb Kevin White:
    > Well, it took a while, but this huge user in now fully indexed.
    >
    > Looking at the history on this account, it looks like a large number of
    > emails were moved into it, all at the same time.
    > I guess this would have massively increased the pending quick finder
    > jobs, leading to a backlog with other users on this post office.


    By default, that shouldn't happen. Each quickfinder run by default only
    indexes 500 Items per User, but it always processes all users. A single
    mailbox with a huge backlog (and plenty of activity) might never
    complete an index because more than 500 items change per QF interval,
    but that doesn't affect other mailboxes. Most certainly not when the QF
    attampt on that mailbox fails with an error, then it never affects
    anything else.

    What *can* happen is, when QF is configured to always index everything
    per mailbox, and you have a mailbox with a huge backlog, that it takes a
    very long time to inex that one, which will delay processing on other
    mailboxes. This can now be overcome by configuring multiple threads for
    indexing (means, more than one mailbox is indexed at the same time, so
    that no single mailbox can delay thew whole process).

    That said, your idea to use a "archive" PO for this is interesting for
    toher reasons (like backup for instance), but for your issue not really
    relevant.

    CU,
    --
    Massimo Rosen
    Micro Focus Knowledge Partner
    No emails please!
    http://www.cfc-it.de