Highlighted
Trusted Contributor.
Trusted Contributor.
1400 views

Large number of Quick Finder pending jobs

Hi,

I've been having a problem with a user losing missing some of his folders, and stumbled upon another problem with Quick Finder pending jobs.

Running a GWCheck Analyze/Fix Databases, I have a number of users on this post office with lines such as :

STRUCTURAL VERIFICATION of database /var/opt/novell/groupwise/mail/tek/ofuser/userydz.db
Verifying data
- Database is structurally consistent
- Space that could be recovered: 0 bytes
Quickfinder: Pending jobs = 2009538 Files = 539caf3d.idx, 58d81e29.inc

This is one of the worst cases, but over half of the users have at least 1000+ pending jobs.

I've following the steps in https://www.novell.com/support/kb/doc.php?id=7012997, which did solve a couple of users. However, the above user resulted in this in the post office log :

15:51:14 EECD Starting QuickFinder index compression
15:51:15 EECD Scanning Post Office for Orphaned Files
15:51:18 EECD Finished Post Office Scan for Orphaned Files
15:51:22 EECD Compressing QuickFinder index: userydz.db (2009538)
15:51:39 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:10 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:49 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:52:49 EECD The database function 87 reported error [F03E] on userydz.db
15:52:49 EECD Error updating QuickFinder indexes: [F03E]
15:52:49 EECD Updating QuickFinder index: userydz.db (2009538)
15:53:02 EECD Error: Memory Allocation error [F03E] in Squeeze.QFSqzIndex.2 ()
15:53:02 EECD The database function 87 reported error [F03E] on userydz.db
15:53:02 EECD Error updating QuickFinder indexes: [F03E]
15:53:04 EECD QuickFinder indexing thread finished

The server has 6GB of RAM, so not huge, but doesn't seem to have any memory issues - swap is barely used for example.

Groupwise system is 14.2.2, recently upgraded from 2012 SP4. All servers are OES2015 SP1

Any help would be appreciated,
Thanks
Kevin

Labels (1)
0 Likes
12 Replies
Highlighted
Knowledge Partner
Knowledge Partner

Re: Large number of Quick Finder pending jobs

Hi.

Some basics first:

How is your quickfinder configured, e.g what's the start time, and how
often does it run?

Second, that Mailbox you're having issues with is *ENORMOUS*. 2 Million
unindexed objects (aka mails, appointments, you name it) is a league of
it's own. What's up with that?

Third, to fix it, after you have set your quickfinder to run often
enough and index enough items per run, do a structural rebuild on that
user database. That will nuke the current index for the mailbox annd
start over.

Set the post office logs to verbose, and check it for a while if the
number of items left to index decreases (and no errors are shown) when
QF runs.



Am 15.05.2017 um 17:10 schrieb Kevin White:
> Hi,
>
> I've been having a problem with a user losing missing some of his
> folders, and stumbled upon another problem with Quick Finder pending jobs.
>
> Running a GWCheck Analyze/Fix Databases, I have a number of users on
> this post office with lines such as :
>
> STRUCTURAL VERIFICATION of database
> /var/opt/novell/groupwise/mail/tek/ofuser/userydz.db
> Verifying data
> - Database is structurally consistent
> - Space that could be recovered: 0 bytes
> Quickfinder: Pending jobs = 2009538 Files = 539caf3d.idx,
> 58d81e29.inc
>
> This is one of the worst cases, but over half of the users have at least
> 1000+ pending jobs.
>
> I've following the steps in
> https://www.novell.com/support/kb/doc.php?id=7012997, which did solve a
> couple of users. However, the above user resulted in this in the post
> office log :
>
> 15:51:14 EECD Starting QuickFinder index compression
> 15:51:15 EECD Scanning Post Office for Orphaned Files
> 15:51:18 EECD Finished Post Office Scan for Orphaned Files
> 15:51:22 EECD Compressing QuickFinder index: userydz.db (2009538)
> 15:51:39 EECD Error: Memory Allocation error [F03E] in
> Squeeze.QFSqzIndex.2 ()
> 15:52:10 EECD Error: Memory Allocation error [F03E] in
> Squeeze.QFSqzIndex.2 ()
> 15:52:49 EECD Error: Memory Allocation error [F03E] in
> Squeeze.QFSqzIndex.2 ()
> 15:52:49 EECD The database function 87 reported error [F03E] on userydz.db
> 15:52:49 EECD Error updating QuickFinder indexes: [F03E]
> 15:52:49 EECD Updating QuickFinder index: userydz.db (2009538)
> 15:53:02 EECD Error: Memory Allocation error [F03E] in
> Squeeze.QFSqzIndex.2 ()
> 15:53:02 EECD The database function 87 reported error [F03E] on userydz.db
> 15:53:02 EECD Error updating QuickFinder indexes: [F03E]
> 15:53:04 EECD QuickFinder indexing thread finished
>
> The server has 6GB of RAM, so not huge, but doesn't seem to have any
> memory issues - swap is barely used for example.
>
> Groupwise system is 14.2.2, recently upgraded from 2012 SP4. All servers
> are OES2015 SP1
>
> Any help would be appreciated,
> Thanks
> Kevin
>



--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Large number of Quick Finder pending jobs

Hi Kevin,

Your Quick Finder Indexing is horribly behind in indexing your user databases. The "Squeeze" error is very indicative of this as is the number of un-indexed items per mail box.

To reindex this entire Post Office in one go is going to cripple your system. The TID you followed is a good start but I'm going to recommend some more information as to what steps to take on the users so incredibly out of date. After we have dealt with such users we are going to have to fix indexing on your entire Post Office in order to get the entire system up to date, and then to ensure QF Indexing is configured is such a way that you never get to this situation again.

So starting with individual users:

Go to the Web Console of your post office.
Login as a user with full rights to the post office or your admin user.
Go to the Configuration tab.
Find Log Settings and click on that to open the configuration page for logging.
Select Diagnostic as the log level.
Click on Submit.
Without Logging being in Diagnostic level several of the next options regarding quick finder indexing may not be visible.
Return to the Configuration page.
Find Quick Finder Indexing and click on that link.
Under Indexing Options do the following:

- Set the Indexing Level to Unlimited
- Then there are four tick boxes. Three of them should already be ticked, leave them ticked.
- Put a tick in the fourth box to delete the old qfinder files
- then place the user FID into both the Beginning and End box.

Click Submit.

Considering that there are more than 2 million messages for this user alone requiring indexing this may take a very long time i.e. several hours. Only do one user at a time and let it complete before attempting another one.

Try that, let us know if this user database is indexed. If you still get "Squeeze" errors then further intervention will be required.

Looking forward to hearing back from you.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: Large number of Quick Finder pending jobs

Laura,

Am 16.05.2017 um 07:54 schrieb laurabuckley:
>
> Go to the Configuration tab.
> Find Log Settings and click on that to open the configuration page for
> logging.
> Select Diagnostic as the log level.
> Click on Submit.
> Without Logging being in Diagnostic level several of the next options
> regarding quick finder indexing may not be visible.


Diagnostics is overkill here. Verbose will do.

> - Set the Indexing Level to Unlimited
> - Then there are four tick boxes. Three of them should already be
> ticked, leave them ticked.
> - *Put a tick in the fourth box to delete the old qfinder files*
> - then place the user FID into both the Beginning and End box.
>
> Click Submit.


With his error, that will unfortunately not change a thing. Been there,
done that. The only way to get QF going again for that mailbox is to do
a structural rebuild (not because of the rebuild itself, but because a
structural rebuild will *really* trigger a nuke/replace of his existing
index, which the POA console will *not* do.

> Considering that there are more than 2 million messages for this user
> alone requiring indexing this may take a very long time i.e. several
> hours.


Hours? You mean days...


CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Large number of Quick Finder pending jobs

Thank you for your input Massimo. Greatly appreciated.

Only yesterday I saw the same "squeeze" error with large number of unindexed items. No GWCheck was necessary to get the process going.

Cheers,
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: Large number of Quick Finder pending jobs

Am 16.05.2017 um 12:34 schrieb laurabuckley:
>
> Thank you for your input Massimo. Greatly appreciated.
>
> Only yesterday I saw the same "squeeze" error with large number of
> unindexed items. No GWCheck was necessary to get the process going.


You must have been incredibly lucky. I've been doing this routine quite
a bit recently (it seems every "real" system updated from earlier
version to 2104R2 seems to suffer from this error on at least some
meailboxes), and the http console way has never done anything good.

Nevertheless, I strongly believe the first step here should be to check
why this mailbox has *minimum* 2 million items. This isn't anything
that's remotely feasible.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: Large number of Quick Finder pending jobs

Thank you for sharing your experiences Massimo.
Laura Buckley

Views/comments expressed here are entirely my own.
If you find this post helpful, please show your appreciation and click on "Like" below...
0 Likes
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: Large number of Quick Finder pending jobs

Thanks Laura and Massimo for your help here.

I did a structural rebuild on this user, and then followed the steps to completely rebuild the QF index for this user. So far, it's done 1.8 million records in about 20 hours.

As to why this user has so many emails - we had a large number of staff leave and moved all of their emails onto one 'Archive' account.
This 'Archive' account has around 200GB of email that we need access to.

The overnight QF updates are on the default settings, and this works ok for all of our other post offices.

We don't actually need QF indexes on this 'Archive' account. Is it possible to disable QF indexing on a per user basis, or is it just by post office?
If it's by PO, I could create another PO with QF disabled, and then move this 'Archive' onto that.

2 million items does seem excessive though... I've just ran gwcheck mailbox stats for this user, and it has a total on 113905 mails / appointments / notes / tasks.
Am I misunderstanding what the 2 million items are?

Thanks again
Kevin

On 16/05/2017 12:54, Massimo Rosen wrote:
> Am 16.05.2017 um 12:34 schrieb laurabuckley:
>>
>> Thank you for your input Massimo. Greatly appreciated.
>>
>> Only yesterday I saw the same "squeeze" error with large number of
>> unindexed items. No GWCheck was necessary to get the process going.

>
> You must have been incredibly lucky. I've been doing this routine quite a bit recently (it seems every "real" system updated from earlier version to 2104R2 seems to suffer from this error on at least some meailboxes), and the http console way has never done anything good.
>
> Nevertheless, I strongly believe the first step here should be to check why this mailbox has *minimum* 2 million items. This isn't anything that's remotely feasible.
>
> CU,


0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: Large number of Quick Finder pending jobs

Hi.

Am 16.05.2017 um 16:22 schrieb Kevin White:
> Thanks Laura and Massimo for your help here.
>
> We don't actually need QF indexes on this 'Archive' account. Is it
> possible to disable QF indexing on a per user basis, or is it just by
> post office?
> If it's by PO, I could create another PO with QF disabled, and then move
> this 'Archive' onto that.


It's by PO, but don't bother. As I assume this account doesn't really
chnage much, QF won't be a problem once everything is indexed
succesfully. From then, it'll only do deltas.

> 2 million items does seem excessive though... I've just ran gwcheck
> mailbox stats for this user, and it has a total on 113905 mails /
> appointments / notes / tasks.
> Am I misunderstanding what the 2 million items are?


No, you aren't. There must be something else. An Alarm of an appointment
for instance is an item too. But I would really run in-depth gwchecks on
that account to see what's up there.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: Large number of Quick Finder pending jobs

Well, it took a while, but this huge user in now fully indexed.

Looking at the history on this account, it looks like a large number of emails were moved into it, all at the same time.
I guess this would have massively increased the pending quick finder jobs, leading to a backlog with other users on this post office.

If we have another batch of staff leave, there is a good chance this will happen again. So I'm going to create a new post office with QF indexing disabled, and move this account there.

Thanks for your help & advice.
Kevin

On 16/05/2017 06:54, laurabuckley wrote:
>
> Hi Kevin,
>
> Your Quick Finder Indexing is horribly behind in indexing your user
> databases. The "Squeeze" error is very indicative of this as is the
> number of un-indexed items per mail box.
>
> To reindex this entire Post Office in one go is going to cripple your
> system. The TID you followed is a good start but I'm going to recommend
> some more information as to what steps to take on the users so
> incredibly out of date. After we have dealt with such users we are
> going to have to fix indexing on your entire Post Office in order to get
> the entire system up to date, and then to ensure QF Indexing is
> configured is such a way that you never get to this situation again.
>
> So starting with individual users:
>
> Go to the Web Console of your post office.
> Login as a user with full rights to the post office or your admin user.
>
> Go to the Configuration tab.
> Find Log Settings and click on that to open the configuration page for
> logging.
> Select Diagnostic as the log level.
> Click on Submit.
> Without Logging being in Diagnostic level several of the next options
> regarding quick finder indexing may not be visible.
> Return to the Configuration page.
> Find Quick Finder Indexing and click on that link.
> Under Indexing Options do the following:
>
> - Set the Indexing Level to Unlimited
> - Then there are four tick boxes. Three of them should already be
> ticked, leave them ticked.
> - *Put a tick in the fourth box to delete the old qfinder files*
> - then place the user FID into both the Beginning and End box.
>
> Click Submit.
>
> Considering that there are more than 2 million messages for this user
> alone requiring indexing this may take a very long time i.e. several
> hours. Only do one user at a time and let it complete before attempting
> another one.
>
> Try that, let us know if this user database is indexed. If you still
> get "Squeeze" errors then further intervention will be required.
>
> Looking forward to hearing back from you.
>
> Cheers,
>
>


0 Likes
Highlighted
Knowledge Partner
Knowledge Partner

Re: Large number of Quick Finder pending jobs

Hi.

Am 17.05.2017 um 11:17 schrieb Kevin White:
> Well, it took a while, but this huge user in now fully indexed.
>
> Looking at the history on this account, it looks like a large number of
> emails were moved into it, all at the same time.
> I guess this would have massively increased the pending quick finder
> jobs, leading to a backlog with other users on this post office.


By default, that shouldn't happen. Each quickfinder run by default only
indexes 500 Items per User, but it always processes all users. A single
mailbox with a huge backlog (and plenty of activity) might never
complete an index because more than 500 items change per QF interval,
but that doesn't affect other mailboxes. Most certainly not when the QF
attampt on that mailbox fails with an error, then it never affects
anything else.

What *can* happen is, when QF is configured to always index everything
per mailbox, and you have a mailbox with a huge backlog, that it takes a
very long time to inex that one, which will delay processing on other
mailboxes. This can now be overcome by configuring multiple threads for
indexing (means, more than one mailbox is indexed at the same time, so
that no single mailbox can delay thew whole process).

That said, your idea to use a "archive" PO for this is interesting for
toher reasons (like backup for instance), but for your issue not really
relevant.

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Highlighted
Trusted Contributor.
Trusted Contributor.

Re: Large number of Quick Finder pending jobs

Hi Massimo,

Thanks for the explanation. I've got the pending QF jobs down to a manageable level now (ie they're all less than 500 per user), so I'll monitor this over the next few weeks and see if our QF settings are suitable for this post office.

Regards
Kevin

On 17/05/2017 10:31, Massimo Rosen wrote:
> Hi.
>
> Am 17.05.2017 um 11:17 schrieb Kevin White:
>> Well, it took a while, but this huge user in now fully indexed.
>>
>> Looking at the history on this account, it looks like a large number of
>> emails were moved into it, all at the same time.
>> I guess this would have massively increased the pending quick finder
>> jobs, leading to a backlog with other users on this post office.

>
> By default, that shouldn't happen. Each quickfinder run by default only indexes 500 Items per User, but it always processes all users. A single mailbox with a huge backlog (and plenty of activity) might never complete an index because more than 500 items change per QF interval, but that doesn't affect other mailboxes. Most certainly not when the QF attampt on that mailbox fails with an error, then
> it never affects anything else.
>
> What *can* happen is, when QF is configured to always index everything per mailbox, and you have a mailbox with a huge backlog, that it takes a very long time to inex that one, which will delay processing on other mailboxes. This can now be overcome by configuring multiple threads for indexing (means, more than one mailbox is indexed at the same time, so that no single mailbox can delay thew whole
> process).
>
> That said, your idea to use a "archive" PO for this is interesting for toher reasons (like backup for instance), but for your issue not really relevant.
>
> CU,


0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.