Adel_HPOO_1020 Super Contributor.
Super Contributor.
626 views

Buffer is full

Jump to solution

Hi every one, 

I am an HPOO 10.20 consultant. 

Since 2 days i have a problem with  OO central worker which do not execute any flow and the list of flows " in execution"  is bigger and bigger without any resolution. 

From server log,  i detect the following warning (in more than 20 occurences) 

(AbstractEventsBuffer. Java 63) WARN -  Step log execution events: buffer is full. 

Could some one tell me what does it means please?  And why it appears ?  

Thanks a lot

 

0 Likes
1 Solution

Accepted Solutions
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

If I had to guess based on what I've seen so far: Central isnt processing its internal queues as efficiently as it probably needs to. I think its probably exacerbated by the fact there is only one worker to assign jobs to. However, I would focus on the trigger firings - my prod env now has 9 clustered centrals and 30 dedicated ras processing a few thousand jobs an hour (mostly via scheduler) and anytime I've seen issues such as you've described it was mostly resolvable by trigger tuning. I've also made other changes to the engine and quartz configs but the conditions that led me there arent in your current error messages.

Although, i would work to get upgraded to the latest version sooner as well... a lot of enhancements that you're missing out on that aided in job execution speed etc...

0 Likes
58 Replies
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

take a look at the monitoring page (https://<central>/oo/monitoring). Check for long running SQL, long running jobs, resource issues with JVM.

I'd also suggest looking at upgrading, you're about 7 versions behind - several fixes have gone in to resolve queueing issues. 

 

 

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

You can also clear the queue/ set all currenting executing jobs to cancelled using the following SQL on your OO DB - doesnt help you resolve the issue long term but it'll get you functional again

TRUNCATE TABLE OO_FINISHED_BRANCHES
GO

DELETE FROM [OO_SUSPENDED_EXECUTIONS]
GO

TRUNCATE TABLE OO_EXECUTION_INTERRUPTS
GO

TRUNCATE TABLE OO_EXECUTION_QUEUES
GO
truncate table OO_EXECUTED_FLOW_GRAPH
GO
TRUNCATE TABLE OO_EXECUTION_STATE
GO
TRUNCATE TABLE OO_EXECUTION_STATES
GO
TRUNCATE TABLE OO_EXECUTION_STATES_1
GO
TRUNCATE TABLE OO_PAUSE_DATA
GO
TRUNCATE TABLE OO_SUSPENDED_EXECUTIONS
GO
delete from [OO_RUNTIME_VALUE_STORE]
go
UPDATE OO_EXECUTION_SUMMARY
SET STATUS = 'CANCELED', END_TIME = START_TIME
WHERE STATUS IN ('RUNNING', 'PAUSED', 'PENDING_PAUSE', 'PENDING_CANCEL')
GO

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

thanks a lot for your answers.

i will study each one with my colleagues and go back to you.

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

Hi again,

After disscussion with colleagues :

The monitoring page does not help us, because , we had applied a database restore , so this page does not contain information about the critical period (when OO stops executiong flows and we had the buffer is full message), the restore of the database took us to 3 days earlier.

Just for information, we have found other warning in the critical window:

ERROR - HHH000315: Exception executing batch [Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1]

WARN  - HHH020003: Could not find a specific ehcache configuration for cache named [com.hp.oo.security.authn.entities.UserRef]; using defaults.

WARN  - SQL Error: 2627, SQLState: 23000

ERROR - Violation of UNIQUE KEY constraint 'OO_FINISHED_BRANCHES_UC'. Cannot insert duplicate key in object 'dbo.OO_FINISHED_BRANCHES'. The duplicate key value is ......

thanks

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

the data in this page isnt kept in the DB but a file store on central in the tomcat temp directory.... it may still provide some insight - in the event you run into this again this is one of the first places i would suggesst looking 

just to verify the environment was restored to a point prior to the error conditiion (so you're operating normally now?)... if so, i would focus on upgrading to a more recent edition to take advantage of the numerous updates since 10.20

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

Hi jarodMB 

So for the upgrade , we will go to the last version 2018.** in few months .

For our situation now,  really,  we are not operating normally because every day at 8 am the buffer is full message appears, the worker stops and we apply a databade restore (which is not a good solution because we lose all recent executions). 

The problem is that we have not recent developped flows,  all are old and they are in execution for at least 2 months without problems. 

Regards

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

When did the issues start?
Are you performing regular DB maintenance?
Can you explain your enviornment (number of centrals/ workers) / what DB are you using/ Size of DB (actual not allocated)

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

So the problem starts as follows

Sunday at 8 am we found in logs that the disk partition containing OO DB ldf (DB log)  is full so we resolved that by getting more space. 

After that we had problems with worker each 1 hour (during sunday)  so we proceed to many service restarts (but the problem stay the same) and then we executed the first DB restore. 

Monday morning,  worker didn't work starting from 8 am

Tuesday same thing and today same thing. 

We are using SQL Server DB. 

For DB maintenance,  we have only the DB purge  which runs at 2 am everyday to keep only last week executions data. 

For the architecture we have one central (which is the only worker). 

Db size ~400 GB. 

Thanks a lot 

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

You might check your index fragmentation and reindex where necessary. 

Can you share your config in database.properties and central-wrapper.conf (please make sure you mask sensitive info)

Also, look at central/tomcat/temp - do you see a large number of jtds files? 

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

So for conf files,  i wilk share it tomorrow because i am out of office. 

In tomcat tmp direcyory i checked it today in the mlrning  there was ~15000 files. 

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

you might stop oo and delete all files under temp (excluding the javamelody directory (this is where the /oo/monitoring data is stored))

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

I asked my colleague to delete these files now,  but could you please tell me what does these files mean ? 

Thanks a lot

0 Likes
Outstanding Contributor.. JarodMB Outstanding Contributor..
Outstanding Contributor..

Re: Buffer is full

Jump to solution

Tomcat uses this directory for some of its temporary activities - included with that are the jdbc transactions against the db... these files are usually deleted after an operation completes. Ive seen in my environment that if this directory fills with stale updates that I get unpredictability in the application. In the newer versions this has been less of a noticable problem but on 10.2 when our DB was experiencing issues clearing this content aided us. 

0 Likes
Adel_HPOO_1020 Super Contributor.
Super Contributor.

Re: Buffer is full

Jump to solution

Thanks a lot for these details,  i will come back as soon as possible to tell you if it resolve my issue. 

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.