Reason for Cost Rollup errors

Hello Team,  we currently have 3 projects that have been getting cost rollup failures.  We run our rollup service every hour and monitor the 'itg_pending_rollups' table.  We would like to try to understand  what are the potential causes for a project to fail cost rollup?  We have checked the audit trails of the projects and cannot determine anything out of the ordinary that was done and tasks and schedule are set accordingly.  Even if we try to delete the 'error' row from the table once another charge or event is done to these projects, they get stuck again.  Is there anyway to clear or reset the projects again in order for them to successfully process a rollup?  Thanks for any help or suggestions.  Frank 

  • Hi Frank,


    Can you please share here the full error?

  • Hi Frank,


    Can you please share here the full error?

  • Hi Frank,



    You also can try to set more time to the cost rollup service to run this is a heavy service and it requires time to execute.

    As an example, if you have your service set to run for an hour, set it to an hour and a half or 2 hours.


    Another good idea is to have two services nodes, one for the light services and one for the heavy ones like the cost rollup service.



  • Hi Randall, thank you for the timing suggestion.  We extended the timing of the service  in our Certification but the projects in error did not clear out.   Since I have a portlet that monitors the pending rollups that come in we can see if any do not process within the hour.  We don't see that we are having any backlog or leftover rollups with each hourly run.  These three projects have been failing for more than 2 months.    We were just trying to understand what steps or task process could have been done to cause the collup to fail?  We tried deleting the error row from the rollup table.  But if someone charges or updates these projects and begin to fail again.


    Also, We have noticed that in PPM 9.31 there is now a cost message that appears "The cost for this project is not up to date. The last update event was detected at 5/19/15 03:28:05 PM CDT.The service has failed to rollup cost for the current project 19 times, please check with PPM administrator."  


    So now we are questioning as to what we as adminstrators are expected to do?   


    We have already gone the support route and have had numerous discussions but no one has been able to specifically tell us how to clear off the error.  It's like the error has been queued up somewhere so wondering if there is a procedure or process we can do so straighten out the error so it does not re-occur.   Is there a workaround to clear.


    Appreciate your advice or any additional suggestions you have.



  • Hello, please find the attached server text log that indicates the projects failing.  Thank you.

  • Hi Frank,


    I found an internal article, Please see the below information and let me know if this helped you:


    A project has stopped the rollup of costs for actual time reported through time management.
    The stack trace error below is recorded in serverLog, on execution of Cost Rollup Service:

    ERROR :ppmHeavyServiceListenerContainer-1:com.mercury.itg.core.cost:2012/06/14-08:46:41.670 CEST: cost rollup with id 6464085 failed.
    STATUS server:ppmHeavyServiceListenerContainer-1:SystemErr:2012/06/14-08:46:41.779 CEST: java.lang.NullPointerException
    STATUS server:ppmHeavyServiceListenerContainer-1:SystemErr:2012/06/14-08:46:41.779 CEST:  at STATUS server:ppmHeavyServiceListenerContainer-1:SystemErr:2012/06/14-08:46:41.779 CEST:  at
    STATUS server:ppmHeavyServiceListenerContainer-1:SystemErr:2012/06/14-08:46:41.779 CEST:  at$BaseCostRollupVisitor.calculateTaskEV(
    STATUS server:ppmHeavyServiceListenerContainer-1:SystemErr:2012/06/14-08:46:41.779 CEST:  at$WorkPlanRollupVisitor.visit(\

    Data corruption from unknown source, leading to some planned values in WP_TASK_COSTS have NULL value. If this is an upgraded from a previous PPM version project, having old baselines created to it, data corruption related to the old baselines could also be related to that error.

    The following Sql queries could be used to check the data in table WP_TASK_COSTS:
     SELECT *
     FROM wp_task_costs
     WHERE planned_value_bse is NULL or planned_value_lcl is NULL or planned_value_cur_lcl is NULL

    If the SQL statement returns any record, it could be fixed by using the next SQL statements:

     UPDATE wp_task_costs SET planned_value_bse=0 WHERE planned_value_bse is NULL
     UPDATE wp_task_costs SET planned_value_lcl=0 WHERE planned_value_lcl is NULL
     UPDATE wp_task_costs SET planned_value_cur_lcl='USD' WHERE planned_value_cur_lcl is NULL

     *replace "USD" in the third query with the base currency setup in that instance

    If that project has old baselines created, a new baseline for that project's workplan could be created, to avoid data corruption issues in its baselines.


    Best Regards,


  • Hi Carolina, our tech lead ran the scripts you noted in this case


    SELECT *
    FROM wp_task_costs
    WHERE planned_value_bse is NULL or planned_value_lcl is NULL or planned_value_cur_lcl is NULL


    but it came back showing we have over 170 thousand rows where these fields are Null, spanning over the years, so needing to go in and update all those rows sounds our of the ordinary .


    Are the NULL fields expected or should they be there based on how the planned values are being calculated for the projects.


    Let us know if you have any other information or suggestions.