Tuning OBM for 100+ concurrent UI sessions

Hi all,

in an HA setup with 2 single servers, both configured for GW&DPS we face significant UI performance degradation when more than 15-20 users are logged in concurrently. I tried to manually adapt some of the process memory settings, but I am not sure, which of these processes would be the main culprit for the slowdown (up to UI messages, that the server does not respond).

Is there a best practice guide to tune OBM for a higher number of UI sessions? We have 44GB of memory in each server and about 22k CIs currently in the RTSM.Which process typically slows down UIs and what is an acceptable heap usage in the corresponding performance graph for the OBM servers?

Thanks in advance for any sharing of best practice or tuning hints.

Best regards,
Lars Droege

  • Please take a look at https://softwaresupport.softwaregrp.com/doc/KM03801492. OBM supports up to 200 concurrent session per GW. But the performance depends on the CPUs mainly for the user session. Make sure your CPU number matches the deployment that you have and if needed add additional. Also make sure the LB is setup as per the guides using stickiness by IP. The process that runs the application is the MercuryAS on the GW so more memory should be there. 

  • Hello Rosen,

    Thank you for your reference to the sizing guide.. At the moment we do have 12 CPUs on both "Single Servers" of the HA setup. Adding some more Memory to the Application Server helped the situation at the moment.

    One reason for the high load may be the usage of large views for some operators. The guide recommends not to use views with more than 3000 CIs but for some roles we currently have 10k or even 14k CIs in a view. Can this be better supported with more memory or CPUs or is this a general problem that we need to fix (split the views)?

    Best regards,

    Lars

  • Verified Answer

    OBM can handle views even up to 20K CIs.

    each GW can handle up to 120 concurrent users, but depends on your deployment levels, you may need more than 12 CPUs... I think 16 is better...

  • Thank you Asaf,

    unfortunately it seems like there is no clear metric or message that could point us to the actual bottleneck, right?

    We could add more CPUs if needed, but for the moment just based on your "feelings", not based on measurements that would clearly indicate a CPU resource problem.

    The OS level CPU load did not look too high when we faced the slow response times...

    If you know of any further analytic tools for GW performance, I would be happy to try them.

    Best regards,
    Lars

  • Verified Answer

    check OBM self monitoring dashboards...

    open Performance Perspective, select "OMi Deployment" view, then select the GW and DPS CIs. each has few OOTB dashboards.

  • Yes, I know these dashboards. Some Java heaps peaked at 50% and one even at 70%. So we added some memory to these processes. Of course the figures went down afterwards and it seemed like the imminent performance problem was solved. But we were only running 25 UIs at the time and the customer plans to have more than hundred.

    Therefore the customer is a bit tense to put this environment into production. But we probably have to run this as "trial and error" and add resources as we deem appropriate on the fly.

    Thank you for your support!

  • Usually the testing and specs are done based on distributed environment. I believe numbers would change if we have stand alone or all in one servers.

    Not sure about DB situation..

    if all is on same box this will impact the session load. Considering the views are also large that will add up in query performance and UI rendering...

    We used to have user session/connection guide back in VPO/ITO days.. may be something still there in KB.

    Also there might be some JMX tweaking/JBoss tune-up needed for single server situation.

  • Hello,

    just to clarify the environment setup: It´s a single server HA (two single servers in HA setup) with an external oracle DB, all behind a netscaler load balancer.

    Is it worth checking the DB´s performance as well? I´m not sure about that as the situation (concurrent users being logged on) doesn´t seem to be DB intense

  • UI session load and hit  will be usually a GW scaling issue.

    If your views and event load in the browser are large that will defi. hit the DB. I suggest to read the white paper refereed above in detail (OBM_Classic_2020.05_Perfsizing) by .

    UI session loading and designing is an interesting part of solution architecture(That I am not). You have to consider load on browser, network response, DB queries, SSL or not and DB load at that time. Considering if you have very large views and 1000s of events browser will take quiet a bit to load/refresh . ALso if dashboards have either too many objects or custom services, will be another avenue to check.

    Having both GW and DP on same server also introduces bottleneck for network interface and disk load.

    In similar situation we usually create baseline(default recommended sizing)  and 80%+ load scenario in Lab and then come to a realistic solution but that depends on customer budget.