wrusman Absent Member.
Absent Member.
698 views

Access Gateway 4.4.4 memory issues or tuning?

Hello,
we recently upgraded our NAM and AG's to version 4.4 and also moved the virtualized the envionment from LXC on SLES11 to another container platform using LXC and cgroups.
Since this move/upgrade we suffer from the AG's memory filling up further every day until apache starts defunctioning and a novell-apache2 restart is needed to fix the functionality
Caching is disabled, and only two sites are form-filled with the AG's so load is relatively low.

Both AG's were cleanly installed with SLES12SP3 and novell-access-gateway-4.4.4.0-22 and both were equipped with 4 GB memory as recommended per the install guide.
We have already upgraded them to 6 GB which did not seem to fix the issue and now the test is running with 8 GB per AG.

Is there a known memory leak or maybe there is a tuning parameter to fix this behaviour ?
0 Likes
11 Replies
Knowledge Partner
Knowledge Partner

Re: Access Gateway 4.4.4 memory issues or tuning?

On 18-04-2019 7:04 PM, wrusman wrote:
>
> Hello,
> we recently upgraded our NAM and AG's to version 4.4 and also moved the
> virtualized the envionment from LXC on SLES11 to another container
> platform using LXC and cgroups.
> Since this move/upgrade we suffer from the AG's memory filling up
> further every day until apache starts defunctioning and a novell-apache2
> restart is needed to fix the functionality
> Caching is disabled, and only two sites are form-filled with the AG's so
> load is relatively low.
>
> Both AG's were cleanly installed with SLES12SP3 and
> novell-access-gateway-4.4.4.0-22 and both were equipped with 4 GB memory
> as recommended per the install guide.
> We have already upgraded them to 6 GB which did not seem to fix the
> issue and now the test is running with 8 GB per AG.
>
> Is there a known memory leak or maybe there is a tuning parameter to fix
> this behaviour ?


Which process is consuming all the memory? httpd, tomcat or anything else? As for the formfil, i wouldn't expect mem issues with that but more
slowness if you apply the FF policy to something like /*. Best to keep the path as small as possible.


--
Cheers,
Edward
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

Hello Edward,
The process seems to be apache2, after a restart of this process the memory is also free again.

The formfill is'nt (and hasn't been) working slow and the paths are as exact as they can be.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Access Gateway 4.4.4 memory issues or tuning?

On 24-04-2019 9:26 PM, wrusman wrote:
>
> Hello Edward,
> The process seems to be apache2, after a restart of this process the
> memory is also free again.
>
> The formfill is'nt (and hasn't been) working slow and the paths are as
> exact as they can be.
>
>


no one has reported any memory leaks in here for apache2, tho I have seen them in the past. They are probably hard to resolve without an actual SR in
place.

Can you tell us a bit more about your environment? Do you have high load? Multiple AGs load balanced? Same behaviour on all AGs?

--
Cheers,
Edward
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

Hello Edward,

The environment is University Groningen in the Netherlands.
We recently switched from SLES11SP4 LXC containers on SLES hosts to SLES12SP3 LXC containers on proxmox for ease of use.

One of the two (F5) load-balanced acces gateways shows these stats, the other one is about similar

CPU Utilization 75.0 %
Cache Hit 0.0 %
Mounted Partitions Disk Space 61.41 GB
Mounted Partitions Disk Space Used 1.834 GB
Mounted Partitions Disk Space Free 59.57 GB
Swap Partition Disk Space 0.0 Bytes
Swap Partition Disk Space Used 0.0 Bytes
Swap Partition Disk Space Free 0.0 Bytes
Cache Disk Space 0 KB
Cache Disk Space Utilization 0.0 %
Total Installed Memory 64328 MB
Start Up Time Monday, April 15, 2019 2:50:48 PM GMT
Up Time 9 Days, 17 Hours, 16 Minutes, 14 Seconds
Number of Objects Cached 0

Load
Connections
Current Connections to Origin Server 69
Current Connections to Browsers 112
Current Total Connections 181
Total WebSocket Connections 0
Idle WebSocket Connections 0
Connections to Origin Server 145003
Peak Connections from Origin Server 133
Connections to Browsers 25817
Peak Connections to Browsers 17
Failed Connection Attempts 0

Bytes
Throughput of the Origin Server 0.0 Bytes
Throughput of the Browser 19.90 KB
Total Bytes per Second 19.90 KB
Bytes Sent To Origin Server 0.0 Bytes
Bytes Received from Origin Server 0.0 Bytes
Bytes Sent to Browser 15.09 GB
Bytes Recieved from Browser 871.6 MB
Total Bytes 15.94 GB

Requests
Current Requests to Origin Server 0
Current Requests from Browsers 46
Total Current Requests 46
Successful Requests to Origin Server 0
Failed Requests to Origin Server 0
Cumulative Requests to Origin Server 0
Cumulative Requests to Browsers 450225
Total Cumulative Requests 450225
Requests per Second to Origin Server 0
Requests per Second from Browsers 0.53
Total Requests per Second 0.53
Peak Requests per Second to Origin Server 302
Peak Requests per Second from Browsers 133

Cache Freshness
Total "Get If Modified Since" Request 0
Total Not Modified Replies 0


The only quirk is the "Total Installed Memory", this shows the total memory of the host instead of the 8 GB memory limitation for the LXC, but I guess this is a cgroup thing
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

This is the graph for the memory usage which shows a daily increment. When the AG wasn't used the graph was stable at 1 usage.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Access Gateway 4.4.4 memory issues or tuning?

On 25-04-2019 6:34 PM, wrusman wrote:
>
> Hello Edward,
>
> The environment is University Groningen in the Netherlands.
> We recently switched from SLES11SP4 LXC containers on SLES hosts to
> SLES12SP3 LXC containers on proxmox for ease of use.
>


Was it working ok on SLES 11?

> The only quirk is the "Total Installed Memory", this shows the total
> memory of the host instead of the 8 GB memory limitation for the LXC,
> but I guess this is a cgroup thing
>
>


Did you change httpd-mpm.conf at all? When you run top is apache consuming a lot of RES or VIRT mem? The graph, what exactly does it monitor? I guess
you enabled SNMP but what exactly does it query for?

--
Cheers,
Edward
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

edmaa;2498871 wrote:
On 25-04-2019 6:34 PM, wrusman wrote:
>
> Hello Edward,
>
> The environment is University Groningen in the Netherlands.
> We recently switched from SLES11SP4 LXC containers on SLES hosts to
> SLES12SP3 LXC containers on proxmox for ease of use.
>


Was it working ok on SLES 11?

Yes it was, the memory usage was not measured though



> The only quirk is the "Total Installed Memory", this shows the total
> memory of the host instead of the 8 GB memory limitation for the LXC,
> but I guess this is a cgroup thing
>
>


Did you change httpd-mpm.conf at all? When you run top is apache consuming a lot of RES or VIRT mem? The graph, what exactly does it monitor? I guess
you enabled SNMP but what exactly does it query for?

The graph is from proxmox, the LXC host and shows the memory usage of the LXC container

I didn't change the mpm config, as I understood when caching is disabled this module is not actively used.
It shows around 12.5 GB VIRT memory for every of the 9 httpd processes and 11 GB VIRT for tomcat

0 Likes
Knowledge Partner
Knowledge Partner

Re: Access Gateway 4.4.4 memory issues or tuning?

On 26-04-2019 7:54 PM, wrusman wrote:
>


> I didn't change the mpm config, as I understood when caching is disabled
> this module is not actively used.


This config file controls the number of threads apache uses.

> It shows around 12.5 GB VIRT memory for every of the 9 httpd processes
> and 11 GB VIRT for tomcat
>
> [image:
> https://i.ibb.co/3rcz17H/Screenshot-from-2019-04-26-11-47-31.png]


the apache processes seem to be using quite a bit of memory (~350M each * 9 = 3G) and that plus tomcat equates to 5G+ which explains your memory
graph. Question, the proxy config of your backend app/web servers in NAM, are they based on a DNS hostname or IP address? If on DNS hostname, does
that happen to be a hostname offloaded to the F5 GTM and is load balancing between multiple LTMs?


--
Cheers,
Edward
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

Hello Edward, the load-balancing is by means of ip adresses through one virtual ip with LTM.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Access Gateway 4.4.4 memory issues or tuning?

On 27-04-2019 4:14 PM, wrusman wrote:
>
> Hello Edward, the load-balancing is by means of ip adresses through one
> virtual ip with LTM.
>
>


ok, that rules that one out. Not sure why apache seems to be consuming a lot of mem. I guess on thing you could check is if you create a SSH tunnel to
127.0.0.1:8181 on each server and then check the shared memory for the SSL/TLS session cache status on http://127.0.0.1:8181/server-status (browse all
the way to the bottom).



--
Cheers,
Edward
0 Likes
wrusman Absent Member.
Absent Member.

Re: Access Gateway 4.4.4 memory issues or tuning?

Hello Edward,

the status for both servers at this moment is:


Current Time: Monday, 06-May-2019 08:59:55 CEST
Restart Time: Tuesday, 30-Apr-2019 11:47:40 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 5 days 21 hours 12 minutes 15 seconds
Server load: 0.42 0.44 0.45
Total accesses: 263282 - Total Traffic: 8.8 GB
CPU Usage: u8842.52 s138.35 cu.01 cs.19 - 1.77% CPU load
Total load average:.518 requests/sec - 18.2 kB/second - 35.2 kB/request
Current load:3.59 requests/sec
57 requests currently being processed, 2943 idle workers

SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 524288 bytes, current entries: 10
subcaches: 32, indexes per subcache: 91
time left on oldest entries' objects: avg: 147 seconds, (range: 50...251)
index usage: 0%, cache usage: 0%
total entries stored since starting: 13987
total entries replaced since starting: 0
total entries expired since starting: 13977
total (pre-expiry) entries scrolled out of the cache: 0
total retrieves since starting: 50711 hit, 12648 miss
total removes since starting: 0 hit, 0 miss




Current Time: Monday, 06-May-2019 08:58:47 CEST
Restart Time: Tuesday, 30-Apr-2019 11:49:00 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 5 days 21 hours 9 minutes 46 seconds
Server load: 0.26 0.40 0.46
Total accesses: 295201 - Total Traffic: 10.3 GB
CPU Usage: u9384.8 s164.35 cu0 cs0 - 1.88% CPU load
Total load average:.581 requests/sec - 21.2 kB/second - 36.5 kB/request
Current load:3.74 requests/sec
57 requests currently being processed, 2943 idle workers


SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 524288 bytes, current entries: 10
subcaches: 32, indexes per subcache: 91
time left on oldest entries' objects: avg: 141 seconds, (range: 23...268)
index usage: 0%, cache usage: 0%
total entries stored since starting: 14938
total entries replaced since starting: 0
total entries expired since starting: 14928
total (pre-expiry) entries scrolled out of the cache: 0
total retrieves since starting: 54989 hit, 13590 miss
total removes since starting: 0 hit, 0 miss
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.