Memory management from the application's point of view and how it affects Service Manager
Every program needs memory to store different kind of information. It is the operating system’s responsibility to provide and to organize memory. If a program, such as Service Manager, runs out of memory it can not complete its task and usually abends. This article describes memory management from the application’s point of view and how it affects Service Manager.
What are the different types of memory?
Every application which is executed has an address space that is the maximum memory it can address. Since most computers today are 32-bit computers, the address space of an application is limited to 4 GB, some platforms only allow 31 bits for addressing thus limiting the user address space to 2 GB. This address space has to hold all the information that the application requires.
When the operating system starts a program it first of all creates the address space for the program. Then it loads the application into that address space. Next it checks if the application requires any shared libraries or DLLs, in such a case those are loaded into the user address space as well. Only after that control is passed on to the application and the application can start.
The memory which is now allocated to the application is just the minimum to barely start the application. The application also needs local and global variables, as well as some dynamic memory or even shared memory.
Global variables are variables that are visible and accessible to all subroutines of an application. Such global variables are usually stored inside the executable and therefore are allocated the moment the operating system loads the executable.
Local variables, on the other hand, are only visible and accessible to the currently active subroutine. If a subroutine is never called there is no need to allocate its local variables. Therefore, local variables are only automatically allocated on the so-called stack (see below) when the subroutine is invoked.
An application might also need to dynamically allocate memory. For global or local variables it is already known at compile time how many variables are needed and what types they are (e.g. one loop variable, one counter, one message string). Sometimes this information is not known at compile time but only when the application is running. (For example, the application could use some data structures for accessing a SQL database, but finds during run-time that it does not need to connect to the SQL database. Keeping all variables needed for SQL access around all the time would be just a waste of memory). All dynamic allocations are managed in a data structure called heap (see below).
The memory allocations above have one thing in common: they are available to the application only, the operating system ensures that no other process has access to above memory allocations. If an application wants other processes to have access to some data it has to specifically ask the operating system for shared memory. This happens during run-time of the application and is comparable to dynamic memory allocation. Other processes can connect to the shared memory and read or update it, given they have proper access rights to do so (see below)
All of the above memory allocations are automatically freed by the operating system when the application terminates.
What is a stack?
A stack is a data structure which is automatically created whenever an application is started. It is divided into so-called stack frames. Whenever a subroutine is called a new stack frame is placed onto the stack. The stack frame contains the following information:
- The return address. The application has to remember where to continue once the subroutine has completed.
- Every subroutine can receive parameters holding information the subroutine needs to its task.
- Local variables. Every subroutine may use as many local variables as it wants. Those variables are placed on stack automatically when the subroutine is called and discarded when the subroutine has finished.
When the subroutine has completed, the application simply discards the last stack frame and continues at the stored return address. This all happens without the application having to take care of managing memory.
The stack is usually preallocated by the operating system and cannot be extended once its allocation is exceeded. In such a case the application will abend and issue messages concerning a ‘stack overflow’. On UNIX platforms, you can change the stack size by either adjusting kernel parameters or using the ‘ulimit -s’ command (not available on HP-UX).
What is a heap?
A heap is a data structure that is maintained by the operating system and contains all dynamic memory allocations of the application. If the heap is full and a request for more memory is done, it is extended. The heap, however, is extended by adding more memory to its end so that the heap itself consists out of just one big piece of memory. The heap therefore can run out of memory even the machine still has plenty of memory available, if the next piece to be added to the heap is already use by something else, e.g. a shared library or shared memory.
If the allocated memory on heap is not freed, it simply grows until all memory resources are exhausted. In such a case, typically the application issues an error message (such as ‘out of memory’) and abends. A memory leak occurs when a program dynamically allocates memory on the heap but ‘forgets’ to free it again. On the other hand, a program could allocate memory but intentionally never frees it because the overhead over freeing and reallocating memory is too high.
What is shared memory?
An application can request an area of memory to be shared with other processes. All other memory allocations are considered private meaning one process cannot access another process’s memory. Typically, information that is important to every process is stored in shared memory, Service Manager keeps track all connected processes/users in shared memory, for example, or caches data records there.
Since every process can read or update data stored in shared memory at any time, access to it must be synchronized. For example, if two processes want to increase a counter by one at the exact same time, both processes read the counter first (both getting a value of e.g. 5), both increase by one (to 6) and both write it back. The counter would then be at 6, whereas it should have been 7 because two processes wanted to increase it one each. Service Manager uses semaphores to serialize access; every process tries get a semaphore first before accessing shared memory, if the semaphore is not available (because another process is updating shared memory) the process just waits until the semaphore is available again and performs its update then.
What are shared libraries?
Shared libraries contain external subroutines which applications can re-use. This way not every application has to write a subroutine of its own, for example, to print a string to the screen. There are many different shared libraries available. The most important one is libc.so (on UNIX) or MSVCRT.DLL (on Windows). These contain standard C functions such as malloc() and free(), fopen(), fclose() and printf(). When an application is started and loaded into the address space, dependencies to such libraries are detected and the libraries necessary are loaded automatically. If the libraries loaded require other libraries, those too are loaded until all dependencies are satisfied.
Shared libraries on UNIX and DLLs on Windows are basically the same, with one major difference: shared libraries on UNIX do not contain any absolute addresses and are therefore relocatable. In other words, a shared library on UNIX does not care where in the process’s address space it is located. On Windows, all references are absolute addresses (locations of global variables, jump addresses) and therefore every DLL has a base address to which it wants to be loaded. When an application is started and a DLL is loaded, the operating system checks if the base address is available and that there is enough space to hold the complete DLL. If it is available the DLL is loaded there, if not the DLL gets loaded to different base address and the operating system then searches the DLL for all absolute references and changes those to the new address.
How does all this memory fit into the process’ address space?
Every platform, even every UNIX flavor, organizes those types of memory requests differently. Following you will find maps of typical address spaces on different platforms. Those maps might change with different operating system releases.
Reserved for kernel
0x08048000 size of executable
On Linux, you can actually see the user address space for a process by using the following command:
08048000-0829f000 r-xp 00000000 00:08 561026 /home/cgeist/dev/linux/scenter
0829f000-082ab000 rw-p 00256000 00:08 561026 /home/cgeist/dev/linux/scenter
082ab000-08330000 rwxp 00000000 00:00 0
40000000-40013000 r-xp 00000000 08:02 4049 /lib/ld-2.1.3.so
40013000-40014000 rw-p 00012000 08:02 4049 /lib/ld-2.1.3.so
40014000-40017000 rw-p 00000000 00:00 0
4001b000-40026000 r-xp 00000000 08:02 4087 /lib/libpthread-0.8.so
40026000-4002d000 rw-p 0000a000 08:02 4087 /lib/libpthread-0.8.so
4002d000-4002e000 rw-p 00000000 00:00 0
4025d000-4025f000 rw-p 00000000 00:00 0
4025f000-40267000 r-xp 00000000 08:02 4079 /lib/libnss_files-2.1.3.so
40267000-40268000 rw-p 00007000 08:02 4079 /lib/libnss_files-2.1.3.so
50000000-51e85000 rwxs 00000000 00:00 0
bfffa000-c0000000 rwxp ffffb000 00:00 0
0xFFBF0000 – stack size
0x00010000 size of executable
You can use the following command on Solaris to print the user address space of a process:
/home/cgeist/scdev/solaris_7: /usr/proc/bin/pmap 27149
27149: scenter -listener
00010000 3256K read/exec /dept/development/cgeist/dev/solaris_7/scenter
0034C000 208K read/write/exec /dept/development/cgeist/dev/solaris_7/scenter
00380000 536K read/write/exec [ heap ]
10000000 31256K read/write/exec/shared [ shmid=0xaf1 ]
FE402000 8K read/write/exec [ anon ]
FE504000 8K read/write/exec [ anon ]
FE606000 8K read/write/exec [ anon ]
. . .
FF384000 8K read/write/exec /usr/lib/libm.so.1
FF390000 8K read/exec /usr/lib/libintl.so.1
FF3A0000 8K read/write/exec [ anon ]
FF3B0000 120K read/exec /usr/lib/ld.so.1
FF3DC000 8K read/write/exec /usr/lib/ld.so.1
FFBEA000 24K read/write/exec [ stack ]
There is no known command to dump out the user address space of a process on AIX.
0x80000000 size of stack
0x40080000 size of executable
There is no known command to dump out the user address space of a process on HP-UX.
0x00400000 size of executable
Above picture does not really make the fragmentation of memory visible which occurs due to the Windows DLL concept. Since there is no rule of any kind where a DLL should be located, every DLL can specify whatever base address it wants. If some DLL wants to be right in the middle of the address space, it divides the address space into two pieces. At that point of time, a large chunk of memory cannot be allocated in one piece anymore. The more DLLs are loaded this way, the less big memory pieces are available.
Most DLLs try to load at a default address of 0x10000000 because that is the default used by Microsoft Development Studio if no other address is specified. There are also some DLLs which want to be loaded at 0x30000000. Windows system DLLs are generally loaded in the 0x70000000 to 0x7FFFFFFF address range
There is no command to dump out the user address space of a process on Windows. However, there is at least one freeware application called ‘Process Explorer’ (available at http://www.sysinternals.com) which is capable to show where the DLLs are located in the user address space of any running process and can highlight those DLLs which had to be relocated.
Why do I sometimes need the shared_memory_address and what does it do for me?
The shared_memory_address parameter is honored on Windows, Linux and Solaris. The first SM process has to do two steps in order to create and use shared memory: first it just creates it and then it maps it into its address space. The first step barely creates the shared memory but it does not make the memory available to the process, yet. The second step makes the shared memory visible within the process’ address space. Every other SM process skips the first step (since the shared memory has already been created) but still has to map the shared memory.
The above address space maps only show the default configuration for every platform. By using the shared_memory_address parameter you can specify where you want shared memory to be located within the address space. There are two reasons why you want to this:
- At the shared memory’s default address something else is already located. Therefore the shared memory cannot be mapped there anymore. This happens on Windows quite often. If the shared memory is larger than about 210MB it would extend from its start address 0x03020000 to an end address of over 0x10000000. At this end address there is usually already a DLL loaded. In such a case you will find the following messages in the SM.log and the process won’t start:
1260 08/16/2002 10:33:47 shmat: MapViewOfFileEx failed
1260 08/16/2002 10:33:47 Error 487 in call MapViewOfFileEx - Attempt to access invalid address.
1260 08/16/2002 10:33:47 sm_init: shmat(shmid = 572) failed, errno=2
1260 08/16/2002 10:33:47 sm_init: unable to attach shared memory
In this case, the shared_memory_address parameter must be used to map the shared memory at an address where this much memory is available in one piece. Typically, shared_memory_address:0x30030000 works, but there is no guarantee. The memory fragmentation due to Window’s DLL handling can make it quite difficult to find a free spot.
The shared memory can be mapped without problems, but the Service Manager process runs out of memory for no apparent reason. This happens to LFSCAN/LFMAP on Solaris. All the information collected during an LFSCAN is stored in dynamically allocated data structures and therefore on the so-called heap.
On Solaris, the heap is located right behind the executable at roughly 0x00380000. Shared memory is mapped by default at 0x10000000. So while LFSCAN is running, the heap increases in size until it hits the 0x10000000 address which is already in use. The allocation cannot be the fulfilled and LFSCAN terminates with an ‘out of memory’ error message.
In this case, the shared_memory_address parameter must be used to map shared memory where it does not restrict the heap too much. Typically, shared_memory_address:0x60000000 should work since it gives the heap about 1.5GB (0x60000000 – 0x00380000) of memory.
What do the following messages mean?
sm_hook: Shared memory was created at base address 0x30000000.
sm_hook: Detaching from 0x20000000 and attaching at 0x30000000.
Service Manager uses pointers (absolute addresses of data) in shared memory to refer to data within shared memory. Therefore, every Service Manager process has to map shared memory to the same base address. If process A maps shared memory to 0x30000000 and process B maps it to 0x20000000 and process A stores the address of a cached record at 0x30001234 within shared memory, the address of 0x30001234 would not be a valid address for process B and an attempt to access this address would cause process B to abend.
Service Manager verifies at which address shared memory was originally created. If process A creates the shared memory and maps it to 0x30000000, every other SM process maps the shared memory to their preferred address (usually the same) and verifies that it is using the same base address. If process B is started with shared_memory_address:0x20000000 it maps shared to that address first, detects that this is wrong address, unmaps (detaches) shared memory again and tries to re-map it at the correct address again. When this happens, the above messages are issued.