Problem
I’m having trouble with a Java program that runs on Linux.
Using the toppers application, I can see that the program is given 240 MB of virtual Memory when I launch it with the default maximum heap size (64 MB). This causes problems with some other applications on the computer, which is resource-constrained.
As far as I understand it, the reserved virtual memory will not be used in any case, because once we approach the heap limit, an OutOfMemoryError is thrown. Under Windows, I ran the identical application and noticed that the Virtual Memory and Heap sizes are similar.
Is it possible to customize the Virtual Memory used by a Java process under Linux?
Edit 1: The issue isn’t with the Heap. The issue is that even if I set a Heap size of 128 MB, Linux still allocates 210 MB of Virtual Memory, which is never needed. **
Edit 2: The command ulimit -v can be used to limit the amount of virtual memory available. The application will not execute if the size set is less than 204 MB, even though it only requires 64 MB. So I’m curious as to why Java uses so much virtual memory. Is it possible to modify this?
Edit 3: There are a number of additional apps operating in the embedded system. In addition, the system has a virtual memory limit (from comments, important detail).
Asked by Mario Ortegón
Solution #1
This has been a long-standing gripe about Java, however it’s mostly useless and usually stems from misinterpreting data. The standard phrase goes something like this: “It takes ten gigabytes to run Hello World in Java! What is the point of that?” So, here’s how to make Hello World on a 64-bit JVM claim to take up over 4 gigabytes… at least by one metric.
java -Xms1024m -Xmx4096m com.example.Hello
The top command in Linux displays multiple distinct memory values. What it says about the Hello World example is as follows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2120 kgregory 20 0 4373m 15m 7152 S 0 0.2 0:00.10 java
For Windows Task Manager, the situation is a little more complicated. The “Memory Usage” and “Virtual Memory Size” columns exist in Windows XP, although the official literature is vague on what they imply. More columns are included in Windows Vista and Windows 7, and they’re truly documented. The “Working Set” measurement is the most helpful of these; on Linux, it roughly equates to the sum of RES and SHR.
The total amount of virtual memory consumed by a process is equal to the total amount of memory in the process memory map. This comprises not only data (for example, the Java heap), but also all shared libraries and memory-mapped files that the program uses. To see everything mapped into the process space on Linux, use the pmap command (from here on out, I’ll solely refer to Linux because that’s what I use; I’m sure there are analogous tools for Windows). Here’s a sample of the “Hello World” program’s memory map; the complete memory map is over 100 lines long, and a thousand-line list is not uncommon.
0000000040000000 36K r-x-- /usr/local/java/jdk-1.6-x64/bin/java
0000000040108000 8K rwx-- /usr/local/java/jdk-1.6-x64/bin/java
0000000040eba000 676K rwx-- [ anon ]
00000006fae00000 21248K rwx-- [ anon ]
00000006fc2c0000 62720K rwx-- [ anon ]
0000000700000000 699072K rwx-- [ anon ]
000000072aab0000 2097152K rwx-- [ anon ]
00000007aaab0000 349504K rwx-- [ anon ]
00000007c0000000 1048576K rwx-- [ anon ]
...
00007fa1ed00d000 1652K r-xs- /usr/local/java/jdk-1.6-x64/jre/lib/rt.jar
...
00007fa1ed1d3000 1024K rwx-- [ anon ]
00007fa1ed2d3000 4K ----- [ anon ]
00007fa1ed2d4000 1024K rwx-- [ anon ]
00007fa1ed3d4000 4K ----- [ anon ]
...
00007fa1f20d3000 164K r-x-- /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
00007fa1f20fc000 1020K ----- /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
00007fa1f21fb000 28K rwx-- /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
...
00007fa1f34aa000 1576K r-x-- /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3634000 2044K ----- /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3833000 16K r-x-- /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3837000 4K rwx-- /lib/x86_64-linux-gnu/libc-2.13.so
...
A simple explanation of the format: each row begins with the segment’s virtual memory address. The segment size, permissions, and source of the segment are then displayed. This last item is either a file or “anon,” which denotes a mmap-allocated memory block.
We’ll start at the top and work our way down.
Each shared library comprises at least two segments: a read-only segment containing the library code, and a read-write segment containing the library’s global per-process data (I’m not sure what the segment with no rights is; I’ve only seen it on x64 Linux). All processes that utilize the library can share the read-only section of the library; for example, libc contains 1.5M of virtual memory space that can be shared.
There’s a lot of information in the virtual memory map. It’s read-only in some places, shared in others, and assigned but never touched in others (eg, almost all of the 4Gb of heap in this example). However, because the operating system is intelligent enough to just load what it requires, the quantity of virtual memory is mainly immaterial.
If you’re using a 32-bit operating system, virtual memory size is crucial since you can only assign 2GB (or, in some circumstances, 3GB) of process address space. In that situation, you’re working with a limited resource and may need to make sacrifices, such as reducing heap size to memory-map a huge file or create a large number of threads.
However, considering the widespread use of 64-bit processors, I don’t think it will be long before Virtual Memory Size becomes obsolete.
The size of the Resident Set is the amount of virtual memory space that is actually in RAM. If your RSS takes up a substantial amount of your overall physical memory, you should be concerned. If your RSS grows to take up all your physical memory, and your system starts swapping, it’s well past time to start worrying.
However, RSS can be deceiving, especially on a low-powered system. The operating system doesn’t put much effort into recuperating the pages that a process has utilized. There’s little benefit to be gained by doing so, and the potential for an expensive page fault if the process touches the page in the future. As a result, the RSS statistic may include lots of pages that aren’t in active use.
Don’t be too bothered with the various memory statistics unless you’re swapping. With the proviso that a rapidly increasing RSS could signal a memory leak.
It’s significantly more necessary to pay attention to what’s going on in the heap in a Java program. The total quantity of space used is significant, and there are several things you can do to cut down on it. The amount of time you spend collecting rubbish and which portions of the heap are collected are more essential.
Accessing the disk (i.e., a database) is costly, whereas memory is inexpensive. If you can swap one for the other, go ahead and do so.
Answered by kdgregory
Solution #2
A known issue exists with Java and glibc >= 2.10 (which includes Ubuntu 10.04 and RHEL 6).
The cure is to set the following environment variable:
export MALLOC_ARENA_MAX=4
If you are running Tomcat, you can add this to TOMCAT_HOME/bin/setenv.sh file.
Add this to the Dockerfile for Docker.
ENV MALLOC_ARENA_MAX=4
Setting MALLOC ARENA MAX is described in an IBM paper. https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux glibc 2 10 rhel 6 malloc may show excessive virtual memory usage?lang=en
According to this blog post
JDK-8193521 “glibc wastes memory with default settings” is another open JDK problem.
For further information, look up MALLOC ARENA MAX on Google or SO.
To optimize for reduced fragmentation of allocated memory, you might wish to modify other malloc parameters as well:
# tune glibc memory allocation, optimize for low fragmentation
# limit the number of arenas
export MALLOC_ARENA_MAX=2
# disable dynamic mmap threshold, see M_MMAP_THRESHOLD in "man mallopt"
export MALLOC_MMAP_THRESHOLD_=131072
export MALLOC_TRIM_THRESHOLD_=131072
export MALLOC_TOP_PAD_=131072
export MALLOC_MMAP_MAX_=65536
Answered by Lari Hotari
Solution #3
The quantity of RAM assigned to the Java process is roughly in line with my expectations. I’ve seen similar issues with Java on embedded/low-memory computers. Any programme that is run with arbitrary VM restrictions or on systems with insufficient swap tends to fail. Many recent apps don’t appear to be designed for operation on computers with restricted resources.
You have a couple more choices for reducing the memory footprint of your JVM. This may result in a smaller virtual memory footprint:
You should also adjust your -Xmx (max heap size) to a figure that is as close as feasible to your application’s real peak memory usage. I believe the JVM’s default behavior is still to double the heap size every time it extends it to its maximum capacity. If you start with a 32MB heap and your app grows to 65MB, the heap will expand from 32MB to 64MB to 128MB.
You may also try this to make the VM less aggressive when it comes to heap growth:
Also, I recall that the amount of native libraries loaded had a significant impact on the smallest footprint when I experimented with this a few years ago. Java.net is being loaded. If my memory serves me correctly (which it doesn’t), Socket contributed more than 15M.
Answered by James Schek
Solution #4
HotSpot requires a lot of memory from the Sun JVM, which is stored in shared memory in the runtime libraries.
If memory is an issue consider using another JVM suitable for embedding. IBM has j9, and there is the Open Source “jamvm” which uses GNU classpath libraries. Also Sun has the Squeak JVM running on the SunSPOTS so there are alternatives.
Answered by Thorbjørn Ravn Andersen
Solution #5
Playing with the -XX:MaxHeapFreeRatio setting may be one technique to reduce the heap size of a system with restricted resources. The maximum percentage of the heap that is free before the GC decreases it is commonly set to 70. Setting it to a lower value, and you will see in eg the jvisualvm profiler that a smaller heap sice is usually used for your program.
EDIT: If you want to specify a tiny value for -XX:MaxHeapFreeRatio, you must also set -XX:MinHeapFreeRatio, for example.
java -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=25 HelloWorld
EDIT2: Added an example of a real programme that starts and performs the same process, with default parameters and parameters of 10 and 25. Although java in theory should spend more time to enlarge the heap in the latter example, I didn’t detect any major speed differences.
Finally, the maximum heap is 905, while the used heap is 378.
At the end, max heap is 722, used heap is 378
This has some impact because our program runs on a remote desktop server and can be used by multiple users at the same time.
Answered by runholen
Post is based on https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used