01 February 2016

JAVA_OPTIONS

different java options available to tweak your application or server to enable/disable any features which
one is not aware of, here I tried to collect and list for everyone


 # to set time zone at jvm level, define in env file
 $ -Duser.timezone=EST

 # to start WLS managed server with local ldap
 $ -Dweblogic.security.MSILocalLDAPOnly=true

 # to enable GC logs in WLS
 $ -Xverboselog:C:/Oracle/../base_domain/gclogs/gc.log

 # to enable heap dump at OOM condition
 $ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:/Oracle/../base_domain/heapdumps"

 # SSL/TLSv1 related JAVA_OPTIONS

 # to enable JSSE
 -Dweblogic.ssl.JSSEEnabled=true"

 # to enable SSL debug
 $ -Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true

 # to disable SSL in weblogic and enable TLS
 $ -Dweblogic.security.SSL.protocolVersion=TLS1

 # to allow all TLS protocols for the most common SSLSocket or SSLSocketFactory classes
 $ -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2

 # Applications using the HttpsClient,
   HttpsURLConnection classes can use the https.protocols system property
 $ -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

 # to allow client/server to support a minimum protocol
 $ -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.1


19 January 2016

What is Native OOM in Java Heap?

Native OutOfMemory is a scenario
Native memory is a Process Area

---------Native OutOfMemory---------

Native OutOfMemory is a scenario when the JVM is not able to allocate the required Native Libraries and JNI Codes in the memory.
Native Memory is an area that is usually used by the JVM for it’s internal operations and to execute the JNI codes. The JVM Uses Native Memory for Code Optimization and for loading the classes and libraries along with the intermediate code generation.
The Size of the Native Memory depends on the Architecture of the Operating System and the amount of memory that is already committed to the Java Heap. Native memory is a Process Area where the JNI codes get loaded or JVM Libraries get loaded or the native Performance packs and the Proxy Modules get loaded…
Native OutOfMemory can happen due to the following main reasons:

Point-1) Setting very small StackSize (-Xss). StackSize is a memory area that is allocated to individual threads where they can place their thread local objects/variables.

Point-2) Usually it may be seen because of Tuxedos incorrect setting. WebLogic Tuxedo Connectors allows the interoperability between the Java Applications deployed on WebLogic Server and the Native Services deployed on Tuxedo Servers. Because Tuxedos uses JNI code intensively.

Point-3) Less RAM or Swap Space.

Point-4) Usually it may occur if our Application is using a very large number of JSPs in our application. The JSPs need to be converted into the Java Code and then need to be compiled. Which requires DTD and Custom Tag Library resolution as well. Which usually consumes more native memory.

---------What to do in case of Native OutOfMemory---------

Point-1) Usually Native OutOfMemory causes Server/JVM Crash. So it is always recommended to apply the following JAVA_OPTIONS flags in the Server Start Script to instruct the JVM to generate the HeapDump ”-XX:+HeapDumpOnOutOfMemoryError“
By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the “-XX:HeapDumpPath=C:/someLocation/“
Note: Above Flags are also suitable to collect HeapDump in the case of JavaHeap OutOfMemory as well. But these flags never guarantee that the JVM will always generate the Heap Dump in case of any OutOfMemory Situation.

Point-2) Usually in case of Native OutOfMemory a “hs_err_pid.log” file is created in case of Sun JDK and “xxxx.dump” file is created in case of JRockit JDK. These log files are usually Text Files and tells about the Libraries which caused the Crash. These files need to be collected and analyzed to find out the root cause.

Point-3) Make Sure that the -XX:MaxHeapSize is not set to a Very Large Space…because it will cause a very less Native Space allocation. Because as soon as we increase the HeapSize, the Native Area decreases. 

Please see the Post:http://middlewaremagic.com/weblogic/?p=4456

Point-4) Keep Monitoring the process’s memory using the Unix utility ‘ps’ like following:
ps -p -o vsz
Here you need to pass the WebLogic Server’s PID (Process ID) to get it’s Threading Details with respect to the Virtual Memory Space.

Point-5) If the Heap Usages is less Or if you see that Your Application usages less Heap Memory then it is always better to reduls the MaxHeapSize so that the Native Area will automatically gets increased.

Point-6) Sometimes the JVMs code optimization causes Native OutOfMemory or the Crash…So in this case we can disable the Code Optimization feature of JVM.
(Note: disabling the Code Optimization of JVM will decrease the Performance of JVM)
For JRockit JVM Code Optimization can be disabled using JAVA_OPTION -Xnoopt
For Sun JDK Code Optimization can be disabled using JAVA_OPTION -Xint

21 July 2015

Study on JAVA_HEAP


1). Just for Example we can see that the process Size is 2048 MB (2GB)

2). The Java Heap Size is 1024MB (means 1GB) -Xmx1024m

3). Native Space = ( ProcessSize – MaxHeapSize – MaxPermSize) It means around 768 MB of Native Space.

4). MaxPermSpace is around -XX:MaxPermSize=256m

5). Young Generation Space is around 40% of the Maximum Java Heap.

What Are these Different Parts?

Eden Space
Eden Space is a Part of Java Heap where the JVM initially creates any objects, where most objects die
and quickly are cleaned up by the minor Garbage Collectors (Note: Full Garbage Collection is different
from Minor Garbage Collection). Usually, any new objects created inside a Java Method go into Eden
space and the objects space is reclaimed once the method execution completes. Whereas the Instance
Variables of a Class usually live longer until the Object-based on that class gets destroyed. When Eden
fills up it causes a minor collection, in which some surviving objects are moved to an older generation.

Survivor Spaces
Eden Space has two Survivor spaces. One survivor space is empty at any given time. This Survivor
Spaces serve as the destination of the next copying collection of any living objects in eden and the other
survivor space.
The parameter SurvivorRatio can be used to tune the size of the survivor spaces.
-XX:SurvivorRatio=6 sets the ratio between each survivor space and eden to be 1:6
If survivor spaces are too small copying collection overflows directly into the tenured generation.

Young Generation: (-XX:MaxNewSize)
Till JDK1.3 and 1.4 we used to set the Young Generation Size using -XX:MaxNewSize. But from
JDK1.4 onwards we set the YoungGeneration size using (-Xmn) JVM option.
Young Generation size is controlled by NewRatio. It means setting -XX:NewRatio=3 means that the ratio
between the Old Generation and the Young Generation is 1:3
.
Similarly -XX:NewRatio=8 means that 8:1 ratio of the tenured and young generation.

NewRatio
NewRatio is actually the ratio between the (YoungGenaration/Old Generations) has default
values of 2 on Sparc , 12 on client Intel, and 8 everywhere else.

NOTE: After JDK 1.4 The Young Generation Size can be set using (-Xmn) as well.

Virtual Space-1: (MaxNewSize – NewSize)
The First Virtual Space is actually shows the difference between the -XX:NewSize and -
XX:MaxNewSize. Or we can say that it is basically a difference between the Initial Young Size and the
Maximum Young Size.

Java Heap Area: (-Xmx and -Xms)
Java Heap is a Memory area inside the Java Process which holds the java objects. Java
Heap is a combination of Young Generation Heap and Old Generation Heap. We can set
the Initial Java Heap Size using -Xms JVM parameter similarly if we want to set the
Maximum Heap Size then we can use -Xmx JVM parameter to define it.
Example:
-Xmx1024m —> Means Setting the Maximum limit of Heap as 1 GB
-Xms512m —> Means setting Java Heap Initial Size as 512m
.
NOTE-1): It is always recommended to set the Initial and the Maximum Heap size values as same for
better performance.
NOTE-2): The Theoretical limitation of Maximum Heap size for a 32 bit JVM is upto 4GB. Because of
the Memory Fragmentation, Kernel Space Addressing, Swap memory usages and the Virtual Machine
Overheads are some factors JVM does not allow us to allocate whole 4GB memory for Heap in a 32 bit
JVM. So usually on 32-bit Windows Operating Systems the Maximum can be from 1.4 GB to 1.6 GB.
.
If we want a Larger memory allocation according to our application requirement then we must choose the
64-bit operating systems with 64 bit JVM. 64-bit JVM provides us a larger address space. So we can have
much larger Java Heap with the increased number of Threads allocation area. Based on the Nature of
your Operating system in a 64 bit JVM you can even set the Maximum Heap size upto 32GB.
Example: -Xms32g -Xmx32g -Xmn4g

Virtual Space-2: (MaxHeapSize – InitialHeapSize)
The Second Virtual Space is actually the Difference between the Maximum Heap size (-Xmx)and the
Initial Heap Size(-Xms). This is called as virtual space because initially the JVM will allocate the Initial
Heap Size and then according to the requirement the Heap size can grow till the MaxHeapSize.

.
PermGen Space: (-XX:MaxPermSize)
PermGen is a non-heap memory area where the Class Loading happens and the JVM allocates spaces for
classes, class meta data, java methods and the reference Objects here. The PermGen is independent from
the Heap Area. It can be resized according to the requirement using -XX:MaxPermSize and -
XX:PermSize JVM Options. The Garbage collection happens in this area of JVM Memory as well. The
Garbage collection in this area is called as “Class GC”. We can disable the Class Garbage Collection
using the JVM Option -noclassgc. if ”-noclassgc” Java Option is added while starting the Server. In that
case the Classes instances which are not required will not be Garbage collected.

Native Area
Native Memory is an area which is usually used by the JVM for its internal operations and to execute the
JNI codes. The JVM Uses Native Memory for Code Optimization and for loading the classes and libraries
along with the intermediate code generation.
The Size of the Native Memory depends on the Architecture of the Operating System and the amount of
memory which is already committed to the Java Heap. Native memory is a Process Area where the JNI
codes gets loaded or JVM Libraries gets loaded or the native Performance packs and the Proxy Modules
gets loaded.
There is no JVM Option available to size the Native Area. But we can calculate it approximately using
the following formula:

NativeMemory = (ProcessSize – MaxHeapSize – MaxPermSize)

HEAP DUMP

How do you analyze heap dump?

$ /apps/java/jdk1.6_25-64/bin/jmap -dump:format=b, file=/apps/heapJmap.bin (PID)

it will say heap dumped into the location

$ /jhat -J-mx1024m /apps/heapJmap.bin

it will start the HTTP client stating - started HTTP server on port 7000 server is ready
then hit the URL http://localhost:7000
NOTE: due to JDK-6614052 : jhat fails to read heap dump >2GB

--

25 November 2014

DBPING - a java-weblogic utility

There are ways to test DB connection from an application server, one of the most successful ways is to test as a java utility.

For running such utilities you need the proper environment to be set, You need to source the ssetWLSEnv.sh script in order to execute the script in the context of the running shell, and have the actual environment of the shell be set.

You have to execute as ". setWLSEnv.sh" and then you can check that the CLASSPATH is set with the "env" command.

and then you can begin dbping utility to check DB connectivity from the weblogic server

set domain environment as explained above & then initialize utility as:

 
 Usage: 
 java utils.dbping DB2B [-d dynamicSections] USER PASS HOST:PORT/DBNAME
 or     java utils.dbping DERBY        USER PASS HOST:PORT/DBNAME
 or     java utils.dbping JCONN2       USER PASS HOST:PORT/DBNAME
 or     java utils.dbping JCONN3       USER PASS HOST:PORT/DBNAME
 or     java utils.dbping JCONNECT     USER PASS HOST:PORT/DBNAME
 or     java utils.dbping INFORMIXB    USER PASS HOST:PORT/DBNAME/INFORMIXSERVER
 or     java utils.dbping MSSQLSERVERB USER PASS HOST:PORT/[DBNAME]
 or     java utils.dbping MYSQL        USER PASS [HOST][:PORT]/[DBNAME]
 or     java utils.dbping ORACLEB      USER PASS HOST:PORT/DBNAME
 or     java utils.dbping ORACLE_THIN  USER PASS HOST:PORT:DBNAME
 or     java utils.dbping POINTBASE    USER PASS HOST[:PORT]/DBNAME
 or     java utils.dbping SYBASEB      USER PASS HOST:PORT/DBNAME
 

--

18 June 2014

JAVA FLIGHT RECORDER

To run JAVA FLIGHT RECORDER for any java process do the following

# set Domain Env
$ /bin/./setDomainEnv.sh $ jcmd PID JFR.start duration=60s filename=myrecording.jfr

ref: how to use JFR

remember after running the above steps it will unlock commercial features of weblogic, will not allow to run any new java instance on the same host, and gives a message like



<"ResourceManagement" is not enabled in this JVM. Enable "ResourceManagement" to use the WebLogic Server "Resource Consumption Management" feature. To enable "ResourceManagement", you must specify the following JVM options in the WebLogic Server instance in which the JVM runs: -XX:+UnlockCommercialFeatures -XX:+ResourceManagement.> 


so you need to disable this commercial feature by restarting the JVM for which you have used the above tool.

jcmd unlock's commercial features of product at first use.

--

15 June 2014

Threads

What is Thread?
A thread is a single sequential flow of control within a program.

What is Stuck Thread?
A Stuck Thread is a thread which is processing a request for more than maximum time that you
configured in admin console.

How to deal with Stuck Thread?
Take multiple thread dumps immediately.
Review thread dumps or from console (managed server > monitoring > threads).
See how many threads got stuck?
If the stuck thread count is increasing or constant?
If constant then if got stuck on same area (application code etc ) or at different places ?
If getting increase then there would be some serious problem and you have to do a quick health check of you application server, database and other integrated technologies wherever your application reaching like ldap server for authentication, some other API’s or web services etc, and in parallel review thread dumps for stuck threads and share same with your developers to analyze quickly.
If you have one, two or few constant stuck  threads and it’s not increasing then you can monitor it for some more time to check if they get clear or not, if not then to clear them you have only option to restart your managed server(s), and its better to restart and clear them before they make further any impact.

What is Hogging Thread? 
 A hogging thread is a thread which is taking more than usual time to complete the request and can be declared as Stuck .

How Weblogic determine a thread to declare as hogging?
A thread declared as Stuck if it runs over 600 secs (default configuration which you can increase or decrease from admin console).
There is an internal WebLogic polar which runs every 2 secs (by default 2 secs and can be alter)
It checks for the number of requests completed in last two minutes
Then it check how much times each took to complete
Then it takes the average time of all completed request (completed in last 2 sec)
Then multiply average time with 7, and the value came consider as “usual time to complete the request”
Now weblogic check each current executed thread in last 2 secs and compare with above average time, if for any of the thread it’s above this value then that thread will declare as Hogged thread.
For example –
At a particular moment, total number of completed requests in last two seconds – 4
Total time took by all 4 requests – 16 secs
Req1 took – 5 secs, Req2 took – 3 secs, Req3 took – 7 secs, Req4 took – 1 sec
Average time = 16/4 = 4 secs
7*4 = 28 secs
Now weblogic check all executed threads to see which taking more than 28 secs, if any then that thread(s) declared as Hogged Thread.
Only the thing you can change with respect to hogging threads configuration is Polar time (Stuck Thread Timer Interval parameter)which is 2 secs by default. You can change this polar value to some different value like 4 secs if you want polar to run in every 4 secs instead of 2 secs.