Home > Oracle > JMS Server Could Not Restart Because Of OutOfMemoryError In Recover Messages Process: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery

JMS Server Could Not Restart Because Of OutOfMemoryError In Recover Messages Process: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery

JMS Server could not restart because of an OutOfMemoryError (OOME) in the recover messages process:

Doc ID 2078641.1 Oracle support

weblogic

CAUSE

The cause of this issue is that the JMS server received more messages than it could process. This caused the OOME initially. The messages were cached in the managed server’s temporary directory. When the JMS server was restarted, it first tries to process the cached messages, but because there are too many messages for the available memory, this fails and the JMS server again crashes. Then the error weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery was thrown.

The following files are cached JMS messages:

In the directory <DOMAIN_HOME>/servers/<server_name>/data/:

File Size File Name
1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
…….
1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

In the directory <DOMAIN_HOME>/servers/<server_name>/tmp/:

File Size File Name
1336938496 ./ESB_JMS_SERVER2000000.TMP
1339035648 ./ESB_JMS_SERVER2000001.TMP
…….
1331695616 ./ESB_JMS_SERVER2000021.TMP
933236736 ./ESB_JMS_SERVER2000022.TMP

SOLUTION

To restart the managed servers, please execute the following steps:

  1. Ensure that you have taken a backup of your system before applying the recommended solution.
  2. Run the following scripts in a TEST environment first:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

  3. Once the scripts complete, confirm that the data is corrected. You can use the following file list:
    File size File Name
    1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
    1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
    …….
    1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
    933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

    File Size File Name
    1336938496 ./ESB_JMS_SERVER2000000.TMP
    1339035648 ./ESB_JMS_SERVER2000001.TMP
    …….
    1331695616 ./ESB_JMS_SERVER2000021.TMP
    933236736 ./ESB_JMS_SERVER2000022.TMP

  4. To delete all the above *.DAT and *.TMP files:
    cd <DOMAIN_HOME>/servers/<server_name>/data/store/default
    rm -f _WLS_LOG_SERVER2*.DAT

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    rm -f /ESB_JMS_SERVER2*.TMP

  5. Confirm that the files were deleted successfully:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

    This should return no *DAT or *.TMP files.

  6. Restart the crashed managed server.
Advertisements
Categories: Oracle
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: