Archive

Archive for June, 2016

JMS Server Could Not Restart Because Of OutOfMemoryError In Recover Messages Process: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery

June 6, 2016 Leave a comment

JMS Server could not restart because of an OutOfMemoryError (OOME) in the recover messages process:

Doc ID 2078641.1 Oracle support

weblogic

CAUSE

The cause of this issue is that the JMS server received more messages than it could process. This caused the OOME initially. The messages were cached in the managed server’s temporary directory. When the JMS server was restarted, it first tries to process the cached messages, but because there are too many messages for the available memory, this fails and the JMS server again crashes. Then the error weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery was thrown.

The following files are cached JMS messages:

In the directory <DOMAIN_HOME>/servers/<server_name>/data/:

File Size File Name
1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
…….
1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

In the directory <DOMAIN_HOME>/servers/<server_name>/tmp/:

File Size File Name
1336938496 ./ESB_JMS_SERVER2000000.TMP
1339035648 ./ESB_JMS_SERVER2000001.TMP
…….
1331695616 ./ESB_JMS_SERVER2000021.TMP
933236736 ./ESB_JMS_SERVER2000022.TMP

SOLUTION

To restart the managed servers, please execute the following steps:

  1. Ensure that you have taken a backup of your system before applying the recommended solution.
  2. Run the following scripts in a TEST environment first:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

  3. Once the scripts complete, confirm that the data is corrected. You can use the following file list:
    File size File Name
    1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
    1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
    …….
    1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
    933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

    File Size File Name
    1336938496 ./ESB_JMS_SERVER2000000.TMP
    1339035648 ./ESB_JMS_SERVER2000001.TMP
    …….
    1331695616 ./ESB_JMS_SERVER2000021.TMP
    933236736 ./ESB_JMS_SERVER2000022.TMP

  4. To delete all the above *.DAT and *.TMP files:
    cd <DOMAIN_HOME>/servers/<server_name>/data/store/default
    rm -f _WLS_LOG_SERVER2*.DAT

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    rm -f /ESB_JMS_SERVER2*.TMP

  5. Confirm that the files were deleted successfully:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

    This should return no *DAT or *.TMP files.

  6. Restart the crashed managed server.
Advertisements
Categories: Oracle

Oracle Auditing

June 3, 2016 Leave a comment

Query DBA_AUDIT_TRAIL view

COLUMN username FORMAT A10
COLUMN owner    FORMAT A10
COLUMN obj_name FORMAT A10
COLUMN extended_timestamp FORMAT A35

SELECT username,
extended_timestamp,
owner,
obj_name,
action_name
FROM   dba_audit_trail
WHERE  owner = ‘AUDIT_TEST’
ORDER BY timestamp;

Query XML format OS Audit

COLUMN db_user       FORMAT A10
COLUMN object_schema FORMAT A10
COLUMN object_name   FORMAT A10
COLUMN extended_timestamp FORMAT A35

SELECT db_user,
extended_timestamp,
object_schema,
object_name,
action
FROM   v$xml_audit_trail
WHERE  object_schema = ‘AUDIT_TEST’
ORDER BY extended_timestamp;

Privilege level

Audit records can be found in DBA_PRIV_AUDIT_OPTS.
SQL> select * from DBA_PRIV_AUDIT_OPTS;

Object Level
Audit records can be found in DBA_OBJ_AUDIT_OPTS.
SQL> select * from DBA_OBJ_AUDIT_OPTS;
 

Maintenance

The audit trail must be deleted/archived on a regular basis to prevent the SYS.AUD$ table growing to an unacceptable size.

To delete all audit records from the audit trail:
SQL> DELETE FROM sys.aud$;
 

 

 

 

 

Categories: Oracle

After a Dataguard Broker switchover, the “show configuration” command returns errors ORA-16501, ORA-16625, DGM-17017

June 3, 2016 Leave a comment

A dataguard broker is being used to manage a standby database environment.  A switchover operation is performed, for example:

DGMGRL> switchover to FBOXI4SB
Site “fboxi4sb” was not found
DGMGRL> switchover to ‘FBOXI4SB’
Performing switchover NOW, please wait…exit
Operation requires a connection to instance “FBOXI4SB” on database “FBOXI4SB”
Connecting to instance “FBOXI4SB”…
Connected.
New primary database “FBOXI4SB” is opening…
Operation requires startup of instance “FBOXI4P” on database “FBOXI4P”
Starting instance “FBOXI4P”…
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is “FBOXI4SB”

 

Afterwards the Dataguard command “show configuration” returns errors ORA-16501, ORA-16625

DGMGRL> show configuration


Fast-Start Failover: DISABLED
Configuration Status:
ORA-16501: the Data Guard broker operation failed
ORA-16625: cannot reach database “FBOXI4P”
DGM-17017: unable to determine configuration status

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

 

In SQLPlus we can see that the standby database is running.  Stopping the standby database via sqlplus with “shutdown immediate” and then restarting the standby with srvctl command works around the issue.

SQL> shutdown immediate

 

$ srvctl start database -d FBOXI4P -o mount
Cause

The tnsnames.ora was not present in $GRID_HOME directory, namely in $GRID_HOME/network/admin

Solution

Make sure the tnsnames.ora file is present in $GRID_HOME/network/admin directory;  For example copy the tnsnames.ora file from the $ORACLE_HOME/network/admin to $GRID_HOME/network/admin

 

 

 

 

 

 

Categories: Oracle