STANDBY_FILE_MANAGEMENT parameter is not configured Auto

August 12, 2016 Leave a comment

If you have not set standby_file_management to Auto on the primary  and you add a datafile to the primary  you will see the following in the alert log of the standby

File #9 added to control file as ‘UNNAMED00009’ because
the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
The file should be manually created to continue.
MRP0: Background Media Recovery terminated with error 1274

This will stop you standby database from being in recovery mode until this is fixed , so if you are using OMF do the following to fix it

alter database create datafile ‘/u01/app/oracloud/product/12.1.0/db_1/dbs/UNNAMED00009’ as new

also change the following parameters using dataguard broker

edit database clouddr set property StandbyFileManagement=AUTO;

edit database cloud set property StandbyFileManagement=AUTO;

then restart the dataguard broker

 

Categories: Oracle, Weblogic

JMS Server Could Not Restart Because Of OutOfMemoryError In Recover Messages Process: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery

June 6, 2016 Leave a comment

JMS Server could not restart because of an OutOfMemoryError (OOME) in the recover messages process:

Doc ID 2078641.1 Oracle support

weblogic

CAUSE

The cause of this issue is that the JMS server received more messages than it could process. This caused the OOME initially. The messages were cached in the managed server’s temporary directory. When the JMS server was restarted, it first tries to process the cached messages, but because there are too many messages for the available memory, this fails and the JMS server again crashes. Then the error weblogic.messaging.kernel.KernelException: Unexpected store exception in messaging kernel recovery was thrown.

The following files are cached JMS messages:

In the directory <DOMAIN_HOME>/servers/<server_name>/data/:

File Size File Name
1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
…….
1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

In the directory <DOMAIN_HOME>/servers/<server_name>/tmp/:

File Size File Name
1336938496 ./ESB_JMS_SERVER2000000.TMP
1339035648 ./ESB_JMS_SERVER2000001.TMP
…….
1331695616 ./ESB_JMS_SERVER2000021.TMP
933236736 ./ESB_JMS_SERVER2000022.TMP

SOLUTION

To restart the managed servers, please execute the following steps:

  1. Ensure that you have taken a backup of your system before applying the recommended solution.
  2. Run the following scripts in a TEST environment first:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

  3. Once the scripts complete, confirm that the data is corrected. You can use the following file list:
    File size File Name
    1340084224 ./store/default/_WLS_LOG_SERVER2000000.DAT
    1331695616 ./store/default/_WLS_LOG_SERVER2000001.DAT
    …….
    1331695616 ./store/default/_WLS_LOG_SERVER2000020.DAT
    933236736 ./store/default/_WLS_LOG_SERVER2000021.DAT

    File Size File Name
    1336938496 ./ESB_JMS_SERVER2000000.TMP
    1339035648 ./ESB_JMS_SERVER2000001.TMP
    …….
    1331695616 ./ESB_JMS_SERVER2000021.TMP
    933236736 ./ESB_JMS_SERVER2000022.TMP

  4. To delete all the above *.DAT and *.TMP files:
    cd <DOMAIN_HOME>/servers/<server_name>/data/store/default
    rm -f _WLS_LOG_SERVER2*.DAT

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    rm -f /ESB_JMS_SERVER2*.TMP

  5. Confirm that the files were deleted successfully:
    cd <DOMAIN_HOME>/servers/<server_name>/data/
    find ./ -name “*.DAT” -type f | xargs wc -c

    cd <DOMAIN_HOME>/servers/<server_name>/tmp/
    find ./ -name “*.TMP” -type f | xargs wc -c

    This should return no *DAT or *.TMP files.

  6. Restart the crashed managed server.
Categories: Oracle

Oracle Auditing

June 3, 2016 Leave a comment

Query DBA_AUDIT_TRAIL view

COLUMN username FORMAT A10
COLUMN owner    FORMAT A10
COLUMN obj_name FORMAT A10
COLUMN extended_timestamp FORMAT A35

SELECT username,
extended_timestamp,
owner,
obj_name,
action_name
FROM   dba_audit_trail
WHERE  owner = ‘AUDIT_TEST’
ORDER BY timestamp;

Query XML format OS Audit

COLUMN db_user       FORMAT A10
COLUMN object_schema FORMAT A10
COLUMN object_name   FORMAT A10
COLUMN extended_timestamp FORMAT A35

SELECT db_user,
extended_timestamp,
object_schema,
object_name,
action
FROM   v$xml_audit_trail
WHERE  object_schema = ‘AUDIT_TEST’
ORDER BY extended_timestamp;

Privilege level

Audit records can be found in DBA_PRIV_AUDIT_OPTS.
SQL> select * from DBA_PRIV_AUDIT_OPTS;

Object Level
Audit records can be found in DBA_OBJ_AUDIT_OPTS.
SQL> select * from DBA_OBJ_AUDIT_OPTS;
 

Maintenance

The audit trail must be deleted/archived on a regular basis to prevent the SYS.AUD$ table growing to an unacceptable size.

To delete all audit records from the audit trail:
SQL> DELETE FROM sys.aud$;
 

 

 

 

 

Categories: Oracle

After a Dataguard Broker switchover, the “show configuration” command returns errors ORA-16501, ORA-16625, DGM-17017

June 3, 2016 Leave a comment

A dataguard broker is being used to manage a standby database environment.  A switchover operation is performed, for example:

DGMGRL> switchover to FBOXI4SB
Site “fboxi4sb” was not found
DGMGRL> switchover to ‘FBOXI4SB’
Performing switchover NOW, please wait…exit
Operation requires a connection to instance “FBOXI4SB” on database “FBOXI4SB”
Connecting to instance “FBOXI4SB”…
Connected.
New primary database “FBOXI4SB” is opening…
Operation requires startup of instance “FBOXI4P” on database “FBOXI4P”
Starting instance “FBOXI4P”…
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is “FBOXI4SB”

 

Afterwards the Dataguard command “show configuration” returns errors ORA-16501, ORA-16625

DGMGRL> show configuration


Fast-Start Failover: DISABLED
Configuration Status:
ORA-16501: the Data Guard broker operation failed
ORA-16625: cannot reach database “FBOXI4P”
DGM-17017: unable to determine configuration status

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

 

In SQLPlus we can see that the standby database is running.  Stopping the standby database via sqlplus with “shutdown immediate” and then restarting the standby with srvctl command works around the issue.

SQL> shutdown immediate

 

$ srvctl start database -d FBOXI4P -o mount
Cause

The tnsnames.ora was not present in $GRID_HOME directory, namely in $GRID_HOME/network/admin

Solution

Make sure the tnsnames.ora file is present in $GRID_HOME/network/admin directory;  For example copy the tnsnames.ora file from the $ORACLE_HOME/network/admin to $GRID_HOME/network/admin

 

 

 

 

 

 

Categories: Oracle

restoring OMF and NON OMF datafiles via RMAN

March 30, 2016 Leave a comment

Problem
if when restoring a database which has a mixture of OMF and NON OMF datafiles
you come across this error

channel c1: ORA-19870: error while restoring backup piece bk_6559_1_906629460
ORA-19504: failed to create file "+ASM_DATA/HFMP/DATAFILE/oddevcontent.dbf"
ORA-17502: ksfdcre:3 Failed to create file +ASM_DATA/HFMP/DATAFILE/oddevcontent.dbf
ORA-15001: diskgroup "ASM_DATA" does not exist or is not mounted
ORA-15001: diskgroup "ASM_DATA" does not exist or is not mounted

failover to previous backup

Recovery catalog is down or not connected to catalog, trying to reconnect.
Reconnection with the recovery catalog is successful.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03012: fatal error during compilation of command
RMAN-03028: fatal error code for command restore : 600
RMAN-00600: internal error, arguments [7530] [] [] [] []
[orahfm@rac1 (LMT:hfmp1) HFMP_20160329 ] $
[orahfm@rac1 (LMT:hfmp1) HFMP_20160329 ] $

Solution
The solution is to use “SET NEWNAME” when restoring non-OMF files. If using OMF files
then only the DB_CREATE_FILE_DEST needs to be set. But when it is a mixed env you must
use “SET NEWNAME” for all non-OMF datafiles that will be restored.

Eg
set newname for DATAFILE 7 to ‘+PRE_ASM_DATA/HFMP/DATAFILE/oddevcontent.dbf’;
switch DATAFILE 7 to datafilecopy ‘+PRE_ASM_DATA/HFMP/DATAFILE/oddevcontent.dbf’;

Categories: Oracle, Oracle Rac

EM 12c: Enterprise Manager 12.1.0.4 Cloud Control Agent Removal Using AgentDeinstall.pl on the Target Host and Agent Decommission in the Console

June 3, 2015 Leave a comment
Categories: Oracle

Clean Up Your Old OEM Blackouts

May 29, 2015 Leave a comment
Categories: Oracle
Follow

Get every new post delivered to your Inbox.

Join 152 other followers