Overview of Oracle Local Registry (OLR) in RAC 11gR2

In Oracle Clusterware 11g Release 2, each node in a cluster has a local registry for node specific resources, called an Oracle Local Registry (OLR), which is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional. The OLR provides various Oracle Clusterware processes with access to key configuration information even when Oracle Clusterware is not running on the node. One of its functions is to facilitate the Oracle Clusterware startup process in situations where the ASM stores the OCR and voting disks. During the startup process, the OLR is referenced to determine the exact location of the voting disks. This enables the node to join the cluster. After this initial phase, ASM is started. After ASM is started, processes that require the full OCR can start and the clusterware startup process completes.

By default, OLR is located at grid_home/cdata/hostname.olr.

The OLR contains the information about ORA_CRS_HOME, localhost version, active version, GPnP details, OCR latest backup time and location, information about OCR daily, weekly backup location
and node name etc.

You can manage the OLR by using the ocrcheck, ocrdump, and ocrconfig utilities with the -local option.

To see the location of the OLR, use the ocrcheck utility:
$ ocrcheck -config -local

You can check the status of the OLR as follows:
# ocrcheck -local

You can display the content of OLR to the text terminal that initiated the program using the OCRDUMP utility, as follows:
$ ocrdump -local -stdout

You can perform administrative tasks on OLR using the OCRCONFIG utility. To export OLR to a file:
# ocrconfig –local –export file_name

To import a specified file to OLR:
# ocrconfig –local –import file_name

To modify the OLR file on the local node:
# ocrconfig –local –repair olr file_name
Note: The olr keyword used with the -repair option is valid only when -local is used.

Thats it. Hope this will help you.

Thanks,
Chowdari

Adding disk to ASM disk group in oracle 11g

Following steps will help you to add disks to oracle ASM disk group.

Below query will identify disk groups information to add disks

SQL> SET LINESIZE  145
SQL> SET PAGESIZE  9999
SQL> SET VERIFY    off
SQL> COLUMN group_name             FORMAT a20           HEAD 'Disk Group|Name'
SQL> COLUMN sector_size            FORMAT 99,999        HEAD 'Sector|Size'
SQL> COLUMN block_size             FORMAT 99,999        HEAD 'Block|Size'
SQL> COLUMN allocation_unit_size   FORMAT 999,999,999   HEAD 'Allocation|Unit Size'
SQL> COLUMN state                  FORMAT a11           HEAD 'State'
SQL> COLUMN type                   FORMAT a6            HEAD 'Type'
SQL> COLUMN total_mb               FORMAT 999,999,999   HEAD 'Total Size (MB)'
SQL> COLUMN used_mb                FORMAT 999,999,999   HEAD 'Used Size (MB)'
SQL> COLUMN pct_used               FORMAT 999.99        HEAD 'Pct. Used'
SELECT
SQL>   2      name                                     group_name
  , sector_size                              sector_size
  3    4    , block_size                               block_size
  5    , allocation_unit_size                     allocation_unit_size
  6    , state                                    state
  7    , type                                     type
  8    , total_mb                                 total_mb
  9    , (total_mb - free_mb)                     used_mb
  , ROUND((1- (free_mb / total_mb))*100, 2)  pct_used
 10   11  FROM
 12      v$asm_diskgroup
 13  ORDER BY
 14      name
 15  /

Disk Group            Sector   Block   Allocation
Name                    Size    Size    Unit Size State       Type   Total Size (MB) Used Size (MB) Pct. Used
-------------------- ------- ------- ------------ ----------- ------ --------------- -------------- ---------
MKRSTDATA                512   4,096    4,194,304 MOUNTED     EXTERN         819,168        726,848     88.73
MKRSTFRA                 512   4,096    4,194,304 MOUNTED     EXTERN         819,168            192       .02
MKRSTREDO                512   4,096    4,194,304 MOUNTED     EXTERN          20,464          4,344     21.23
OCRVOTEDG                512   4,096    4,194,304 MOUNTED     NORMAL          15,348          1,040      6.78

Find ASM disk group number with using below query

SQL> select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from  v$asm_diskgroup where name='MKRSTDATA';

GROUP_NUMBER NAME                           State       Total Size (MB)    FREE_MB
------------ ------------------------------ ----------- --------------- ----------
           2 MKRSTDATA                      MOUNTED             819,168      92320

Following query will help to find existing disk names and disk sizes for MKRSTDATA disk group

SQL> set lines 200 pages 200
col path for a30
select GROUP_NUMBER,HEADER_STATUS,STATE,TOTAL_MB,FREE_MB,PATH from v$asm_disk where GROUP_NUMBER=2;SQL> SQL>

GROUP_NUMBER HEADER_STATU State       Total Size (MB)    FREE_MB PATH
------------ ------------ ----------- --------------- ---------- ------------------------------
           2 MEMBER       NORMAL              102,396      11556 /dev/mapper/t2_fra07p1
           2 MEMBER       NORMAL              102,396      11544 /dev/mapper/t1_data05p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data06p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data01p1
           2 MEMBER       NORMAL              102,396      11548 /dev/mapper/t1_data07p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data03p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data02p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data04p1

8 rows selected.

Find available disks information

ASMCMD [+] > lsdsk
Path
/dev/mapper/t1_data01p1
/dev/mapper/t1_data02p1
/dev/mapper/t1_data03p1
/dev/mapper/t1_data04p1
/dev/mapper/t1_data05p1
/dev/mapper/t1_data06p1
/dev/mapper/t1_data07p1
/dev/mapper/t1_redo01p1
/dev/mapper/t1_redo02p1
/dev/mapper/t1_redo03p1
/dev/mapper/t1_redo04p1
/dev/mapper/t1_vote1p1
/dev/mapper/t1_vote2p1
/dev/mapper/t1_vote3p1
/dev/mapper/t2_fra01p1
/dev/mapper/t2_fra02p1
/dev/mapper/t2_fra03p1
/dev/mapper/t2_fra04p1
/dev/mapper/t2_fra05p1
/dev/mapper/t2_fra06p1
/dev/mapper/t2_fra07p1
ASMCMD [+] >

So here will add disks on MKRSTDATA disk group and group number 2. The size of the each disks are 100GB. So here I am adding additional 500GB to MKRSTDATA disk group. Request unix team to provide 5*100GB disks. After getting disks from Unix team verify CANDIDATE disks information.

Following query will help us to find CANDIDATE disks,

SQL> set lines 200 pages 200
col path for a30
select GROUP_NUMBER,HEADER_STATUS,STATE,TOTAL_MB,FREE_MB,PATH from v$asm_disk;SQL> SQL>

GROUP_NUMBER HEADER_STATU State       Total Size (MB)    FREE_MB PATH
------------ ------------ ----------- --------------- ---------- ------------------------------
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t1_data12p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t1_data11p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t1_data10p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t1_data09p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t1_data08p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t2_fra08p1
           3 MEMBER       NORMAL              102,396     102372 /dev/mapper/t2_fra06p1
           2 MEMBER       NORMAL              102,396      11556 /dev/mapper/t2_fra07p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra04p1
           3 MEMBER       NORMAL              102,396     102376 /dev/mapper/t2_fra05p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra03p1
           1 MEMBER       NORMAL                5,116       4772 /dev/mapper/t1_vote3p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote1p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote2p1
           4 MEMBER       NORMAL                5,116       4016 /dev/mapper/t1_redo01p1
           4 MEMBER       NORMAL                5,116       4032 /dev/mapper/t1_redo02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo04p1
           3 MEMBER       NORMAL              204,792     204744 /dev/mapper/t2_fra02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo03p1
           3 MEMBER       NORMAL              204,792     204748 /dev/mapper/t2_fra01p1
           2 MEMBER       NORMAL              102,396      11544 /dev/mapper/t1_data05p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data06p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data01p1
           2 MEMBER       NORMAL              102,396      11548 /dev/mapper/t1_data07p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data03p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data02p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data04p1

27 rows selected.

SQL> select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from  v$asm_diskgroup where name='MKRSTDATA';

GROUP_NUMBER NAME                           State       Total Size (MB)    FREE_MB
------------ ------------------------------ ----------- --------------- ----------
           2 MKRSTDATA                      MOUNTED             819,168      92320

Here I am creating dummy disk group with provided disks because want to find out is they added 100GB of each disk or not.

Following query to create new ASM disk group

SQL> create diskgroup dummy external redundancy disk 

'/dev/mapper/t1_data08p1','/dev/mapper/t1_data09p1','/dev/mapper/t1_data10p1','/dev/mapper/t1_data11p1','/dev/mapper/t1_data12p1';

Diskgroup created.

SQL>
SQL> alter diskgroup dummy rebalance power 6;

Diskgroup altered.

SQL>

Note: The maximum power for a rebalancing operation on an ASM instance. The valid values range from 1 to 11, with 1 being the default. The higher the limit the more resources are allocated resulting in faster rebalancing operations. This value is also used as the default when the POWER  clause is omitted from a rebalance operation.

Below query shows newly added disk group with disks information, Confirmed that Unix team added 100GB of disks each.

SQL> SQL> set lines 200 pages 200
col path for a30
select GROUP_NUMBER,HEADER_STATUS,STATE,TOTAL_MB,FREE_MB,PATH from v$asm_disk;SQL> SQL>

GROUP_NUMBER HEADER_STATU State       Total Size (MB)    FREE_MB PATH
------------ ------------ ----------- --------------- ---------- ------------------------------
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t2_fra08p1
           3 MEMBER       NORMAL              102,396     102372 /dev/mapper/t2_fra06p1
           2 MEMBER       NORMAL              102,396      11556 /dev/mapper/t2_fra07p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra04p1
           3 MEMBER       NORMAL              102,396     102376 /dev/mapper/t2_fra05p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra03p1
           1 MEMBER       NORMAL                5,116       4772 /dev/mapper/t1_vote3p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote1p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote2p1
           4 MEMBER       NORMAL                5,116       4016 /dev/mapper/t1_redo01p1
           4 MEMBER       NORMAL                5,116       4032 /dev/mapper/t1_redo02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo04p1
           3 MEMBER       NORMAL              204,792     204744 /dev/mapper/t2_fra02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo03p1
           3 MEMBER       NORMAL              204,792     204748 /dev/mapper/t2_fra01p1
           2 MEMBER       NORMAL              102,396      11544 /dev/mapper/t1_data05p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data06p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data01p1
           2 MEMBER       NORMAL              102,396      11548 /dev/mapper/t1_data07p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data03p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data02p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data04p1
           5 MEMBER       NORMAL              102,398     102387 /dev/mapper/t1_data12p1
           5 MEMBER       NORMAL              102,398     102386 /dev/mapper/t1_data11p1
           5 MEMBER       NORMAL              102,398     102386 /dev/mapper/t1_data10p1
           5 MEMBER       NORMAL              102,398     102386 /dev/mapper/t1_data09p1
           5 MEMBER       NORMAL              102,398     102387 /dev/mapper/t1_data08p1

27 rows selected.

Dropping dummy ASM disk group

SQL> drop diskgroup dummy;

Diskgroup dropped.

SQL> 
SQL>set lines 200 pages 200
col path for a30
select GROUP_NUMBER,HEADER_STATUS,STATE,TOTAL_MB,FREE_MB,PATH from v$asm_disk;SQL> SQL>

GROUP_NUMBER HEADER_STATU State       Total Size (MB)    FREE_MB PATH
------------ ------------ ----------- --------------- ---------- ------------------------------
           0 FORMER       NORMAL                    0          0 /dev/mapper/t1_data12p1
           0 FORMER       NORMAL                    0          0 /dev/mapper/t1_data11p1
           0 FORMER       NORMAL                    0          0 /dev/mapper/t1_data10p1
           0 FORMER       NORMAL                    0          0 /dev/mapper/t1_data09p1
           0 FORMER       NORMAL                    0          0 /dev/mapper/t1_data08p1
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t2_fra08p1
           3 MEMBER       NORMAL              102,396     102372 /dev/mapper/t2_fra06p1
           2 MEMBER       NORMAL              102,396      11556 /dev/mapper/t2_fra07p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra04p1
           3 MEMBER       NORMAL              102,396     102376 /dev/mapper/t2_fra05p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra03p1
           1 MEMBER       NORMAL                5,116       4772 /dev/mapper/t1_vote3p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote1p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote2p1
           4 MEMBER       NORMAL                5,116       4016 /dev/mapper/t1_redo01p1
           4 MEMBER       NORMAL                5,116       4032 /dev/mapper/t1_redo02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo04p1
           3 MEMBER       NORMAL              204,792     204744 /dev/mapper/t2_fra02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo03p1
           3 MEMBER       NORMAL              204,792     204748 /dev/mapper/t2_fra01p1
           2 MEMBER       NORMAL              102,396      11544 /dev/mapper/t1_data05p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data06p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data01p1
           2 MEMBER       NORMAL              102,396      11548 /dev/mapper/t1_data07p1
           2 MEMBER       NORMAL              102,396      11532 /dev/mapper/t1_data03p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data02p1
           2 MEMBER       NORMAL              102,396      11536 /dev/mapper/t1_data04p1

27 rows selected.

SQL>

Now adding disks to existing MKRSTDATA disk group

SQL> alter diskgroup MKRSTDATA add disk '/dev/mapper/t1_data08p1','/dev/mapper/t1_data09p1','/dev/mapper/t1_data10p1','/dev/mapper/t1_data11p1','/dev/mapper/t1_data12p1'  rebalance power 11;

Diskgroup altered.

SQL> set lines 200 pages 200
col path for a30
select GROUP_NUMBER,HEADER_STATUS,STATE,TOTAL_MB,FREE_MB,PATH from v$asm_disk;SQL> SQL>

GROUP_NUMBER HEADER_STATU State       Total Size (MB)    FREE_MB PATH
------------ ------------ ----------- --------------- ---------- ------------------------------
           0 CANDIDATE    NORMAL                    0          0 /dev/mapper/t2_fra08p1
           3 MEMBER       NORMAL              102,396     102372 /dev/mapper/t2_fra06p1
           2 MEMBER       NORMAL              102,396      11864 /dev/mapper/t2_fra07p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra04p1
           3 MEMBER       NORMAL              102,396     102376 /dev/mapper/t2_fra05p1
           3 MEMBER       NORMAL              102,396     102368 /dev/mapper/t2_fra03p1
           1 MEMBER       NORMAL                5,116       4772 /dev/mapper/t1_vote3p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote1p1
           1 MEMBER       NORMAL                5,116       4768 /dev/mapper/t1_vote2p1
           4 MEMBER       NORMAL                5,116       4016 /dev/mapper/t1_redo01p1
           4 MEMBER       NORMAL                5,116       4032 /dev/mapper/t1_redo02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo04p1
           3 MEMBER       NORMAL              204,792     204744 /dev/mapper/t2_fra02p1
           4 MEMBER       NORMAL                5,116       4036 /dev/mapper/t1_redo03p1
           3 MEMBER       NORMAL              204,792     204748 /dev/mapper/t2_fra01p1
           2 MEMBER       NORMAL              102,396      11868 /dev/mapper/t1_data05p1
           2 MEMBER       NORMAL              102,396      11856 /dev/mapper/t1_data06p1
           2 MEMBER       NORMAL              102,396      11856 /dev/mapper/t1_data01p1
           2 MEMBER       NORMAL              102,396      11872 /dev/mapper/t1_data07p1
           2 MEMBER       NORMAL              102,396      11856 /dev/mapper/t1_data03p1
           2 MEMBER       NORMAL              102,396      11856 /dev/mapper/t1_data02p1
           2 MEMBER       NORMAL              102,396      11860 /dev/mapper/t1_data04p1
           2 MEMBER       NORMAL              102,396     101872 /dev/mapper/t1_data12p1
           2 MEMBER       NORMAL              102,396     101872 /dev/mapper/t1_data11p1
           2 MEMBER       NORMAL              102,396     101876 /dev/mapper/t1_data10p1
           2 MEMBER       NORMAL              102,396     101876 /dev/mapper/t1_data09p1
           2 MEMBER       NORMAL              102,396     101876 /dev/mapper/t1_data08p1

27 rows selected.

SQL>
SQL> SET LINESIZE  145
SQL> SET PAGESIZE  9999
SQL> SET VERIFY    off
SQL> COLUMN group_name             FORMAT a20           HEAD 'Disk Group|Name'
SQL> COLUMN sector_size            FORMAT 99,999        HEAD 'Sector|Size'
SQL> COLUMN block_size             FORMAT 99,999        HEAD 'Block|Size'
SQL> COLUMN allocation_unit_size   FORMAT 999,999,999   HEAD 'Allocation|Unit Size'
SQL> COLUMN state                  FORMAT a11           HEAD 'State'
COLUMN type                   FORMAT a6            HEAD 'Type'
SQL> SQL> COLUMN total_mb               FORMAT 999,999,999   HEAD 'Total Size (MB)'
SQL> COLUMN used_mb                FORMAT 999,999,999   HEAD 'Used Size (MB)'
SQL> COLUMN pct_used               FORMAT 999.99        HEAD 'Pct. Used'
SQL> SELECT
  2      name                                     group_name
  3    , sector_size                              sector_size
  4    , block_size                               block_size
  5    , allocation_unit_size                     allocation_unit_size
  , state                                    state
  6    7    , type                                     type
  8    , total_mb                                 total_mb
  9    , (total_mb - free_mb)                     used_mb
 10    , ROUND((1- (free_mb / total_mb))*100, 2)  pct_used
 11  FROM
 12      v$asm_diskgroup
 13  ORDER BY
 14      name
/ 15

Disk Group            Sector   Block   Allocation
Name                    Size    Size    Unit Size State       Type   Total Size (MB) Used Size (MB) Pct. Used
-------------------- ------- ------- ------------ ----------- ------ --------------- -------------- ---------
MKRSTDATA                512   4,096    4,194,304 MOUNTED     EXTERN       1,331,148        726,888     54.61
MKRSTFRA                 512   4,096    4,194,304 MOUNTED     EXTERN         819,168            192       .02
MKRSTREDO                512   4,096    4,194,304 MOUNTED     EXTERN          20,464          4,344     21.23
OCRVOTEDG                512   4,096    4,194,304 MOUNTED     NORMAL          15,348          1,040      6.78

SQL>

Done.. Hope this will help you. Feel free to comment here for any help required... :)

Best Regards,
Chowdai

ORA-04031 unable to allocate 3896 bytes of shared memory in ASM Instance

Today I have faced CHECK TIMED OUT and INTERMEDIATE state errors in listener and ASM disk group services. Then I have started investigation we found "ORA-04031: unable to allocate 3896 bytes of shared memory" error in ASM alert log file.

Sat Aug 23 11:23:00 2014
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_26804658.trc  (incident=19712):
ORA-04031: unable to allocate 3896 bytes of shared memory ("shared pool","SELECT value FROM v$paramete...","sga heap(1,0)","kglsim object batch")
Use ADRCI or Support Workbench to package the incident.

bash-4.2$ crsctl stat res -t
---------------------------------------------------------------------------------------------------------
NAME             TARGET    STATE                SERVER                   STATE_DETAILS   Local Resources
---------------------------------------------------------------------------------------------------------
ora.DATA.dg        
                       ONLINE    INTERMEDIATE  PRODDB1                    CHECK TIMED OUT     
                       ONLINE    INTERMEDIATE  PRODDB2                    CHECK TIMED OUT     
ora.FRA.dg
                      ONLINE     INTERMEDIATE  PRODDB1                    CHECK TIMED OUT     
                      ONLINE     INTERMEDIATE  PRODDB2                    CHECK TIMED OUT     
ora.LISTENER.lsnr
                     ONLINE      ONLINE                PRODDB1                                        
                     ONLINE      INTERMEDIATE  PRODDB2                    Not All Endpoints Registered 
ora.OCRVD.dg
                     ONLINE      INTERMEDIATE  PRODDB1                    CHECK TIMED OUT     
                     ONLINE      INTERMEDIATE  PRODDB2                    CHECK TIMED OUT     
ora.asm
                     ONLINE      INTERMEDIATE  PRODDB1                    CHECK TIMED OUT     
                     ONLINE      ONLINE                PRODDB2                    Started             
ora.gsd
               OFFLINE OFFLINE      PRODDB1                                        
               OFFLINE OFFLINE      PRODDB2                                        
ora.net1.network
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.ons
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.registry.acfs
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  UNKNOWN      PRODDB2                                        
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       PRODDB1                                        
ora.cvu
      1        ONLINE  ONLINE       PRODDB1                                        
ora.PRODDB1.vip
      1        ONLINE  ONLINE       PRODDB1                                        
ora.PRODDB2.vip
      1        ONLINE  ONLINE       PRODDB2                                        
ora.ehishyd.db
      1        ONLINE  ONLINE       PRODDB1                    Open                
      2        ONLINE  ONLINE       PRODDB2                    Open                
ora.oc4j
      1        ONLINE  ONLINE       PRODDB2                                        
ora.scan1.vip
      1        ONLINE  ONLINE       PRODDB1                                        
bash-4.2$

bash-4.2$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  INTERMEDIATE PRODDB1                    CHECK TIMED OUT     
               ONLINE  INTERMEDIATE PRODDB2                    CHECK TIMED OUT     
ora.FRA.dg
               ONLINE  INTERMEDIATE PRODDB1                    CHECK TIMED OUT     
               ONLINE  INTERMEDIATE PRODDB2                    CHECK TIMED OUT     
ora.LISTENER.lsnr
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  OFFLINE      PRODDB2                                        
ora.OCRVD.dg
               ONLINE  INTERMEDIATE PRODDB1                    CHECK TIMED OUT     
               ONLINE  INTERMEDIATE PRODDB2                    CHECK TIMED OUT     
ora.asm
               ONLINE  INTERMEDIATE PRODDB1                    CHECK TIMED OUT     
               ONLINE  ONLINE       PRODDB2                    Started             
ora.gsd
               OFFLINE OFFLINE      PRODDB1                                        
               OFFLINE OFFLINE      PRODDB2                                        
ora.net1.network
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.ons
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.registry.acfs
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  UNKNOWN      PRODDB2                                        
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       PRODDB1                                        
ora.cvu
      1        ONLINE  ONLINE       PRODDB1                                        
ora.PRODDB1.vip
      1        ONLINE  ONLINE       PRODDB1                                        
ora.PRODDB2.vip
      1        ONLINE  ONLINE       PRODDB2                                        
ora.ehishyd.db
      1        ONLINE  ONLINE       PRODDB1                    Open                
      2        ONLINE  ONLINE       PRODDB2                    Open                
ora.oc4j
      1        ONLINE  ONLINE       PRODDB2                                        
ora.scan1.vip
      1        ONLINE  ONLINE       PRODDB1                                        
bash-4.2$ 

I came to know that its an ASM instance memory related issue. Checked below memory parameters in ASM instance.

SQL> show parameter sga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
lock_sga                             boolean     FALSE
sga_max_size                         big integer 272M
sga_target                           big integer 0
SQL>
SQL> show parameter memory

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 272M
memory_target                        big integer 272M
SQL>

Automatic memory management is enabled by default on an Oracle ASM instance and oracle recommends that to use Automatic Memory Management (AMM) for ASM.  In Oracle 11gR2 the default value used for MEMORY_TARGET is 272 MB. Here SGA_TARGET is 256MB and 16MB is for PGA_TARGET.

Based on "ORA-04031: unable to allocate 3896 bytes of shared memory" error I get to know that ASM memory parameters in ASM instances are too low and it results in ORA-04031 in the ASM instance.

Based on Oracle Doc ID 437924.1, I have incresed memory_max_target to 4096M and memory_target to 1536M. Then I restarted the ASM instances.

SQL> alter system set memory_max_target=4096M scope=spfile;
SQL> alter system set memory_target=1536M scope=spfile;

After changing above parameters the services came to ONLINE state.

bash-4.2$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.FRA.dg
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.LISTENER.lsnr
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.OCRVD.dg
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.asm
               ONLINE  ONLINE       PRODDB1                    Started             
               ONLINE  ONLINE       PRODDB2                    Started             
ora.gsd
               OFFLINE OFFLINE      PRODDB1                                        
               OFFLINE OFFLINE      PRODDB2                                        
ora.net1.network
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.ons
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
ora.registry.acfs
               ONLINE  ONLINE       PRODDB1                                        
               ONLINE  ONLINE       PRODDB2                                        
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       PRODDB2                                        
ora.cvu
      1        ONLINE  ONLINE       PRODDB2                                        
ora.PRODDB1.vip
      1        ONLINE  ONLINE       PRODDB1                                        
ora.PRODDB2.vip
      1        ONLINE  ONLINE       PRODDB2                                        
ora.ehishyd.db
      1        ONLINE  ONLINE       PRODDB1                    Open                
      2        ONLINE  ONLINE       PRODDB2                    Open                
ora.oc4j
      1        ONLINE  ONLINE       PRODDB2                                        
ora.scan1.vip
      1        ONLINE  ONLINE       PRODDB2                                        
bash-4.2$

Hope this will help you.. :)

Best Regards,
Chowdari.

Export Logical Database Backup Dumps to ASM Disk

I have created both directory and folder within the ASM diskgroup.

bash-3.2$ echo $ORACLE_HOME
/oracle/grid_home
bash-3.2$
bash-3.2$ echo $ORACLE_SID
+ASM1

bash-3.2$ asmcmd -p
ASMCMD [+] >

ASMCMD [+] > ls
DATA/
FRA/

ASMCMD [+] > cd FRA
ASMCMD [+FRA] >

ASMCMD [+FRA] > mkdir dpdump
ASMCMD [+FRA] >

ASMCMD [+FRA] > cd dp*
ASMCMD [+FRA/dpdump] >

ASMCMD [+FRA/dpdump] > pwd
+FRA/dpdump
ASMCMD [+FRA/dpdump] >

bash-3.2$ sqlplus / as sysdba

SQL> create directory test_dir as '+FRA/dpdump';

Directory created.

SQL>
SQL> select * from dba_directories;

OWNER          DIRECTORY_NAME  DIRECTORY_PATH
--------------   --------------                ----------------------------------
SYS                TEST_DIR                     +FRA/dpdump
SYS                DATA_PUMP_DIR        /U04/dpdump

SQL>

bash-3.2$
bash-3.2$ expdp directory=TEST_DIR dumpfile=test_full.dmp logfile=test_full.log full=y exclude=statistics content=metadata_only

Export: Release 11.2.0.2.0 - Production on Thu Sep 26 15:43:15 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Username: / as sysdba

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORA-39002: invalid operation
ORA-39070: Unable to open the log file.
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation

bash-3.2$

Here we have seen an error "unable to open log file".

So now the gimmick is, to use existing directory, such as DATA_PUMP_DIR which is placed on a local filesystem and point the logfile to that directory. Using the syntax 'logfile=directory:logfile'. An alternative option is to use the expdp parameter to nologfile=yes

bash-3.2$
bash-3.2$ expdp directory=TEST_DIR dumpfile=test_full.dmp logfile=DATA_PUMP_DIR:test_full.log full=y exclude=statistics content=metadata_only

Export: Release 11.2.0.2.0 - Production on Thu Sep 26 15:48:48 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Username: / as sysdba

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
Starting "SYS"."SYS_EXPORT_FULL_05":  /******** AS SYSDBA directory=TEST_DIR dumpfile=test_full.dmp logfile=DATA_PUMP_DIR:test_full.log full=y exclude=statistics content=metadata_only
Processing object type DATABASE_EXPORT/TABLESPACE
Processing object type DATABASE_EXPORT/PASSWORD_VERIFY_FUNCTION
Processing object type DATABASE_EXPORT/PROFILE
Processing object type DATABASE_EXPORT/SYS_USER/USER
Processing object type DATABASE_EXPORT/SCHEMA/USER
.
.
.
Dump file set for SYS.SYS_EXPORT_FULL_05 is:
  +FRA/dpdump/test_full.dmp
Job "SYS"."SYS_EXPORT_FULL_05" successfully completed at 16:00:52
bash-3.2$

Hope this help you.

Best Regards,