Saturday, November 16, 2013

Listener INTERMEDIATE status with "Not All Endpoints Registered" in 11gR2 RAC

When I issue clusterware services status resources command "crsctl stat res -t" it shows the listeners are in INTERMEDIATE status on node 1. The problem is caused by another listener defined statically in listener.ora,  using the same port and IP is running from the  ORACLE_HOME, started manually causing the default listener starting from GRID_HOME can not register its endpoint.

bash-3.2$ crsctl stat res -t
--------------------------------------------------------------------------------------------------------
NAME       TARGET  STATE   SERVER              STATE_DETAILS     Local Resources
---------------------------------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.FRA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE rac1            Not All Endpoints Registered                                              
               ONLINE  ONLINE       rac2                                      
ora.asm
               ONLINE  ONLINE       rac1                  Started            
               ONLINE  ONLINE       rac2                  Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                      
               OFFLINE OFFLINE      rac2                                      
ora.net1.network
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.ons
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.registry.acfs
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                      
ora.cvu
      1        ONLINE  ONLINE       rac2                                      
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                      
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                      
ora.racdb.db
      1        ONLINE  ONLINE       rac1                   Open              
      2        ONLINE  ONLINE       rac2                   Open              
ora.oc4j
      1        ONLINE  ONLINE       rac2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                                      
bash-3.2$

bash-3.2$ ps -ef|grep tns
  oracle  7471166        1   0   Oct 10      -  0:04 /u01/app/oracle/product/11.2.0/db_1/bin/tnslsnr LISTENER -inherit
  oracle 30343230        1   0 13:11:50      -  0:00 /u01/app/grid/bin/tnslsnr LISTENER -inherit
  oracle 53674128 58720430   0 13:13:13  pts/1  0:00 grep tns
bash-3.2$


Solution for the above problem is, From 11g release 2 onwards all listeners should be running from GRID_HOME, listener and listener_scan(n) entry should be added automatically in listener.ora file.

1.Stop the listener running from ORACLE_HOME (RDBMS)

$/bin/lsnrctl stop
 
2. stop the listener from GRID_HOME

$/bin/srvctl stop listener -n node_name
$/bin/srvctl stop scan_listener -i scan#

3. restart the LISTENER and LISTENER_SCAN1  from GRID_HOME

$/bin/srvctl start listener -n node_name
$/bin/srvctl start scan_listener -i scan#

4. check crsctl stat res -t output, they both should show ONLINE status now.

Demonstration:

       
bash-3.2$ crsctl stat res -t
--------------------------------------------------------------------------------------------------------
NAME       TARGET  STATE   SERVER              STATE_DETAILS     Local Resources
---------------------------------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.FRA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE rac1                    Not All Endpoints Registered                                              
               ONLINE  ONLINE       rac2                                      
ora.asm
               ONLINE  ONLINE       rac1                             Started            
               ONLINE  ONLINE       rac2                             Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                      
               OFFLINE OFFLINE      rac2                                      
ora.net1.network
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.ons
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.registry.acfs
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                      
ora.cvu
      1        ONLINE  ONLINE       rac2                                      
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                      
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                      
ora.racdb.db
      1        ONLINE  ONLINE       rac1                      Open              
      2        ONLINE  ONLINE       rac2                      Open              
ora.oc4j
      1        ONLINE  ONLINE       rac2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                                      
bash-3.2$
You have mail in /usr/spool/mail/oracle

bash-3.2$ ps -ef|grep tns
  oracle  7471166        1   0   Oct 10      -  0:04 /u01/app/oracle/product/11.2.0/db_1/bin/tnslsnr LISTENER -inherit
  oracle 30343230        1   0 13:11:50      -  0:00 /u01/app/grid/bin/tnslsnr LISTENER -inherit
  oracle 53674128 58720430   0 13:13:13  pts/1  0:00 grep tns
bash-3.2$

bash-3.2$ cd $ORACLE_HOME/bin
bash-3.2$
bash-3.2$ lsnrctl stop        

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 31-OCT-2013 13:18:08
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
The command completed successfully

bash-3.2$ lsnrctl status

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 31-OCT-2013 13:18:15
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   IBM/AIX RISC System/6000 Error: 79: Connection refused
bash-3.2$
bash-3.2$ ps -ef |grep tns
  oracle 58261656 58720430   0 13:18:32  pts/1  0:00 grep tns
bash-3.2$
bash-3.2$ srvctl stop listener -n rac1
bash-3.2$
bash-3.2$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac2
bash-3.2$
bash-3.2$ srvctl start listener -n rac1
bash-3.2$
bash-3.2$ ps -ef|grep tns
  oracle 17432626        1   0 13:20:00      -  0:00 /u01/app/grid/bin/tnslsnr LISTENER -inherit
  oracle 33357886 58720430   0 13:20:07  pts/1  0:00 grep tns
bash-3.2$
bash-3.2$ crsctl stat res -t
--------------------------------------------------------------------------------------------------------
NAME       TARGET  STATE   SERVER              STATE_DETAILS     Local Resources
---------------------------------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.FRA.dg
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.asm
               ONLINE  ONLINE       rac1                           Started            
               ONLINE  ONLINE       rac2                           Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                      
               OFFLINE OFFLINE      rac2                                      
ora.net1.network
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.ons
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
ora.registry.acfs
               ONLINE  ONLINE       rac1                                      
               ONLINE  ONLINE       rac2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                      
ora.cvu
      1        ONLINE  ONLINE       rac2                                      
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                      
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                      
ora.racdb.db
      1        ONLINE  ONLINE       rac1                          Open              
      2        ONLINE  ONLINE       rac2                          Open              
ora.oc4j
      1        ONLINE  ONLINE       rac2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                                      
bash-3.2$
bash-3.2$
bash-3.2$
bash-3.2$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac2,rac1
bash-3.2$


Thats it...Hope this will help you...:)

Best Regards,

6 comments:

  1. This helped me out. Thank you for taking the time to post, Chowdari.

    ReplyDelete
  2. Thank you for your comment. I am pleased the article has helped you... :)

    ReplyDelete
  3. In 11.2.0.2, both local and san listeners are running on CRS home.
    But Local listener is going offline often. Kindly suggest.

    ReplyDelete
  4. In 11.2.0.2 which is running on windows , both local and san listeners are running on CRS home.
    But Local listener is going offline often. Kindly suggest.

    ReplyDelete