Troubleshooting 12C node2 CRS start fail with ORA-12547 and ORA-15077 in Flex ASM 案例

Flex ASM

在12c以前的版本数据库实例使用操作系统认证连接ASM实例，因为ASM CLIENT(DB INSTANCE)和ASM Server总是在同一个主机上， 从12c版本开始引入的FLEX ASM架构允许数据库实例可以和ASM运行在不同的主机中， 使用FLEX ASM user password文件认证， ASM 密码文件存储在ASM DISKGROUP中， 同时在创建Flex ASM时会默认创建ASM USER。Flex ASM 也支持oracle 12c前版本的rdbms, 同样也是建议使用的ASM架构

ASM Network

在Flex ASM Oracle 12c引入了一种新类型network, 叫做ASM network.  这种network用于ASM和ASM CLIENT及所有节点间通信。集群中的所有ASM client可以访问所有ASM network,    也可以只配置一个network共用于支持private network和asm network。

ASM Listeners

ASM listener用来支持Flex ASM 访问， 为每个ASM network配置一组ASM listener, 每个ASM client数据库实例中最多将三个ASM listener地址注册为remote listeners,所有客户端连接都在整个ASM实例集中进行负载平衡. 默认名为 ASMNET1LSNR_ASM

案例

有了上面的基础认识，开始分析最近遇到的一个FLEX ASM相关的案例， 这是一套12c R2 2-nodes RAC  DG 环境 ， 在检查DG standby side时，发现Standby node2 instance未启动，是在standby node1 上接收并应用。  尝试启动实例2时发现了问题。

root@anbobstb02:~$crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

root@anbobstb02:~$crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

grid@anbobstb02$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.crf
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.crsd
      1        ONLINE  OFFLINE                               STABLE
ora.cssd
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.ctssd
      1        ONLINE  ONLINE       anbobstb02                  OBSERVER,STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.evmd
      1        ONLINE  INTERMEDIATE anbobstb02                  STABLE
ora.gipcd
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.gpnpd
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.mdnsd
      1        ONLINE  ONLINE       anbobstb02                  STABLE
ora.storage
      1        ONLINE  ONLINE       anbobstb02                  STABLE
--------------------------------------------------------------------------------

grid@anbobstb02:~$crsctl get cluster mode status
Cluster is running in "flex" mode

Note:

Flex ASM环境， 在启动NODE2 CRS时失败。

CRS alert log

2019-02-12 10:50:05.229 [OCSSD(51008)]CRS-1713: CSSD daemon is started in hub mode
2019-02-12 10:50:06.670000 +08:00
2019-02-12 10:50:06.670 [OCSSD(51008)]CRS-1707: Lease acquisition for node anbobstb02 number 2 completed
2019-02-12 10:50:07.756000 +08:00
2019-02-12 10:50:07.756 [OCSSD(51008)]CRS-1605: CSSD voting file is online: /dev/asm-disk55; details in /oracle/app/grid/diag/crs/anbobstb02/crs/trace/ocssd.trc.
2019-02-12 10:50:07.759 [OCSSD(51008)]CRS-1605: CSSD voting file is online: /dev/asm-disk52; details in /oracle/app/grid/diag/crs/anbobstb02/crs/trace/ocssd.trc.
2019-02-12 10:50:07.763 [OCSSD(51008)]CRS-1605: CSSD voting file is online: /dev/asm-disk51; details in /oracle/app/grid/diag/crs/anbobstb02/crs/trace/ocssd.trc.
2019-02-12 10:50:14.376000 +08:00
2019-02-12 10:50:14.376 [OCSSD(51008)]CRS-1601: CSSD Reconfiguration complete. Active nodes are anbobstb01 anbobstb02 .
2019-02-12 10:50:17.177000 +08:00
2019-02-12 10:50:17.176 [OCTSSD(55374)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 55374
2019-02-12 10:50:17.193 [OCSSD(51008)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2019-02-12 10:50:18.157 [OCTSSD(55374)]CRS-2403: The Cluster Time Synchronization Service on host anbobstb02 is in observer mode.
2019-02-12 10:50:19.266000 +08:00
2019-02-12 10:50:19.266 [OCTSSD(55374)]CRS-2407: The new Cluster Time Synchronization Service reference node is host anbobstb01.
2019-02-12 10:50:19.266 [OCTSSD(55374)]CRS-2401: The Cluster Time Synchronization Service started on host anbobstb02.
2019-02-12 10:50:35.725000 +08:00
2019-02-12 10:50:35.725 [ORAROOTAGENT(50588)]CRS-5019: All OCR locations are on ASM disk groups [OCRDG], 
and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/oracle/app/grid/diag/crs/anbobstb02/crs/trace/ohasd_orarootagent_root.trc".

Trace ohasd_orarootagent_root.trc

adrci> show trace /oracle/app/grid/diag/crs/anbobstb02/crs/trace/ohasd_orarootagent_root.trc

2019-02-12 10:50:25.765 : AGFW:2530133760: {0:5:3} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:403

2019-02-12 10:50:25.765 : USRTHRD:2519627520: {0:5:3} Check: 0-1

2019-02-12 10:50:25.766 : AGFW:2530133760: {0:5:3} ora.cluster_interconnect.haip 1 1 state changed from: STARTING to: ONLINE

2019-02-12 10:50:25.766 : AGFW:2530133760: {0:5:3} RECYCLE_AGENT attribute not found

2019-02-12 10:50:25.766 : AGFW:2530133760: {0:5:3} Started implicit monitor for [ora.cluster_interconnect.haip 1 1] interval=30000 delay=3000

0

2019-02-12 10:50:25.766 : AGFW:2530133760: {0:5:3} Agent sending last reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:40

3

2019-02-12 10:50:25.768 : USRTHRD:2512111360: {0:5:3} got pubgrpdata, 1-8-2-2-2

2019-02-12 10:50:25.770 : USRTHRD:2512111360: {0:5:3} Completed 1 HAIP assignment, start complete

2019-02-12 10:50:25.770 : USRTHRD:2512111360: {0:5:3} to verify inf event

2019-02-12 10:50:25.813 : AGFW:2530133760: {0:5:3} Agent received the message: RESOURCE_START[ora.storage 1 1] ID 4098:438

2019-02-12 10:50:25.813 : AGFW:2530133760: {0:5:3} Preparing START command for: ora.storage 1 1

2019-02-12 10:50:25.813 : AGFW:2530133760: {0:5:3} ora.storage 1 1 state changed from: OFFLINE to: STARTING

2019-02-12 10:50:25.813 : AGFW:2530133760: {0:5:3} RECYCLE_AGENT attribute not found

2019-02-12 10:50:25.813 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] (:CLSN00107:) clsn_agent::start {

2019-02-12 10:50:25.814 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] StorageAgent::init NodeRole = 1

2019-02-12 10:50:25.814 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] StorageAgent::check NODEROLE_HUB getOCRdetails

2019-02-12 10:50:25.832 : default:2519627520: clsvactversion:4: Retrieving Active Version from local storage.

2019-02-12 10:50:25.840 :GIPCXCPT:2519627520: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)

2019-02-12 10:50:25.840 :GIPCXCPT:2519627520: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalid

Attribute (5) ] failure for obj 0x7f8664436460 [0000000000000955] { gipcEndpoint : localAddr ”, remoteAddr ”, numPend 0, numReady 0, numDone

0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, u

srFlags 0x0 }, name ‘traceLevel’, val 0x7f86962cf004, len 4, flags 0x0

2019-02-12 10:50:25.855 : CLSNS:2519627520: clsns_SetTraceLevel:trace level set to 1.

2019-02-12 10:50:25.859 : default:2519627520: Inited LSF context: 0x7f866453c5d0

2019-02-12 10:50:25.863 : CLSCRED:2519627520: clsCredCommonInit: Inited singleton credctx.

2019-02-12 10:50:25.863 : CLSCRED:2519627520: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.

2019-02-12 10:50:25.886 : USRTHRD:2519627520: {0:5:3} 8154 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2019-02-12 10:50:25.889 : USRTHRD:2519627520: {0:5:3} 8154 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS

2019-02-12 10:50:25.924 : CLSCRED:2519627520: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.bb15e951dcbc4fc2ff3aec3bfe1f0424.root not found

2019-02-12 10:50:25.924 : USRTHRD:2519627520: {0:5:3} 7872 Error 4 opening dom root in 0x7f866441d590

2019-02-12 10:50:25.929 :GIPCXCPT:2519627520: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)

2019-02-12 10:50:25.929 :GIPCXCPT:2519627520: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalid Attribute (5) ] failure for obj 0x7f86647145d0 [0000000000000fc1] { gipcEndpoint : localAddr ”, remoteAddr ”, numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name ‘traceLevel’, val 0x7f86962cf004, len 4, flags 0x0

2019-02-12 10:50:26.014 : AGFW:2743070784: Recvd request to shed the threads

2019-02-12 10:50:26.014 :CLSFRAME:2743070784: TM [MultiThread] is changing desired thread # to 8. Current # is 9

2019-02-12 10:50:26.014 :CLSFRAME:2532235008: {0:1:5} Worker thread is exiting in TM [MultiThread] to meet the desired count of 8. New count is 8

…

…

…

2019-02-12 10:50:29.219 : USRTHRD:2519627520: {0:5:3} 7872 Error 4 opening dom root in 0x7f866465b680

2019-02-12 10:50:29.222 :GIPCXCPT:2519627520: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)

2019-02-12 10:50:29.222 :GIPCXCPT:2519627520: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalid

Attribute (5) ] failure for obj 0x7f8664546cd0 [0000000000002068] { gipcEndpoint : localAddr ”, remoteAddr ”, numPend 0, numReady 0, numDone

0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name ‘traceLevel’, val 0x7f86962cf004, len 4, flags 0x0

…

2019-02-12 10:50:34.664 : USRTHRD:2519627520: {0:5:3} 7872 Error 4 opening dom root in 0x7f86645abf50

2019-02-12 10:50:34.668 :GIPCXCPT:2519627520: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)

2019-02-12 10:50:34.668 :GIPCXCPT:2519627520: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalid Attribute (5) ] failure for obj 0x7f8664707b20 [000000000000392d] { gipcEndpoint : localAddr ”, remoteAddr ”, numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name ‘traceLevel’, val 0x7f86962cf004, len 4, flags 0x0

2019-02-12 10:50:35.715 : USRTHRD:2509133568: HAIP: event GIPCD_METRIC_UPDATE

2019-02-12 10:50:35.715 : USRTHRD:2512111360: {0:5:3} to verify inf event

2019-02-12 10:50:35.724 : default:2519627520: clsCredDomClose: Credctx deleted 0x7f866443f840

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} — trace dump on error exit —

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} Error [kgfoAl06] in [kgfokge] at kgfo.c:3115

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} ORA-12547: TNS:lost contact

ORA-12547: TNS:lost contact

ORA-15077: could not locate ASM instance serving a required diskgroup

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} Category: 7

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} DepInfo: 12547

2019-02-12 10:50:35.724 : USRTHRD:2519627520: {0:5:3} — trace dump end —

2019-02-12 10:50:35.724 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] StorageAgent::parsekgforetcodes retcode = 7, kgfoCheckMount(OCRDG), flag 2

2019-02-12 10:50:35.724 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 1254

7, other: ORA-12547: TNS:lost contact

ORA-12547: TNS:lost contact

ORA-15077: could not locate ASM instance serving a required diskgroup

2019-02-12 10:50:35.724 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] StorageAgent::check kgfo returncode 1

2019-02-12 10:50:35.724 :CLSDYNAM:2519627520: [ora.storage]{0:5:3} [start] (:CLSN00140:)StorageAgent::parsekgforretcodes OCR dgName OCRDG state 1

Note:

从日志看应该是在CRS启动时没有发现ASM DISKGROUP,  asm 启动时在取asm 认证证数时出错，提示是ora-12547和ora-15055， Flex ASM中ASM server启动时要连接所有asm network.  下一步检查NODE1 的ASM listener.

grid@anbobstb01:/home/grid> crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.DATADG.dg
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.MGMT.dg
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.OCRDG.dg
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.chad
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.net1.network
               ONLINE  ONLINE       anbobstb01                  STABLE
ora.ons
               ONLINE  ONLINE       anbobstb01                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       anbobstb01                  169.254.143.82 192.1
                                                             68.43.33,STABLE
ora.asm
      1        ONLINE  ONLINE       anbobstb01                  Started,STABLE
      2        ONLINE  OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       anbobstb01                  Open,STABLE
ora.anbobstb01.vip
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.anbobstb02.vip
      1        ONLINE  INTERMEDIATE anbobstb01                  FAILED OVER,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.rptstby.db
      1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  ONLINE       anbobstb01                  Open,Readonly,HOME=/
                                                             oracle/app/oracle/pr
                                                             oduct/12.2.0/db_1,ST
                                                             ABLE
ora.scan1.vip
      1        ONLINE  ONLINE       anbobstb01                  STABLE
--------------------------------------------------------------------------------
grid@anbobstb01:/home/grid> crsctl stat res -t -init
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       anbobstb01                  Started,STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.crf
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.crsd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.cssd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.ctssd
      1        ONLINE  ONLINE       anbobstb01                  OBSERVER,STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.evmd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.gipcd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.gpnpd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.mdnsd
      1        ONLINE  ONLINE       anbobstb01                  STABLE
ora.storage
      1        ONLINE  ONLINE       anbobstb01                  STABLE
--------------------------------------------------------------------------------

grid@anbobstb01:/home/grid> ocrdump /tmp/ocr.dmp
PROT-310: Not all keys were dumped due to permissions.
grid@anbobstb01:/home/grid> vi /tmp/ocr.dmp

[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001]
ORATEXT : bb15e951dcbc4fc2ff3aec3bfe1f0424:grid   --credentials is exist
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}

grid@anbobstb01:~$oifcfg getif
bond0  133.96.43.0  global  public
bond1  192.168.43.0  global  cluster_interconnect,asm

grid@anbobstb01:/oracle/app/12.2.0/grid/bin> ps -ef|grep lsnr
grid      1411     1  0  2018 ?        00:01:53 /oracle/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
grid      6585     1  0  2018 ?        00:00:21 /oracle/app/12.2.0/grid/bin/tnslsnr listener_dg -inherit
grid     25826 19172  0 10:57 pts/3    00:00:00 grep --color=auto lsnr
grid     72783     1  0  2018 ?        00:02:01 /oracle/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
grid     78242     1  0 Feb12 ?        00:00:05 /oracle/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid     80521     1  0 Feb12 ?        00:00:35 /oracle/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit

grid@anbobstb01:/oracle/app/12.2.0/grid/bin> lsnrctl status ASMNET1LSNR_ASM
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 13-FEB-2019 10:57:43
Copyright (c) 1991, 2016, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))
STATUS of the LISTENER
------------------------
Alias                     ASMNET1LSNR_ASM
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                12-FEB-2019 18:21:02
Uptime                    0 days 16 hr. 36 min. 40 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/app/12.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/grid/diag/tnslsnr/anbobstb01/asmnet1lsnr_asm/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=ASMNET1LSNR_ASM)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.43.33)(PORT=1526)))
The listener supports no services
The command completed successfully

grid@anbobstb01:/oracle/app/grid/diag/tnslsnr/anbobstb01/asmnet1lsnr_asm/trace> vi asmnet1lsnr_asm.log

2019-02-13T10:57:16.203184+08:00
Incoming connection from 192.168.43.34 rejected
13-FEB-2019 10:57:16 * 12546
TNS-12546: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00516: Permission denied
  

grid@anbobstb01:/oracle/app/grid/diag/tnslsnr/anbobstb01/asmnet1lsnr_asm/trace> telnet 192.168.43.33 1526
Trying 192.168.43.33...
Connected to 192.168.43.33.
Escape character is '^]'.
Connection closed by foreign host.

Note:

实例1上看当前使用是没问题的， 但是上面运行的ASM listener没有服务， 使用telnet 发现很快会被拒绝， 检查iptables 没有限制，使用tcpdump 发现是监听进程发出的reset package. 如果当前的ASM Listener没有服务，那么Flex ASM 集群间就没有办法通信。跟监听连接相关的限制可能是sqlnet.ora.

grid@anbobstb01:/oracle/app/12.2.0/grid/network/admin> vi sqlnet.ora
NAMES.DIRECTORY_PATH= (TNSNAMES,EZCONNECT)

ADR_BASE = /oracle/app/grid
TCP.VALIDNODE_CHECKING=yes
TCP.INVITED_NODES=(...)

Note:

发现果然有sqlnet.ora中配置白名单，但是sqlnet.ora文件是从primary database复制过来的， 而primary和standby的Private network(ASM network) 不是一个子网段，所以standby side的白名单中并没有ASM network， 而没有服务。

解决方法

解决起来就简单了，在sqlnet.ora中增加ASM network的网段值。 这里提醒下，以后增长监听白名单，记的除了前端应用IP，还要加PUBLIC NETWORK, PRIVATE NETWORK, SCAN IP, ASM NETWORK..