tnsping时vip能通，public ip报错无监听

travel.liu

root 46727224        1   0   Mar 14      -  0:00 /oracle/app/crs/bin/crsd.bin reboot
crs的时间不对，是3月14号，出现问题了 。检查日志

mos : Listener endpoint for public host is removed for a brief period when crsd.bin crash (Doc ID 1605983.1)

xifenfei

2014-03-14 07:08:02.074: [ora.ons][4122] {0:7:2} [check] (:CLSN00010ons is running ...
2014-03-14 07:08:02.075: [CLSFRAME][1] TM [MultiThread] is changing desired thread # to 8. Current # is 7
2014-03-14 07:08:02.076: [ora.ons][4122] {0:7:2} [check] (:CLSN00010
2014-03-14 07:08:02.076: [ora.ons][4122] {0:7:2} [check] execCmd ret = 0
2014-03-14 07:08:02.628: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010
2014-03-14 07:08:02.628: [ora.ons][4122] {0:7:2} [check] OnsAgent::check }
2014-03-14 07:08:02.629: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.2.0 - Production on 14-MAR-2014 07:08:02
2014-03-14 07:08:02.629: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010
2014-03-14 07:08:02.629: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010Copyright (c) 1991, 2010, Oracle.  All rights reserved.
2014-03-14 07:08:02.629: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010
2014-03-14 07:08:02.645: [ora.LISTENER_SCAN1.lsnr][3352] {0:7:2} [check] (:CLSN00008Utils:execCmd scls_process_spawn() failed 1
2014-03-14 07:08:02.693: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-03-14 07:08:02.693: [ora.LISTENER_SCAN1.lsnr][3352] {0:7:2} [check] (:CLSN00008 category: -2, operation: fork, loc: spawnproc28, OS error: 12, other: forked failed [-1]
2014-03-14 07:07:59.706: [ COMMCRS][5088]clsc_thrd_spawn: (1123b1a18) waited 60000 milliseconds for thread to become active
2014-03-14 07:08:02.716: [    AGFW][4122] {0:7:2} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1
2014-03-14 07:08:02.716: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00010:)
2014-03-14 07:08:02.783: [ora.LISTENER.lsnr][4122] {0:7:2} [check] clsn_agent::abort {
2014-03-14 07:08:02.783: [ora.LISTENER.lsnr][4122] {0:7:2} [check] abort {
2014-03-14 07:08:02.783: [ora.LISTENER.lsnr][4122] {0:7:2} [check] abort command: check
2014-03-14 07:08:02.784: [ora.LISTENER.lsnr][4122] {0:7:2} [check] tryActionLock {
2014-03-14 07:08:02.883: [ora.LISTENER.lsnr][6105] {0:7:2} [check] (:CLSN00009:)Utils:execCmd abort request: killing /oracle/app/crs/bin/lsnrctl
2014-03-14 07:08:02.914: [    AGFW][2314] {1:18767:989} Agent received the message: AGENT_HB[Engine] ID 12293:2510237
2014-03-14 07:08:03.115: [   AGENT][3352] {0:7:2} UserErrorException: Locale is
2014-03-14 07:08:03.115: [   AGENT][6105] {0:7:2} UserErrorException: Locale is
2014-03-14 07:08:03.279: [ora.LISTENER_SCAN1.lsnr][3352] {0:7:2} [check] clsnUtils::error Exception type=2 string=
CRS-5013: Agent "/oracle/app/crs/bin/oraagent.bin" failed to start process "/oracle/app/crs/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/crs/log/gdgz-ps-esb-db01-p780/agent/crsd/oraagent_grid/oraagent_grid.log"

2014-03-14 07:08:03.280: [ora.LISTENER.lsnr][6105] {0:7:2} [check] clsnUtils::error Exception type=2 string=
CRS-5014: Agent "/oracle/app/crs/bin/oraagent.bin" timed out starting process "/oracle/app/crs/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/oracle/app/crs/log/gdgz-ps-esb-db01-p780/agent/crsd/oraagent_grid/oraagent_grid.log"

2014-03-14 07:08:03.316: [    AGFW][5830] {0:7:2} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.asm gdgz-ps-esb-db01-p780 1
2014-03-14 07:08:03.316: [ora.asm][5830] {0:7:2} [check] clsn_agent::abort {
2014-03-14 07:08:03.453: [ora.asm][5830] {0:7:2} [check] abort {
2014-03-14 07:08:03.453: [ora.asm][5830] {0:7:2} [check] abort command: check
2014-03-14 07:08:03.488: [ora.asm][5830] {0:7:2} [check] tryActionLock {
2014-03-14 07:08:03.537: [   AGENT][6105] {0:7:2} UserErrorException: Locale is
2014-03-14 07:08:03.537: [ora.LISTENER.lsnr][6105] {0:7:2} [check] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.LISTENER.lsnr check" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted

2014-03-14 07:08:03.622: [ora.LISTENER_SCAN1.lsnr][3352] {0:7:2} [check] clsn_agent::check: Exception SclsProcessSpawnException
2014-03-14 07:08:03.693: [    AGFW][2314] {0:7:2} ora.LISTENER_SCAN1.lsnr 1 1 state changed from: ONLINE to: UNKNOWN
2014-03-14 07:08:03.693: [    AGFW][2314] {0:7:2} ora.LISTENER_SCAN1.lsnr 1 1 would be continued to monitored!
2014-03-14 07:08:03.724: [    AGFW][2314] {0:7:38} Generating new Tint for unplanned state change. Original Tint: {0:7:2}
2014-03-14 07:08:03.724: [    AGFW][2314] {0:7:38} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:681928
2014-03-14 07:08:05.844: [ora.asm][2057] {0:7:2} [check] clsnUtils::setResAttrib attribute GEN_USR_ORA_INST_NAME value +ASM1
2014-03-14 07:08:06.784: [ora.LISTENER.lsnr][4122] {0:7:2} [check] got lock
2014-03-14 07:08:06.784: [ora.LISTENER.lsnr][4122] {0:7:2} [check] tryActionLock }
2014-03-14 07:08:06.784: [ora.LISTENER.lsnr][4122] {0:7:2} [check] abort  }
2014-03-14 07:08:06.788: [ora.LISTENER.lsnr][4122] {0:7:2} [check] clsn_agent::abort }
2014-03-14 07:08:06.788: [    AGFW][4122] {0:7:2} Command: check for resource: ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1 completed with status: TIMEDOUT
2014-03-14 07:08:06.789: [    AGFW][2314] {0:7:2} ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1 state changed from: ONLINE to: UNKNOWN
2014-03-14 07:08:06.789: [    AGFW][2314] {0:7:2} ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1 would be continued to monitored!
2014-03-14 07:08:06.808: [    AGFW][2314] {0:7:2} ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1 state details has changed from:  to: CHECK TIMED OUT
2014-03-14 07:08:06.808: [    AGFW][2314] {0:7:39} Generating new Tint for unplanned state change. Original Tint: {0:7:2}
2014-03-14 07:08:06.808: [    AGFW][2314] {0:7:39} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:681929
2014-03-14 07:08:08.114: [ora.asm][2057] {0:7:2} [check] setResAttrib: attr GEN_USR_ORA_INST_NAME@SERVERNAME(gdgz-ps-esb-db01-p780) clsagfw attribVal +ASM1 clscrs tmpAttrValue +ASM1
2014-03-14 07:08:09.138: [ora.asm][2057] {0:7:2} [check] AsmProxyAgent::check clsagfw_res_status 0
2014-03-14 07:08:11.488: [ora.asm][5830] {0:7:2} [check] got lock
2014-03-14 07:08:11.488: [ora.asm][5830] {0:7:2} [check] tryActionLock }
2014-03-14 07:08:11.488: [ora.asm][5830] {0:7:2} [check] abort  }
2014-03-14 07:08:11.499: [ora.asm][5830] {0:7:2} [check] clsn_agent::abort }
2014-03-14 07:08:11.499: [    AGFW][5830] {0:7:2} Command: check for resource: ora.asm gdgz-ps-esb-db01-p780 1 completed with status: TIMEDOUT
2014-03-14 07:08:11.499: [    AGFW][2314] {0:7:2} ora.asm gdgz-ps-esb-db01-p780 1 state changed from: ONLINE to: UNKNOWN
2014-03-14 07:08:11.499: [    AGFW][2314] {0:7:2} ora.asm gdgz-ps-esb-db01-p780 1 would be continued to monitored!
2014-03-14 07:08:11.521: [    AGFW][2314] {0:7:2} ora.asm gdgz-ps-esb-db01-p780 1 state details has changed from: Started to: CHECK TIMED OUT
2014-03-14 07:08:11.521: [    AGFW][2314] {0:7:40} Generating new Tint for unplanned state change. Original Tint: {0:7:2}
2014-03-14 07:08:11.521: [    AGFW][2314] {0:7:40} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:681930
2014-03-14 07:08:12.362: [ USRTHRD][3104] ClusterSubscriber::SubscriberWorker::InternalClusterSubscriber::handleEventCBexecuting for reason 1



2014-03-14 07:08:27.084: [ora.LISTENER.lsnr][5830] {0:7:2} [check] (:CLSN00010:)The command completed successfully
2014-03-14 07:08:27.097: [ora.LISTENER.lsnr][5830] {0:7:2} [check] (:CLSN00010:)
2014-03-14 07:08:27.098: [ora.LISTENER.lsnr][5830] {0:7:2} [check] execCmd ret = 0
2014-03-14 07:08:27.098: [ora.LISTENER.lsnr][5830] {0:7:2} [check] return val (clsagfwret) = 0
2014-03-14 07:08:27.098: [ora.LISTENER.lsnr][5830] {0:7:2} [check] LsnrAgent::check }
2014-03-14 07:08:27.108: [    AGFW][2314] {0:7:2} ora.LISTENER.lsnr gdgz-ps-esb-db01-p780 1 state changed from: UNKNOWN to: ONLINE
2014-03-14 07:08:27.108: [    AGFW][2314] {0:7:41} Generating new Tint for unplanned state change. Original Tint: {0:7:2}
2014-03-14 07:08:27.108: [    AGFW][2314] {0:7:41} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:681944
2014-03-14 07:08:27.196: [ora.asm][4122] {0:7:2} [check] clsnUtils::setResAttrib attribute GEN_USR_ORA_INST_NAME value +ASM1
2014-03-14 07:08:28.381: [ora.DATA.dg][5573] {0:7:40} [check] ConnectionPool::releaseConnection InstConnection 11cd75f0
2014-03-14 07:08:28.382: [ora.DATA.dg][5573] {0:7:40} [check] DgpAgent::runCheck:clsagfw_res_status 0
2014-03-14 07:08:28.382: [    AGFW][2314] {0:7:40} Agent sending last reply for: RESOURCE_PROBE[ora.DATA.dg gdgz-ps-esb-db01-p780 1] ID 4097:2510260
2014-03-14 07:08:32.660: [ora.CRSDG.dg][5088] {0:7:2} [check] InstConnection:connectInt:is up
2014-03-14 07:08:33.541: [ora.CRSDG.dg][5088] {0:7:2} [check] ConnectionPool::releaseConnection InstConnection 11ccce10
2014-03-14 07:08:33.541: [ora.CRSDG.dg][5088] {0:7:2} [check] DgpAgent::runCheck:clsagfw_res_status 0
2014-03-14 07:08:33.625: [CLSFRAME][1] TM [MultiThread] is changing desired thread # to 7. Current # is 8
2014-03-14 07:08:33.626: [CLSFRAME][6105] {0:7:43} Worker thread is exiting in TM [MultiThread] to meet the desired count of 7. New count is 7
2014-03-14 07:08:35.377: [ora.CRSDG.dg][3602] {0:7:2} [check] did not get lock
2014-03-14 07:08:35.377: [ora.CRSDG.dg][3602] {0:7:2} [check] tryActionLock }
2014-03-14 07:08:35.414: [ora.CRSDG.dg][3602] {0:7:2} [check] clsn_agent::abort: Exception LockAcquireTimeoutException
2014-03-14 07:08:35.414: [ora.CRSDG.dg][3602] {0:7:2} [check] clsn_agent::abort, agent exiting }
2014-03-14 07:08:35.453: [    AGFW][3602] {0:7:2} Agent is exiting with exit code: -1



从这里看应该是你的listener应该值在7:08的时候发生了一次震荡导致public ip在监听中丢失，至于震荡的具体原因，需要结合系统和其他crs日志进行分析该震荡应该是由于root 46727224        1   0   Mar 14      -  0:00 /oracle/app/crs/bin/crsd.bin reboot的重启有一定关系
建议可以考虑在系统负载轻的时候，尝试kill crsd.bin让其再次重启，观察看看效果

travel.liu

你在节点2上 ps -ef | grep d.bin 看下
以下是11.2.0.3的正常进程状态

1. root      3188     1  2 15:53 ?        00:00:17 /u01/11.2.0/grid/bin/ohasd.bin reboot
   
2. grid      3303     1  0 15:53 ?        00:00:08 /u01/11.2.0/grid/bin/oraagent.bin
   
3. grid      3315     1  0 15:53 ?        00:00:00 /u01/11.2.0/grid/bin/mdnsd.bin
   
4. grid      3325     1  0 15:53 ?        00:00:01 /u01/11.2.0/grid/bin/gpnpd.bin
   
5. root      3336     1  0 15:53 ?        00:00:07 /u01/11.2.0/grid/bin/orarootagent.bin
   
6. grid      3347     1  0 15:53 ?        00:00:03 /u01/11.2.0/grid/bin/gipcd.bin
   
7. root      3353     1  2 15:53 ?        00:00:23 /u01/11.2.0/grid/bin/osysmond.bin
   
8. root      3365     1  0 15:53 ?        00:00:01 /u01/11.2.0/grid/bin/cssdmonitor
   
9. root      3383     1  0 15:53 ?        00:00:02 /u01/11.2.0/grid/bin/cssdagent
   
10. grid      3395     1  1 15:53 ?        00:00:09 /u01/11.2.0/grid/bin/ocssd.bin
    
11. root      3425     1  0 15:53 ?        00:00:03 /u01/11.2.0/grid/bin/ologgerd -M -d /u01/11.2.0/grid/crf/db/rac1
    
12. root      3464     1  1 15:53 ?        00:00:09 /u01/11.2.0/grid/bin/octssd.bin reboot
    
13. grid      3487     1  1 15:53 ?        00:00:10 /u01/11.2.0/grid/bin/evmd.bin
    
14. root      3622     1  3 15:54 ?        00:00:27 /u01/11.2.0/grid/bin/crsd.bin reboot
    
15. grid      3691  3487  0 15:54 ?        00:00:00 /u01/11.2.0/grid/bin/evmlogger.bin -o /u01/11.2.0/grid/evm/log/evmlogger.info -l /u01/11.2.0/grid/evm/log/evmlogger.log
    
16. grid      3733     1  0 15:54 ?        00:00:06 /u01/11.2.0/grid/bin/oraagent.bin
    
17. root      3743     1  2 15:54 ?        00:00:19 /u01/11.2.0/grid/bin/orarootagent.bin
    
18. grid      3858     1  0 15:54 ?        00:00:00 /u01/11.2.0/grid/bin/scriptagent.bin
    
19. grid      3940     1  0 15:54 ?        00:00:00 /u01/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
    
20. grid      3954     1  0 15:54 ?        00:00:00 /u01/11.2.0/grid/bin/tnslsnr LISTENER -inherit
    
21. oracle    3986     1  1 15:54 ?        00:00:08 /u01/11.2.0/grid/bin/oraagent.bin
    
22. root      4686  3152  0 16:06 pts/1    00:00:00 grep d.bin

复制代码

蜗牛

OSWatch中的当时vmstat日志

蜗牛

节点二状态是正常的：
$  ps -ef | grep d.bin
    root  6160394        1   1   Nov 16      - 2086:40 /oracle/app/crs/bin/ohasd.bin reboot
    grid  6619362        1   0   Nov 16      - 132:12 /oracle/app/crs/bin/gpnpd.bin
    root  7143648        1   1   Nov 26      - 2801:59 /oracle/app/crs/bin/crsd.bin reboot
    grid  7471350        1   0   Nov 16      -  2:51 /oracle/app/crs/bin/mdnsd.bin
    grid  8126652        1   0   Nov 16      - 1903:27 /oracle/app/crs/bin/gipcd.bin
    grid  9830450        1   0   Nov 26      - 1611:11 /oracle/app/crs/bin/evmd.bin
    root 13828190        1   0   Nov 26      - 1717:38 /oracle/app/crs/bin/octssd.bin
    grid 21364908 36044948   0 16:18:05  pts/1  0:00 grep d.bin
    grid 24641786 12648566   2   Nov 26      - 3597:31 /oracle/app/crs/bin/ocssd.bin

travel.liu

系统负载比较高！这个机器还是微分区的
pc  消耗物理处理器的数目。只在使用共享处理器运行的分区显示（只在微分区环境中显示）
ec  消耗授权容量的百分比。只在使用共享处理器运行的分区显示（只在微分区环境中显示）

1. zzz ***Fri Mar 14 07:07:08 GMT+08:00 2014
   
2. 
   
3. System configuration: lcpu=16 mem=32768MB ent=4.00
   
4. 
   
5. kthr    memory              page              faults              cpu         
   
6. ----- ----------- ------------------------ ------------ -----------------------
   
7. r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
   
8. 2 19 9271671  5405   0 438 2383 2270 2270   0 7491 69727 21737 36  9 42 13  2.67  66.7
   
9. 2 17 9277873  3081   0 205 4129 6310 6311   0 8033 70028 22515 38 11 41 11  2.84  71.1
   
10. 3 27 9275577 10133   0 246 5016 2836 2836   0 8916 72609 25018 40 11 33 16  2.91  72.7
    
11. zzz ***Fri Mar 14 07:07:19 GMT+08:00 2014
    
12. 
    
13. System configuration: lcpu=16 mem=32768MB ent=4.00
    
14. 
    
15. kthr    memory              page              faults              cpu         
    
16. ----- ----------- ------------------------ ------------ -----------------------
    
17. r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
    
18. 3 22 9306734 12855   0 145 4163 2481 2481   0 7362 61736 20302 26  9 53 12  2.13  53.4
    
19. 1 16 9317612  2998   0 152 1180 3347 3347   0 7137 63813 20834 32  9 45 14  2.37  59.3
    
20. 0 27 9321338  3012   0   0 3763 3894 3894   0 7811 60712 22373 17  8 53 21  1.72  42.9
    
21. zzz ***Fri Mar 14 07:08:11 GMT+08:00 2014
    
22. 
    
23. System configuration: lcpu=16 mem=32768MB ent=4.00
    
24. 
    
25. kthr    memory              page              faults              cpu         
    
26. ----- ----------- ------------------------ ------------ -----------------------
    
27. r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
    
28. 6 22 9375642 63708   0 143   0   0    0   0 6838 56665 23709 52  7 30 11  3.29  82.3
    
29. 6 28 9375802 63440   0  78   0   0    0   0 5804 51272 21176 42  5 40 12  2.81  70.3
    
30. 2 26 9375919 63278   0  35   0   0    0   0 5552 59807 19973 28  6 45 21  2.19  54.7
    
31. zzz ***Fri Mar 14 07:08:25 GMT+08:00 2014
    
32. 
    
33. System configuration: lcpu=16 mem=32768MB ent=4.00
    
34. 
    
35. kthr    memory              page              faults              cpu         
    
36. ----- ----------- ------------------------ ------------ -----------------------
    
37. r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
    
38. 11 14 9413395 15620   0 707   0   0    0   0 6997 82487 21825 64 12 16  8  3.79  94.7
    
39. 7 13 9407597 18168   0 639   0   0    0   0 6062 79402 20079 57 11 25  7  3.53  88.1
    
40. 9 14 9409496 13235   0 552   0   0    0   0 6252 65025 19924 56  7 30  7  3.40  85.1
    
41. zzz ***Fri Mar 14 07:08:32 GMT+08:00 2014
    
42. 
    
43. System configuration: lcpu=16 mem=32768MB ent=4.00
    
44. 
    
45. kthr    memory              page              faults              cpu         
    
46. ----- ----------- ------------------------ ------------ -----------------------
    
47. r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
    
48. 8 10 9419914  4587   0 564 4377 7035 10042   0 8222 80437 24056 57 12 14 17  3.56  89.1
    
49. 7  2 9414540 10552   0 1531 2071 1131 1476   0 7610 68948 21092 51  9 30 10  3.27  81.8
    
50. 4  4 9414343 10002   0 745   0   0    0   0 6419 63686 19852 53  7 33  7  3.30  82.5

复制代码