ORA-07445 lmebucp
有一个客户找到我们,说他们是数据库启动之时报的错误和数据库不能open 报ORA-7445lmebucp错类似,让我们对其进行恢复支持,通过分析确定客户数据库版本为12.2.0.1Oracle Database12c Enterprise Edition Release 12.2.0.1.0 - 64bit ProductioPL/SQL Release12.2.0.1.0 - ProductionCORE 12.2.0.1.0ProductionTNS for Linux: Version12.2.0.1.0 - ProductionNLSRTL Version12.2.0.1.0 - Production
alert日志报错
--最初报错2020-05-11T03:43:06.787164+08:00Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639048):ORA-00600: 内部错误代码, 参数:, , [], [], [], [], [], [], [], [], [], []Incident detailsin: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639048/orcl_m004_3253_i639048.trcUse ADRCI orSupport Workbench to package the incident.See Note 411.1at My Oracle Support for error and packaging details.2020-05-11T03:43:14.250993+08:00Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639049):ORA-00600: 内部错误代码, 参数:, , [], [], [], [], [], [], [], [], [], []Incident detailsin: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639049/orcl_m004_3253_i639049.trc2020-05-11T03:43:21.286310+08:00Use ADRCI orSupport Workbench to package the incident.See Note 411.1at My Oracle Support for error and packaging details.Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639050):ORA-00600: 内部错误代码, 参数:, , [], [], [], [], [], [], [], [], [], []ORA-06512: 在 line 2Incident detailsin:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639050/orcl_m004_3253_i639050.trc2020-05-11T03:43:28.059048+08:00Use ADRCI orSupport Workbench to package the incident.See Note 411.1at My Oracle Support for error and packaging details.2020-05-11T03:43:28.074681+08:00Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc:ORA-00600: 内部错误代码, 参数:, , [], [], [], [], [], [], [], [], [], []ORA-06512: 在 line 22020-05-11T08:31:22.416087+08:00Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_16511.trc:ORA-12012: 自动执行作业 141 出错ORA-30732: 表中不包含用户可见的列ORA-06512: 在 line 1 ---关闭数据库之后重启报错2020-05-11T09:42:43.234769+08:00ALTER DATABASEOPEN2020-05-11T09:42:44.353085+08:00Ping without logforce is disabled:instancemounted in exclusive mode.Endian type of dictionaryset to little2020-05-11T09:42:44.660388+08:00TT00: GapManager starting (PID:31134)2020-05-11T09:42:45.180876+08:00Thread 1 openedat log sequence 78596Currentlog# 2 seq# 78596 mem# 0: /u01/app/oracle/oradata/orcl/redo02.logSuccessful open of redo thread 12020-05-11T09:42:45.181357+08:00MTTR advisory isdisabled because FAST_START_MTTR_TARGET is not setException Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_31125.trc (incident=646445):ORA-07445:exception encountered:coredumpIncident detailsin:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_646445/orcl_ora_31125_i646445.trcUse ADRCI orSupport Workbench to package the incident.See Note 411.1at My Oracle Support for error and packaging details.2020-05-11T09:42:46.070049+08:00*****************************************************************An internalroutine has requested a dump of selected redo.This usuallyhappens following a specific internal error, whenanalysis of theredo logs will help Oracle Support with thediagnosis.It isrecommended that you retain all the redo logs generated (byall theinstances) during the past 12 hours, in case additionalredo dumps arerequired to help with the diagnosis.*****************************************************************2020-05-11T09:42:53.344955+08:00InstanceCritical Process (pid: 41, ospid: 31125) died unexpectedlyPMON (ospid:31026): terminating the instance due to error 127522020-05-11T09:42:53.377209+08:00System statedump requested by (instance=1, osid=31026 (PMON)), summary=.System Statedumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_31046_20200511094253.trc2020-05-11T09:42:56.690224+08:00Instanceterminated by PMON, pid = 31026
对启动过程做10046跟踪
PARSING INCURSOR #139821999713040 len=65 dep=1 uid=0 oct=3 lid=0 tim=2200910467hv=1762642493 ad='1bfd79130'sqlid='aps3qh1nhzkjx'select line#, sql_textfrom bootstrap$ where obj# not in (:1, :2)END OF STMTPARSE#139821999713040:c=378,e=378,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=2200910467BINDS#139821999713040: Bind#0oacdty=02mxl=22(22) mxlc=00 mal=00 scl=00 pre=00oacflg=08fl2=1000001 frm=00 csi=00 siz=24 off=0kxsbbbfp=7f2ad89fe6c8 bln=22avl=02flg=05value=59 Bind#1oacdty=02mxl=22(22) mxlc=00 mal=00 scl=00 pre=00oacflg=08fl2=1000001 frm=00 csi=00 siz=24 off=0kxsbbbfp=7f2ad89fe698 bln=24avl=06flg=05value=4294967295EXEC#139821999713040:c=219,e=658,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=867914364,tim=2200911218WAIT#139821999713040: nam='db file sequential read' ela= 9 file#=1 block#=520blocks=1 obj#=59 tim=2200911300WAIT#139821999713040: nam='db file scattered read' ela= 19 file#=1 block#=521blocks=3 obj#=59 tim=2200911559FETCH#139821999713040:c=404,e=404,p=4,cr=5,cu=0,mis=0,r=0,dep=1,og=4,plh=867914364,tim=2200911646STAT#139821999713040 id=1 cnt=0 pid=0 pos=1 obj=59 op='TABLE ACCESS FULLBOOTSTRAP$ (cr=5 pr=4 pw=0 str=1 time=406 us)' ***2020-05-11T12:25:30.977832+08:00Exception 2020-05-11T12:25:31.026914+08:00Incident 718445created, dump file:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_718445/orcl_ora_14324_i718445.trcORA-07445:exception encountered:core dump
根据该报错,可以大概定位数据库重启之后报ORA-07445 lmebucp()+34错误不能正常启动是由于bootstrap$表异常导致.
BBED> pkcvfhrdbub4kcvfhrdb @96 0x00400208 BBED> set block 523 BLOCK# 523 BBED> map File:F:\temp\12.2\1\system01.dbf (0) Block:523 Dba:0x00000000------------------------------------------------------------ KTB DataBlock (Table/Cluster) structkcbh, 20bytes @0 structktbbh, 48bytes @20 structkdbh, 14bytes @68 structkdbt, 4bytes @82 sb2kdbr @86 ub1freespace @126 ub1rowdata @1141 ub4tailchk @8188 BBED> p*kdbrrowdata-------------ub1rowdata @7572 0x3c BBED> x /rnncrowdata @7572-------------flag@7572: 0x3c(KDRHFL, KDRHFF, KDRHFD, KDRHFH)lock@7573: 0x01cols@7574: 0
通过bbed定位rootdba,然后dump相关block,随机找一条记录,确认bootstrap表无后效记录.但是该数据库在重启之前已经报了ORA-600 kkpo_rcinfo_defstg:delseg和ORA-30732错误,很可能还有其他的基表异常.通过先修复bootstrap$记录,然后根据该表中记录分析其他相关表记录,最终确定tab$中记录也异常,通过bbed 批量循环修复方法,对其进行恢复,open数据库,可以验证数据没有问题.至此该问题解决,但是没有找出来故障原因(是人为破坏【直接人工删除】,还是某种工具带入恶意脚本导致,类似:plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案,亦或者是数据库安装介质带有恶意程序,类似:警告：互联网中有oracle介质被注入恶意程序导致—ORA-60016703)
页:
[1]