ORACLE OMS啟動失敗之BEA-300040 I/O error while writing node manager status

        今天處理一個oracle優化管理工具oms啟動異常的問題。oms問題處理之前，oms所在服務器上的一套asm和db所用

的審計目錄文件系統inode耗盡導致asm和db異常宕機，經清理審計目錄的日志后asm和db正常啟動，但是oms啟動異常。

一、OMS啟動異常分析

1、OMS啟動異常信息

[oracle@orcl bin]$ ./emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 4  

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Starting Oracle Management Server...

Starting WebTier...

WebTier Successfully Started

Oracle Management Server Could Not Be Started

Check EM Server log file for details: /u01/app/oracle/oms/domains/GCDomain/servers/ORCL_OMS1/logs/ORCL_OMS1.out

Oracle Management Server is Down

[oracle@ orcl bin]$ 

2、查看 ORCL_OMS1.out日志

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.audit.AuditServiceImpl registerInternal

WARNING: Cannot register to audit service for component "JPS".

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.policystore.ldap.LdapPolicyStore initial

INFO: Container is not ready at the point null

<Mar 19, 2020 5:55:53 PM CST> <Error> <NodeManager> <BEA-300040> <I/O error while writing node manager status change message.

java.io.IOException: No space left on device

        at java.io.UnixFileSystem.createFileExclusively(Native Method)

        at java.io.File.checkAndCreate(File.java:1705)

        at java.io.File.createTempFile0(File.java:1726)

        at java.io.File.createTempFile(File.java:1803)

        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)

        Truncated. see log file for complete stacktrace

> 

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <Waiting for the process to die: 6906>

<Mar 19, 2020 5:55:54 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

3、根據.out日志提示，跟之前asm和db啟動異常報錯一致，也是之前oracle數據庫審計日志過多導致文件系統inode耗盡

其他進程無法寫入導致。

二、問題處理

1、對于oms來說，單純的重啟oms無法解決問題，因為nodemanager進程未正常停止

2、使用ps -ef|grep java，把oms相關的java進程殺掉kill -9 java_process_number

3、再次嘗試啟動oms進程，成功啟動