Oracle 關閉(shutdown immediate)時hang住

昨天晚上生產的兩套10.2.0.4的數據庫修改了參數，需要重啟。在發出shutdown immediate命令后等了大概10分鐘的時間，數據庫還沒有down下來。檢查后臺alert日志，發現從開始shutdown到最后只輸出幾條日志，其中最后一條日志是：SHUTDOWN: Active processes prevent shutdown operation。

圖為在虛擬機上還原場景時的截圖。

開一個新的會話連接顯示已連接，但無法查視圖，又提示未連接。再次執行shutdown immediate命令得到報錯如下：

立馬百度了一下這個問題，隨便點開一個去看，說是因為在sqlplus / as sysdba 連接的情況下又執行host命令導致的。如下圖這種情況

馬上想到我這邊也是這種情況，趕緊把相應的會話從操作系統中kill掉。kill掉之后，數據庫又正常關閉了（這是在虛擬機上的情況）。

在實際的生產中卻是，把會話kill掉之后又等了幾分種，還是沒有返應，于是只得強制shutdown abort了。好在業務都是提前停了，不過在執行這條命令是還是有些忐忑的，就怕數據庫起不來。

在看貼子的時候，作者提到了他在停庫前做了切換日志，歸檔和checkpoint的操作來以防萬一，我覺得這也是一個可取的地方，以后在停生產庫時不應該業務停了就立馬停數據庫，也應該做這些操作來保護數據。

俗話說禍不單行，真是沒錯，就在第一個數據庫成功重啟后，在shutdown第二套數據庫也hang住了，但這次報錯與上次還不一樣！！！（我的小心臟啊。。）

alert輸出信息如下：

從輸出信息可以了解到有進行還沒有執行完，但業務都已經停了，會是什么進行呢。由于還有其他事情，就沒去管是什么進行，就直接kill掉了，但數據庫還是停不下來啊。

最后還是只得shutdown abort了，還是依然的忐忑。最后平安無事。

這個問題最后也沒有在虛擬機上模擬出來，始終不知道是什么。

總結一下：

1、自己給自己挖坑的滋味真是不好受啊。

2、關閉數據庫前做一些切換日志文件、歸檔和checkpoint的操作，來保證數據的安全性

3、關閉數據庫前一定做好檢查，是否有未解決的事務、JOB、或其他進程，避免出現類似第二種的情況

4、要有一顆強大的內心，相信自己:)

事后在MOS上查到了一些相關信息：

MOS文檔

Troubleshooting Shutdown Immediate/Normal Hanging Issues (文檔 ID 1906014.1)

列舉四種shutdown Hang的情況。

第一種情況的原因和給出的解決方案

Shutdown Immediate Hangs / Active Processes Prevent Shutdown (文檔 ID 416658.1)

CAUSE

This is not a bug.

If the DB Control repository is running on the database target against which shutdown immediate was attempted then an incorrect order of events seems used.
You should stop DB Control first to get rid of all connections between DB Control and the repository database and then shutdown the database with 'shutdown immediate'.

Current database sessions may show:

SQL> select SID, USERNAME, PROGRAM from v$session;
SID   USERNAME               PROGRAM
----- ---------------------- ----------------------------------
  243 SYSTEM                 SQL Developer
  246 SYSMAN                 OMS
  247                        oracle@lgiora09 (q001)
  248                        oracle@lgiora09 (q000)
  251 DBSNMP                 emagent@lgiora09 (TNS V1-V3)
  252 SYSMAN                 OMS
  253 SYSMAN                 OMS
  254 DBSNMP                 emagent@lgiora09 (TNS V1-V3)
  255 SYSTEM                 java.exe
  256 SYSMAN                 OMS

Clearly OMS and OEM are connected (Oracle Enterprise Manager Grid Control or DBConsole) via users SYSMAN and DBSNMP.
These sessions should be de-activated (that is to log off any OEM, OMS, SYSMAN and DBSNMP) before the shutdown immediate is attempted.

Oracle Enterprise Manager, Grid Control, Dbconsole and agents keep doing internal processing.
This may include a few PLSQL notification procedures running on the database by database control like
BEGIN EMD_NOTIFICATION.QUEUE_READY(:1, :2, :3); END;

As per internal documentation of the shutdown immediate, if there are active calls then it would wait for all the active calls to finish.

SOLUTION

To implement the solution:

1. Given OEM connections are active (SYSMAN and DBSNMP), de-activate these sessions, i.e. by stopping the agent/DBConsole

2. Then shutdown immediate as normal

- OR -

There may be processes still running and holding locks at the time a shutdown is issued. 
Sometimes these are failed jobs or transactions, which are effectively 'zombies', which are not able to receive a signal from Oracle. 


If this occurs, the only way to shutdown the database is by doing: 

sql> 
shutdown abort 
startup restrict 
shutdown normal

The startup does any necessary recovery and cleanup, so that a valid cold backup can be taken afterward. 

If this issue occurs frequently, it would be a good practice to see if there are any active user processes running in v$session or v$process before shutting down the instance. 


If the problem persists, and no apparent user processes are active, you can set this event prior to issuing the shutdown command in order to see what is happening. This will dump a systemstate every 5 minutes while shutdown is hanging 

SQL> 
connect / as sysdba 
alter session set events '10400 trace name context forever, level 1'; 

Then issue the shutdown command.

3. You can kill the sessions preventing shutdown. Refer: Alert Log: Shutdown Waiting for Active Calls to Complete (Doc ID 1039389.6)