oracle11.2.0.4 rac搭建中的crs-4000錯誤解析

   在系統環境為rhel6.5的服務器上，搭建數據庫版本為oracle11.2.0.4的兩節點的RAC，安裝GRID時遭遇如下錯誤：

（注：vote、data和閃回盤都在存儲設備上）

ASM created and started successfully.

Disk Group VOTE mounted successfully.

clscfg: -install mode specified

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Failed to create voting files on disk group VOTE.

Change to configuration failed, but was successfully rolled back.

CRS-4000: Command Replace failed, or completed with errors.

Voting file add failed

Failed to add voting disks at /g01/11ggrid/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6930.

/g01/11ggrid/app/11.2.0/grid/perl/bin/perl -I/g01/11ggrid/app/11.2.0/grid/perl/lib -I/g01/11ggrid/app/11.2.0/grid/crs/install /g01/11ggrid/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

這個錯誤是說無法對磁盤組VOTE創建投票文件。改變配置失敗，但成功回滾。

投票文件添加失敗，未能在/g01/11ggrid/app/11.2.0/grid/crs/install添加表決磁盤。

碰到這個錯誤還是頭一遭，很是詭異。因為之前使用ASMLIB掛盤都沒有出現任何問題，但安裝卻出現這個問題，沒法繼續進行下去。

檢查ASM的 alert.log日志發現：

ERROR: Could not create voting files. It spans across 161 AUs (max supported is 64 AUs)

ERROR: Voting file allocation failed for group VOTE

Errors in file /g01/11ggrid/app/11.2.0/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7321.trc:

ORA-15303: Voting files could not be created in diskgroup VOTE due to small Allocation Unit size

查看對應GRID的仲裁盤對應的DM

#multipath -ll

sc_vote (360a980003830302d712b46586276736b) dm-11  NETAPP,LUN

size=5.0G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=2 status=active

  `- 17:0:0:0 sdax 67:16  active ready  running

繼續檢查一下DM-11塊的大小問題，發現一些信息：

# cd /sys/block/dm-11/queue

#  more *block_size

::::::::::::::

logical_block_size

::::::::::::::

512

::::::::::::::

physical_block_size

::::::::::::::

4096

究其原因是它表現出不同的邏輯和物理塊大小。對于大多數的情況下，物理和邏輯塊的大小是相同的“512”。

那么如何解決這個問題呢？查詢ORACLE的官方網站，給出的解決方案是：

1. Oracle bug 11780656 is already fixed in 11.2.0.3 with compatible.asm = 11.2.0.3 but OUI does not allow to specify "compatible.asm" attribute to create OCRVOTE diskgroup.

   - Bug 11780656  - ASM MANAGED VOTING FILES CANNOT BE MORE THAN 64X AU SIZE

2. Oracle bug 13999609 indicates that ASMLib driver only works with the expectation that logical block size and physical block size are 512/512 bytes.   

   - Bug 13999609  - PHYSICAL BLOCK SIZE REPORTED CAN CAUSE ISSUES WITH 10G DATABASES

SOLUTION

1] Possible workaround is to use '/dev/oracleasm/disks/*' path instead of "ORCL:*" when creating OCRVOTE diskgroup in OUI.

OR

2] Install the new “oracleasm-support-2.1.8-1” ASMLIB RPM package (which contains the permanent fix) as described in note 1500460.1

根據以上提示，我們可以在安裝oracleasm-support-2.1.8-1的包，或者也可以在選擇ASM盤時，改用

/dev/oracleasm/disks/*的路徑來解決此問題。根據以上提示進行解決，發現還是無法成功裝上GRID.

反復幾次，沒有找到解決的辦法。由于后端使用的是NETAPP存儲，所以就去NETAPP的官網查找此問題的CASE案例。發現其給出如下解決方案：

For details and caveats regarding this workaround, see the Oracle Alert.

Additionally, Oracle has provided a patch and configuration parameter to enable ASMlib to continue to function using the correct logical block size.

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'

(oracle 設置) 

NetApp has also provided a workaround in versions 8.0.5, 8.1.3 and 8.2 of Data ONTAP 7-Mode. The workaround allows specified LUNs to continue to not report the logical blocks per physical block value. This work around should only be applied to LUNs used by Oracle ASMlib with the symptoms described in this article.

From the Data ONTAP 7-Mode CLI, enter the following commands:

> lun set report-physical-size <path> disable（netapp存儲設置）

根據此提示，然后進入/etc/sysconfig目錄，修改其下的oracleasm文件，將其中的

[oracle@rac1 ~]$ cat /etc/sysconfig/oracleasm

#

# This is a configuration file for automatic loading of the Oracle

# Automatic Storage Management library kernel driver.  It is generated

# By running /etc/init.d/oracleasm configure.  Please use that method

# to modify this file

#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE=""

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size

# reported by the underlying disk instead of the physical. The default

# is 'false'

ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

將ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false改為true

并對應修改NETAPP上的存儲設置參數

（注：Oracle提供了配置參數來啟用ASMLib繼續使用正確的邏輯塊的大小的功能來避免這個問題。）

通過如上的處理，該問題得以解決，GRID最后安裝成功。