RAC

Recommendation for the Real Application Cluster Interconnect and Jumbo Frames

February 9, 2012 Architect, oracle, RAC, software 2 comments

Recommendation for the Real Application Cluster Interconnect and Jumbo Frames

Applies to:


Oracle Server – Enterprise Edition – Version: 9.2.0.1 to 11.2.0.0 – Release: 9.2 to 11.2
Information in this document applies to any platform.
Oracle Server Enterprise Edition – Version: 9.2.0.1 to 11.2
Purpose

This note covers the current recommendation for the Real Application Cluster Interconnect and Jumbo Frames
Scope and Application

This article points out the issues surrounding Ethernet Jumbo Frame usage for the Oracle Real Application Cluster (RAC) Interconnect. In Oracle Real Application Clusters, the Cluster Interconnect is designed to run on a dedicated, or stand-alone network. The Interconnect is designed to carry the communication between the nodes in the Cluster needed to check for the Clusters condition and to synchronize the various memory caches used by the database.

Ethernet is a widely used networking technology for Cluster Interconnects. Ethernet’s variable frame size of 46-1500 bytes is the transfer unit between the all Ethernet participants, such as the hosts and switches. The upper bound, in this case 1500, is called MTU (Maximum Transmission Unit). When an application sends a message greater than 1500 bytes (MTU), it is fragmented into 1500 byte, or smaller, frames from one end-point to another. In Oracle RAC, the setting of DB_BLOCK_SIZE multiplied by the MULTI_BLOCK_READ_COUNT determines the maximum size of a message for the Global Cache and the PARALLEL_EXECUTION_MESSAGE_SIZE determines the maximum size of a message used in Parallel Query. These message sizes can range from 2K to 64K or more, and hence will get fragmented more so with a lower/default MTU.

Jumbo Frames introduces the ability for an Ethernet frame to exceed its IEEE 802 specified Maximum Transfer Unit of 1500 bytes up to a maximum of 9000 bytes. Even though Jumbo Frames is widely available in most NICs and data-center class managed switches it is not an IEEE approved standard. While the benefits are clear, Jumbo Frames interoperability is not guaranteed with some existing networking devices. Though Jumbo Frames can be implemented for private Cluster Interconnects, it requires very careful configuration and testing to realize its benefits. In many cases, failures or inconsistencies can occur due to incorrect setup, bugs in the driver or switch software, which can result in sub-optimal performance and network errors.
Recommendation for the Real Application Cluster Interconnect and Jumbo Frames

Configuration

In order to make Jumbo Frames work properly for a Cluster Interconnect network, careful configuration in the host, its Network Interface Card and switch level is required:
The host’s network adapter must be configured with a persistent MTU size of 9000 (which will survive reboots).
For example, ifconfig -mtu 9000 followed by ifconfig -a to show the setting completed.
Certain NIC’s require additional hardware configuration.
For example, some Intel NIC’s require special descriptors and buffers to be configured for Jumbo Frames to work properly.
The LAN switches must also be properly configured to increase the MTU for Jumbo Frame support. Ensure the changes made are permanent (survives a power cycle) and that both “Jumbo” refer to same size, recommended 9000 (some switches do not support this size).

Because of the lack of standards with Jumbo Frames the interoperability between switches can be problematic and requires advanced networking skills to troubleshoot.
Remember that the smallest MTU used by any device in a given network path determines the maximum MTU (the MTU ceiling) for all traffic travelling along that path.
Failing to properly set these parameters in all nodes of the Cluster and Switches can result in unpredictable errors as well as a degradation in performance.
Testing

Request your network and system administrator along with vendors to fully test the configuration using standard tools such as SPRAY or NETCAT and show that there is an improvement not degradation when using Jumbo Frames. Other basic ways to check it’s configured correctly on Linux/Unix are using:

Traceroute: Notice the 9000 packet goes through with no error, while the 9001 fails, this is a correct configuration that supports a message of up to 9000 bytes with no fragmentation:

[node01] $ traceroute -F node02-priv 9000
traceroute to node02-priv (10.10.10.2), 30 hops max, 9000 byte packets
1 node02-priv (10.10.10.2) 0.232 ms 0.176 ms 0.160 ms

[node01] $ traceroute -F node02-priv 9001
traceroute to node02-priv (10.10.10.2), 30 hops max, 9001 byte packets
traceroute: sendto: Message too long
1 traceroute: wrote node02-priv 9001 chars, ret=-1
* Note: Due to Oracle Bugzilla 7182 (must have logon privileges) — also known as RedHat Bugzilla 464044 — older than EL4.7 traceroute may not work correctly for this purpose.
* Note: Some versions of tracroute, e.g. traceroute 2.0.1 shipped with EL5, add the header size on top of what is specified when using the -F flag (same as ping behavior below). Newer versions of traceroute, like 2.0.14 (shipped with OL6) have the old behavior of traceroute version 1 (size of packet is exactly as what is specified with the -F flag).

Ping: With ping we have to take into account an overhead of about 28 bytes per packet, so 8972 bytes go through with no errors, while 8973 fail, this is a correct configuration that supports a message of up to 9000 bytes with no fragmentation:
[node01]$ ping -c 2 -M do -s 8972 node02-priv
PING node02-priv (10.10.10.2) 1472(1500) bytes of data.
1480 bytes from node02-priv (10.10.10.2): icmp_seq=0 ttl=64 time=0.220 ms
1480 bytes from node02-priv (10.10.10.2): icmp_seq=1 ttl=64 time=0.197 ms

[node01]$ ping -c 2 -M do -s 8973 node02-priv
From node02-priv (10.10.10.1) icmp_seq=0 Frag needed and DF set (mtu = 9000)
From node02-priv (10.10.10.1) icmp_seq=0 Frag needed and DF set (mtu = 9000)
— node02-priv ping statistics —
0 packets transmitted, 0 received, +2 errors
* Note: Ping reports fragmentation errors, due to exceeding the MTU size.
Performance

For RAC Interconnect traffic, devices correctly configured for Jumbo Frame improves performance by reducing the TCP, UDP, and Ethernet overhead that occurs when large messages have to be broken up into the smaller frames of standard Ethernet. Because one larger packet can be sent, inter-packet latency between various smaller packets is eliminated. The increase in performance is most noticeable in scenarios requiring high throughput and bandwidth and when systems are CPU bound.

When using Jumbo Frames, fewer buffer transfers are required which is part of the reduction for fragmentation and reassembly in the IP stack, and thus has an impact in reducing the latency of a an Oracle block transfer.

As illustrated in the configuration section, any incorrect setup may prevent instances from starting up or can have a very negative effect on the performance.
Known Bugs

In some versions of Linux there are specific bugs in Intel’s Ethernet drivers and the UDP code path in conjunction with Jumbo Frames that could affect the performance. Check for and use the latest version of these drivers to be sure you are not running into these older bugs.

The following bugzilla bugs 162197, 125122 are limited to RHEL3.
Recommendation

There is some complexity involved in configuring Jumbo Frames, which is highly hardware and OS specific. The lack of a specific standard may present OS and hardware bugs. Even with these considerations, Oracle recommends using Jumbo Frames for private Cluster Interconnects.

Since there is no official standard for Jumbo Frames, this configuration should be properly load tested by Customers. Any indication of packet loss, socket buffer or DMA overflows, TX and RX error in adapters should be noted and checked with the hardware and operating system vendors.

The recommendation in this Note is strictly for Oracle private interconnect only, it does not applies to other NAS or iSCSI vendor tested and validated Jumbo Frames configured networks.

Oracle VM does not support Jumbo Frame. Refer Oracle VM: Jumbo Frame on Oracle VM (Doc ID 1166925.1) for further information.

Configuring Temporary Tablespaces for RAC Databases for Optimal Performance

January 6, 2012 oracle, RAC No comments

Configuring Temporary Tablespaces for RAC Databases for Optimal Performance
Modified 04-AUG-2010 Type BULLETIN Status ARCHIVED
In this Document
Purpose
Scope and Application
Configuring Temporary Tablespaces for RAC Databases for Optimal Performance
References

Applies to:

Oracle Server – Enterprise Edition – Version: 9.2.0.1 to 11.1.0.6 – Release: 9.2 to 11.1
Information in this document applies to any platform.
***Checked for relevance on 04-Aug-2010***
Purpose

This document discusses issues involved in configuring temporary tablespaces for RAC databases. Also provides best practice recommendations in configuring them for optimal performance.
Scope and Application

Oracle RAC DBAs.
Configuring Temporary Tablespaces for RAC Databases for Optimal Performance

Any DW, OLTP or mixed workload application that uses a lot of temp space for temporary tables, sort segments etc, when running low on temp space, lots of sessions would start waiting on ‘SS enqueue’ and ‘DFS lock handle’ waits. This would cause some severe performance issues. This best practice note for temporary tablespace, explains how this works in RAC environment and offer recommendations.
Space allocated to one instance is managed in the SGA of that instance, and it is not visible to other instances.

Instances do not normally return temp space to the ‘common pool’.

If all the temp space is allocated to instances, and there is no more temp space within an instance, user requests for temp space will cause a request for temp space to be sent to the other instances. The session requesting the space will get the ‘SS enqueue’ for the temporary tablespace and issue a cross instance call (using a CI enqueue) to the other instances (waiting for ‘DFS lock handle’). All inter instance temp space requests will serialize on this ‘CI enqueue, and this can be very expensive.

A heavy query executing in one instance and using lots of temp space might cause all or most of the temp space to be allocated to this instance. This kind of imbalance will lead to increased contention for temp space.

As users on each instance request temp space, space will be allocated to the various instances. During this phase it is possible to get contention on the file space header blocks, and it is recommended to have at least as many temp files as there are instances in the RAC cluster. This normally shows up as ‘buffer busy’ waits and it is different from the ‘SS enqueue’/’DFS lock handle’ wait issue.

Temporary tablespace groups are designed to accommodate very large temp space requirements, beyond the current limits for a single temporary tablespace: 8TB (2k block size) to 128TB (32k block size).

One possible advantage of temporary tablespace groups is that it provides multiple SS enqueues (one per tablespace), but this only shifts the contention to the CI enqueue (only one system wide)

It is easier to share space within a single temporary tablespace, rather than within a temporary tablespace group. If a session starts allocating temp space from a temporary tablespace within a temporary tablespace group, additional space cannot be allocated from another temporary tablespace within the group. With a single temporary tablespace, a session can allocate space across tempfiles.

The following is the recommended best practices for managing temporary tablespace in a RAC environment:
Make sure enough temp space is configured. Due to the way temp space is managed by instance in RAC, it might be useful to allocate a bit extra space compared to similar single instance database.

Isolate heavy or variable temp space users to separate temporary tablespaces. Separating reporting users from OLTP users might be one option.

Monitor the temp space allocation to make sure each instance has enough temp space available and that the temp space is allocated evenly among the instances. The following SQL is used:


select inst_id, tablespace_name, segment_file, total_blocks,
used_blocks, free_blocks, max_used_blocks, max_sort_blocks
from gv$sort_segment;

select inst_id, tablespace_name, blocks_cached, blocks_used
from gv$temp_extent_pool;

select inst_id,tablespace_name, blocks_used, blocks_free
from gv$temp_space_header;

select inst_id,free_requests,freed_extents
from gv$sort_segment;

If temp space allocation between instances has become imbalanced, it might be necessary to manually drop temporary segments from an instance. The following command is used for this:

alter session set events ‘immediate trace name drop_segments level ‘;

See Bug 4882834 for details.

For each temporary tablespace, allocate at least as many temp files as there are instances in the cluster.
References

BUG:4882834 – EXCESSIVE SS AND TS CONTENTION ON NEW 10G CLUSTERS
BUG:6331283 – LONG WAITS ON ‘DFS LOCK HANDLE’

快速升级Oracle 11.2.0.2 RAC到11.2.0.3 on Windows 2008

December 18, 2011 11g, oracle, RAC No comments

基于windows 2008平台升级rac 11.2.0.2.0->11.2.0.3.0

OS环境windows 2008 x64 数据库环境oracle 11g rac ,version 11.2.0.2.0 64bit

SQL> conn / as sysdba
已连接。
SQL> select * from v$version;

BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
PL/SQL Release 11.2.0.2.0 – Production
CORE 11.2.0.2.0 Production
TNS for 64-bit Windows: Version 11.2.0.2.0 – Production
NLSRTL Version 11.2.0.2.0 – Production

1.升级CRS

下载11.2.0.3版本的grid数据库软件,解压到相应目录,此次使用滚动升级,不需要停CRS和database 服务,在升级前备份数据库和CRS。
点击setup.exe来进行安装,选择跳过软件更新,如下图所示:

选择升级Oracle Grid Infrastructure 或Oracle自动存储管理,点击下一步,如下图所示:

选择产品语言,简体中文和英文,点击下一步,如下图所示:

选择要升级的数据库节点,如下图所示:

这里选择grid的安装路径要不同于11.2.0.2 grid的安装路径,如下图所示:

检查先决条件,如下图所示:

问题一,防火墙

问题二:12539000补丁包
此时需要下载p12849789和最新的opatch工具
首先备份以前版本的Opatch,然后将最新的Opatch补丁p6880880解压到grid目录下

C:\app\11.2.0\grid_2\OPatch>opatch version

OPatch Version: 11.2.0.1.9

OPatch succeeded.

在两节点同时需要使用最新的opatch工具,接下来打12539000补丁,使用滚动升级的方式打补丁,首先在节点一上关闭数据库服务
C:\Users\Administrator>sqlplus /nolog

SQL*Plus: Release 11.2.0.2.0 Production on 星期五 12月 16 14:33:55 2011

Copyright (c) 1982, 2010, Oracle. All rights reserved.

SQL> conn / as sysdba
已连接。
SQL> alter system checkpoint local;

系统已更改。

C:\Users\Administrator>srvctl stop instance -d honcho -i honcho1 -o abort

C:\Users\Administrator>crsctl stop crs

使用windows管理工具或者net stop来关闭所有Oracle相关服务

C:\Users\Administrator>cd /

C:\>cd app

C:\app>set ORACLE_HOME=c:\app\11.2.0\grid_2

C:\app>cd 12849789

C:\app\12849789>c:\app\11.2.0\grid_2\OPatch\opatch apply

racle 中间补丁程序安装程序版本 11.2.0.1.9
版权所有 (c) 2011, Oracle Corporation。保留所有权利。

Oracle Home : c:\app\11.2.0\grid_2
Central Inventory : C:\Program Files\Oracle\Inventory
from : n/a
OPatch version : 11.2.0.1.9
OUI version : 11.2.0.2.0
Log file location : c:\app\11.2.0\grid_2\cfgtoollogs\opatch\opatch2011-12-16_14-51-24下午.log

Applying interim patch ‘12849789’ to OH ‘c:\app\11.2.0\grid_2’
Verifying environment and performing prerequisite checks…
Patch 12849789: Optional component(s) missing : [ oracle.ntoledb.odp_net_2, 11.2.0.2.0 ] , [ oracle.ctx, 11.2.0.2.0 ] ,
[ oracle.rdbms.oci, 11.2.0.2.0 ] , [ oracle.rdbms.ic, 11.2.0.2.0 ] , [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle
.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , [ oracle.oraolap, 11.2.0.2.0 ] , [ oracle.precomp.com
mon, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] , [ oracle.odbc.ic, 11.2.0.2.0 ] , [ oracle.ntoledb, 11.2.0.2.0
] , [ oracle.ntoramts, 11.2.0.2.0 ] , [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ] , [ oracle.rdbms.tg4db2, 1
1.2.0.2.0 ] , [ oracle.rdbms.tg4msql, 11.2.0.2.0 ]
All checks passed.

此节点是 Oracle Real Application Cluster 的一部分。
远程节点: ‘rac2’
本地节点: ‘rac1’
请关闭本地系统上在此 ORACLE_HOME 之外运行的 Oracle 实例。
(Oracle 主目录 = ‘c:\app\11.2.0\grid_2’)

本地系统是否已准备打补丁? [y|n]
y
User Responded with: Y
Backing up files…

正在为组件 oracle.rdbms.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.dbscripts, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.plsql, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.rsf.ic, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.crs, 11.2.0.2.0 打补丁…

正在为组件 oracle.network.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.ldap.rsf.ic, 11.2.0.2.0 打补丁…

正在为组件 oracle.ldap.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.precomp.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.rman, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.util, 11.2.0.2.0 打补丁…

正在为组件 oracle.nlsrtl.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.nlsrtl.rsf.ic, 11.2.0.2.0 打补丁…

正在为组件 oracle.xdk.rsf, 11.2.0.2.0 打补丁…

正在为组件 oracle.oraolap.dbscripts, 11.2.0.2.0 打补丁…

正在为组件 oracle.usm.deconfig, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.common, 11.2.0.2.0 打补丁…

正在为组件 oracle.sdo.locator, 11.2.0.2.0 打补丁…

正在为组件 oracle.usm, 11.2.0.2.0 打补丁…

正在为组件 oracle.rdbms.deconfig, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.db, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.deconfig, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.common.cvu, 11.2.0.2.0 打补丁…

正在为组件 oracle.has.cvu, 11.2.0.2.0 打补丁…

本地系统已打补丁。您可以在本地系统上重新启动 Oracle 实例。

在滚动模式下打补丁。

接下来将为节点 ‘rac2’ 打补丁。

请关闭 ‘rac2’ 上在此 ORACLE_HOME 之外运行的 Oracle 实例。
(Oracle 主目录 = ‘c:\app\11.2.0\grid_2’)

该节点是否已准备打补丁? [y|n]

在选择升级第二个节点之前,先开启节点一的服务,同时运行以下脚本:

C:\app\11.2.0\grid_2\BIN>acfsroot.bat install
ACFS-9300: 已找到 ADVM/ACFS 分发文件。
ACFS-9312: 已检测到现有 ADVM/ACFS 安装。
ACFS-9314: 正在删除以前的 ADVM/ACFS 安装。
ACFS-9315: 已成功删除以前的 ADVM/ACFS 组件。
ACFS-9307: 正在安装请求的 ADVM/ACFS 软件。
ACFS-9308: 正在加载已安装的 ADVM/ACFS 驱动程序。
ACFS-9327: 正在验证 ADVM/ACFS 设备。
ACFS-9309: 已验证 ADVM/ACFS 安装的正确性。

然后关闭节点2的所有服务,给节点2打patch

该节点是否已准备打补丁? [y|n]
y
User Responded with: Y
Updating nodes ‘rac2’
Apply-related files are:
FP = “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_files.txt”
DP = “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_dirs.txt”
MP = “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\make_cmds.txt”
RC = “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\remote_cmds.txt”

Instantiating the file “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_files.txt.instantiate
d” by replacing %ORACLE_HOME% in “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_files.txt”
with actual path.
Propagating files to remote nodes…
Instantiating the file “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_dirs.txt.instantiated
” by replacing %ORACLE_HOME% in “c:\app\11.2.0\grid_2\.patch_storage\12849789_Aug_26_2011_22_05_04\rac\copy_dirs.txt” wi
th actual path.
Propagating directories to remote nodes…

节点 ‘rac2’ 已打补丁。您可以在该节点上重新启动 Oracle 实例。

Patch 12849789 successfully applied
Log file location: c:\app\11.2.0\grid_2\cfgtoollogs\opatch\opatch2011-12-16_14-51-24下午.log

OPatch succeeded.

重新检查

忽略域用户这个警告,点击下一步

点击安装,开始升级安装

C:\Users\Administrator>crsctl query crs activeversion
集群上的 Oracle Clusterware 活动版本为 [11.2.0.3.0]

2.升级Database software

下载11.2.0.3数据库软件,并解压到相应目录,采用滚动升级数据库的方式升级,不需要关闭数据库服务。
点击安装目录下的setup.exe开始安装

选择“跳过软件更新”,点击下一步,如下图所示:

选择Oracle Real Application Cluster数据库安装,点击下一步,如下图所示:

选择数据库产品语言——简体中文和英语,现在下一步,如下图所示:

选择安装企业版,点击下一步,如下图所示:

这里也选择一个新的安装路径,点击下一步,如下图所示:

选择忽略该警告,点击下一步,如下图所示:

点击安装,开始安装,如下图所示:

升级数据库实例,可以选择静默升级,也可以选择dbua这个工具来升级,这里选择使用DBUA来升级数据库,点击下一步,如下图所示:

选择要升级的数据库,如下图所示

点击“是”开始升级数据库

升级期间关闭数据库归档功能,点击下一步,如下图所示:

开始升级数据库

查看最终数据库版本信息:

C:\Users\Administrator>crs_stat -t
名称 类型 目标 状态 主机
————————————————————
ora.CRS.dg ora….up.type ONLINE ONLINE rac1
ora.DATA.dg ora….up.type ONLINE ONLINE rac1
ora….ER.lsnr ora….er.type ONLINE ONLINE rac1
ora….N1.lsnr ora….er.type ONLINE ONLINE rac1
ora….VERY.dg ora….up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora.honcho.db ora….se.type ONLINE ONLINE rac1
ora….network ora….rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac2
ora.ons ora.ons.type ONLINE ONLINE rac1
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora….t1.type ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora….t1.type ONLINE ONLINE rac2
ora….ry.acfs ora….fs.type ONLINE ONLINE rac1
ora.scan1.vip ora….ip.type ONLINE ONLINE rac1

C:\Users\Administrator>sqlplus /nolog

SQL*Plus: Release 11.2.0.3.0 Production on 星期五 12月 16 20:08:08 2011

Copyright (c) 1982, 2011, Oracle. All rights reserved.

SQL> conn / as sysdba
已连接。
SQL> select * from v$version;

BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
PL/SQL Release 11.2.0.3.0 – Production
CORE 11.2.0.3.0 Production
TNS for 64-bit Windows: Version 11.2.0.3.0 – Production
NLSRTL Version 11.2.0.3.0 – Production

SQL> set linesize 120
SQL> col comp_name for a40
SQL> select comp_name,version from dba_server_registry;

COMP_NAME VERSION
—————————————- ——————————
Oracle Workspace Manager 11.2.0.3.0
Oracle Database Catalog Views 11.2.0.3.0
Oracle Database Packages and Types 11.2.0.3.0
Oracle Real Application Clusters 11.2.0.3.0

关于redhat 5.5安装10g Rac的问题总结

December 11, 2011 oracle, RAC No comments

环境:redhat 5.5+oracle 10g rac+netapp

首先碰到DMM问题,netapp存储通过DMM软件mapping过来的设备 oracle无法识别 报如下错误:

the location /dev/mapper/ocr1 ,entered for the oracle cluster registry(OCR) is not shared across all the nodes in the cluster…

起初怀疑是device 无法共享, 分别在两个node进程dd if-of操作 发现导入内容在另一个节点可以识别,确保磁盘是共享的,查看mulitpatch毫无问题。

查看metalink发现问题;Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1]

During the installation of Oracle Clusterware 10g Release 2 (10.2.0), the Universal Installer (OUI) is unable to verify the sharedness of block devices, therefore requires the use of raw devices (whether to singlepath or multipath devices) to be specified for OCR and voting disks. As mentioned earlier, this is no longer the case from Oracle11g R1 (11.1.0) that can use multipathed block devices directly

也就是说10g中的ocr,vote必须放在raw设备文件上,11g开始支持DMM mapping出来的文件

启动raw device –> mapping 到 DMM设备

# raw /dev/raw/raw1 /dev/mapper/ocr1
/dev/raw/raw1: bound to major 253, minor 11
# raw /dev/raw/raw2 /dev/mapper/ocr2
/dev/raw/raw2: bound to major 253, minor 8
# raw /dev/raw/raw3 /dev/mapper/ocr3
/dev/raw/raw3: bound to major 253, minor 10
# raw /dev/raw/raw4 /dev/mapper/voting1
/dev/raw/raw4: bound to major 253, minor 5
# raw /dev/raw/raw5 /dev/mapper/voting2
/dev/raw/raw5: bound to major 253, minor 4
# raw /dev/raw/raw6 /dev/mapper/voting3
/dev/raw/raw6: bound to major 253, minor 7

Redhat 5.5版本重新启用了rawdevice service 可以直接使用以上命令

安装继续 到了最后一步 要执行 crs root.sh脚本时 报错:

PROT-1: Failed to initialize ocrconfig
Failed to upgrade Oracle Cluster Registry configuration

./clsfmt ocr /dev/raw/raw1
clsfmt: Received unexpected error 4 from skgfifi
skgfifi: Additional information: -2
Additional information: 1000718336

Changes

It has been found that the following changes can cause this problem to occur:

1. Use Mutiple Path (MP) disk configuration, may hit this issue.
2. Use EMC device (powerpath**) may hit this issue.

But it was not confirmed that these are the only things that can cause this problem to occur, as it has been found that on other hardware and configuration the problem might occur, the key change in this issue is that if the disk size presented from the storage to the cluster nodes are not dividable by 4K the problem should occur.

我们正是使用了DMM软件 触发了此bug,将Patch:4679769 打上 继续到node2节点执行vipca时报错:

[root@db-35 bin]# ./vipca
/home/oracle/product/10.2/crs/jdk/jre//bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

解决方法

两个节点执行

修改vipca 脚本

vi vipca

Linux) LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/srvm/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH

#Remove this workaround when the bug 3937317 is fixed
arch=`uname -m`
if [ “$arch” = “i686” -o “$arch” = “ia64” -o “$arch” = “x86_64” ]
then
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
fi
#End workaround
add -> unset LD_ASSUME_KERNEL

在srvctl 中 同样修改

LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL

add ->unset LD_ASSUME_KERNEL

至此crs 安装完毕,安装database software 下载p8202632_10205_Linux-x86-64.zip 将crs和dbsoft 升级至10.2.0.5 DBCA 建库完成

修改Diagwait:

Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions [ID 559365.1]


Symptoms

Oracle Clusterware evicts the node from the cluster when

Node is not pinging via the network heartbeat
Node is not pinging the Voting disk
Node is hung/busy and is unable to perform either of the earlier tasks
In Most cases when the node is evicted, there is information written to the logs to analyze the cause of the node eviction. However in certain cases this may be missing, the steps documented in this note are to be used for those cases where there is not enough information or no information to diagnose the cause of the eviction for Clusterware versions less than 11gR2 (11.2.0.1).

Starting with 11.2.0.1, Customers do not need to set diagwait as the architecture has been changed.

Changes

None

Cause

When the node is evicted and the node is extremely busy in terms of CPU (or lack of it) it is possible that the OS did not get time to flush the logs/traces to the file system. It may be useful to set diagwait attribute to delay the node reboot to give additional time to the OS to write the traces. This setting will provide more time for diagnostic data to be collected by safely and will NOT increase probability of corruption. After setting diagwait, the Clusterware will wait an additional 10 seconds (Diagwait – reboottime). Customers can unset diagwait by following the steps documented below after fixing their OS scheduling issues.

* — Diagwait can be set on windows but it does not change the behaviour as it does on Unix-Linux platforms

@ For internal Support Staff
Diagwait attribute was introduced in 10.2.0.3 and is included in 10.2.0.4 & 11.1.0.6 and higher releases. It has also been subsequently backported to 10.1.0.5 on most platforms. This means it is possible to set diagwait on 10.1.0.5 (or higher), 10.2.0.3 (or higher) and in 11.1.0.6 (or higher). If the command crsctl set/get css diagwait reports “unrecognized parameter diagwait specified” then it can be safely assumed that the Clusterware version does not the necessary fixes to implement diagwait. If that is the case then customer is adviced to apply the latest patchset available before attempting to set diagwait
Solution

It is important that the clusterware stack must be down on all the nodes when changing diagwait .The following steps provides the step-by-step instructions on setting diagwait.

Execute as root
#crsctl stop crs
#/bin/oprocd stop
Ensure that Clusterware stack is down on all nodes by executing
#ps -ef |egrep “crsd.bin|ocssd.bin|evmd.bin|oprocd”
This should return no processes. If there are clusterware processes running and you proceed to the next step, you will corrupt your OCR. Do not continue until the clusterware processes are down on all the nodes of the cluster.
From one node of the cluster, change the value of the “diagwait” parameter to 13 seconds by issuing the command as root:
#crsctl set css diagwait 13 -force
Check if diagwait is set successfully by executing. the following command. The command should return 13. If diagwait is not set, the following message will be returned “Configuration parameter diagwait is not defined”
#crsctl get css diagwait
Restart the Oracle Clusterware on all the nodes by executing:
#crsctl start crs
Validate that the node is running by executing:
#crsctl check crs
Unsetting/Removing diagwait

Customers should not unset diagwait without fixing the OS scheduling issues as that can lead to node evictions via reboot. Diagwait delays the node eviction (and reconfiguration) by diagwait (13) seconds and as such setting diagwait does not affect most customers.In case there is a need to remove diagwait, the above mentioned steps need to be followed except step 3 needs to be replaced by the following command
#crsctl unset css diagwait -f

(Note: the -f option must be used when unsetting diagwait since CRS will be down when doing so)

至此 这次rac的安装顺利完成

[转] cluvfy使用说明

December 9, 2011 oracle, RAC No comments

在安装rac之前,可以通过Oracle提供的一个工具进行cluster环境验证,这个工具叫做cluvfy(Cluster Verification Utility)。

一、CVU程序的位置

OTN下载: http://otn.oracle.com/RAC

Oracle DVD

–clusterware/cluvfy/runcluvfy.sh

–clusterware/rpm/cvuqdisk-1.0.1-1.rpm (linuxonly)

CRS Home (如果已安装CRS软件)

–/bin/cluvfy

–/cv/rpm/cvuqdisk-1.0.1-1.rpm (linuxonly)

Oracle Home (如果已安装RAC数据库软件)

–$ORACLE_HOME/bin/cluvfy

二、CVU特点

1、CVU使用户可以在硬件的设置、clusterware的安装、数据库软件的安装、增加节点等各个阶段来检查cluster.

2、具有可扩展的结构(平台独立,包含对存储和网络设备等的检查)

3、CVU是没有损害的检查

4、命令行方式

5、即使检查失败也不会做任何修正

6、CVU不涉及任何性能调整和监控

7、不做任何cluster或RAC方面的操作(例如,不会去起动或关闭CRS)

8、不去检查cluster database或cluster的内在关系(如CRS与database的关系等)

三、CVU命令用法

[oracle@zhh1 ~]$ cluvfy

USAGE:

cluvfy [ -help ]

cluvfy stage { -list | -help }

cluvfy stage {-pre|-post}[-verbose]

cluvfy comp { -list | -help }

cluvfy comp[-verbose]

[oracle@zhh1 ~]$ which cluvfy

/usr/oracle/product/10.2.0/db_1/bin/cluvfy

cluvfy只安装在本地节点,会在运行时根据需要自动部署到其它节点上。

主要用法:

阶段检查(Stage Verification)
10g RAC的部署(安装)可以逻辑性的分为几个部分,每个部分就叫做一个‘阶段’。

例如:硬件和操作系统的建立、CFS的建立、CRS软件的安装、数据库软件的安装、数据库的配置等

每个阶段又包含一系列的操作并且分为安装前和安装后检查等两种方式.

部件检查(Component Verification)
检查cluster部件的可用性、完整性、健康性

例如:共享存储设备的可用性、空间可用性、节点通达性、CFS的完整性、Cluster的完整性、CRS的完整性等

1、stage

[oracle@zhh1 ~]$ cluvfy stage -list

USAGE:

cluvfy stage {-pre|-post}[-verbose]

Valid stage options and stage names are:

-post hwos : post-check for hardware and operating system

-pre cfs : pre-check for CFS setup

-post cfs : post-check for CFS setup

-pre crsinst : pre-check for CRS installation

-post crsinst : post-check for CRS installation

-pre dbinst : pre-check for database installation

-pre dbcfg : pre-check for database configuration

有效的选项和阶段名称如下:

-post hwos:硬件和操作系统安装后的检查

-pre cfs: CFS建立之前的检查

-post cfs: CFS建立之后的检查

-pre crsinst: CRS安装之前的检查

-post crsinst: CRS安装之后的检查

-pre dbinst:数据库软件安装之前的检查

-pre dbcfg:数据库配置之前的检查

2、component

[oracle@zhh1 ~]$ cluvfy comp -list

USAGE:

cluvfy comp[-verbose]

Valid components are:

nodereach : checks reachability between nodes

nodecon : checks node connectivity

cfs : checks CFS integrity

ssa : checks shared storage accessibility

space : checks space availability

sys : checks minimum system requirements

clu : checks cluster integrity

clumgr : checks cluster manager integrity

ocr : checks OCR integrity

crs : checks CRS integrity

nodeapp : checks node applications existence

admprv : checks administrative privileges

peer : compares properties with peers

有效的部件名称如下:

nodereach:检查节点之间的通达性

nodecon:检查节点之间的的连通性

cfs:检查CFS的完整性

ssa:检查共享存储的可访问性

space :检查空间的可用性

sys :检查最小的系统需求

clu:检查cluster的完整性

clumgr:检查cluster manager的完整性

ocr:检查OCR的完整性

crs:检查CRS的完整性

nodeapp:检查nodeapp是否存在

admprv:检查管理权限

peer :比较同等属性(即节点差异)

3、使用stage还是component
a、阶段模式-在安装CRS和RAC期间使用CVU阶段模式
在阶段模式中正确使用安装前和安装后检查方式
例如:
安装CRS软件之前:
运行DVD上或从OTN下载的CVU命令runcluvfy.sh
$cluvfy stage -pre crsinst -n zhh1,zhh2 -verbose | tee /tmp/cluvfy_preinst.log
安装RAC软件之前:
运行安装在CRS Home的bin目录下的cluvfy
$ cluvfy stage -pre dbinst -n zhh1,zhh2 -verbose | tee /tmp/cluvfy_dbinst.log

b、部件模式-在CRS或数据库运行过程中如果需要检查某个特殊部件或为了隔离一个cluster子系统做诊断时,使用适当的部件模式.
例如:
诊断网络问题
$ cluvfy comp nodecon -n zhh1,zhh2 -verbose
将检查network interface(eth0,eth1,etc)、public IP、private IP等
诊断共享磁盘问题
$ cluvfy comp ssa -n zhh1,zhh2 -s /dev/sdb1 -verbose
将检查所列的存储设备是否在所有节点间共享

[oracle@zhh1 ~]$ cd /usr/oracle/soft/clusterware/
cluvfy/ install/ rootpre/ runInstaller upgrade/
doc/ response/ rpm/ stage/ welcome.html
[oracle@zhh1 ~]$ cd /usr/oracle/soft/clusterware/cluvfy/
[oracle@zhh1 cluvfy]$ ll
total 30996
-rwxr-xr-x 1 oracle dba 10326112 Oct 23 2005 cvupack.zip
-rwxr-xr-x 1 oracle dba 21356849 Oct 23 2005 jrepack.zip
-rwxr-xr-x 1 oracle dba 3107 Oct 23 2005 runcluvfy.sh
就是这个runcluvfy.sh,主要命令的使用方法都已经列出了。
[oracle@zhh1 cluvfy]$ ./runcluvfy.sh
USAGE:
cluvfy [ -help ]
cluvfy stage { -list | -help }
cluvfy stage {-pre|-post} [-verbose]
cluvfy comp { -list | -help }
cluvfy comp [-verbose]

[oracle@zhh1 ~]$ cluvfy stage -pre crsinst -n zhh1

Performing pre-checks for cluster services setup

Checking node reachability…
Node reachability check passed from node “zhh1”.

Checking user equivalence…
User equivalence check passed for user “oracle”.

Checking administrative privileges…
User existence check passed for “oracle”.
Group existence check failed for “oinstall”.
Check failed on nodes:
zhh1

Administrative privileges check failed.

Checking node connectivity…

Node connectivity check passed for subnet “192.168.5.0” with node(s) zhh1.
Node connectivity check passed for subnet “10.0.0.0” with node(s) zhh1.

Suitable interfaces for the private interconnect on subnet “192.168.5.0”:

zhh1 eth0:192.168.5.235 eth0:192.168.5.233

Suitable interfaces for the private interconnect on subnet “10.0.0.0”:

zhh1 eth1:10.0.0.100

ERROR:
Could not find a suitable set of interfaces for VIPs.

Node connectivity check failed.

Checking system requirements for ‘crs’…
Total memory check passed.
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for “binutils-2.15.92.0.2-13”.
Group existence check passed for “dba”.
Group existence check failed for “oinstall”.
Check failed on nodes:
zhh1
User existence check passed for “nobody”.

System requirement failed for ‘crs’

Pre-check for cluster services setup was unsuccessful on all the nodes.
可以看到检查结果,就可以根据这些检查结果来进行有的放矢地修改了。

 

reference: http://hi.baidu.com/dba_hui/blog/item/99248378e3b2e51a29388ace.html

[转] Oracle 10g RAC中的DRM问题及关闭

December 1, 2011 Internals, oracle, RAC 2 comments

在RAC环境中,Oracle使用GRD(Global Resource Service)来记录各个RAC节点的资源信息,具体通过GCS(Global Cache Service)和GES(Global Enqueue Service)这两个服务进行管理。
由于在RAC中每个节点都有自己的SGA和buffer cache,为了保证Cache资源的一致性和提高性能,GCS和GES会指定RAC中的一个instance来管理Cache,这个节点这时就是Resource Master。
在10g以前,Cache资源是不能在各个节点间移动的,除非重启或者某节点因为其他异常被RAC驱逐等情况。而10g的DRM就解决了这个问题,它可以保证cache能够被remaster到频繁访问这部分数据的节点上,从而提高RAC的性能。DRM的全称是Dynamic Resource Mastering,metalink上的Doc ID: 390483.1文档详细介绍了DRM的信息。

从理论上讲,利用此项技术,非master节点对所需资源有频繁访问需求时,可以提升为master节点,从而减少大量后续的跨节点资源访问需求。
但是,首先从根本上说,一个好的RAC应用设计,本就应该极尽所能的取避免同一资源的多节点访问,如果不存在同一资源的多节点访问,则DRM所要解决的问题,就根本不存在。其次,DRM本身是需要消耗资源的,并且存在诸多bug,对于一个设计较差的系统而言,频繁的DRM,也会引发Libary cache lock而导致实例挂住。
更严重的,在10.2.0.3系统上,曾经遇到一个case,电信行业的巨型数据库,rac的2号节点由于批量处理作业在非业务时间段,首先cache了一张40G的表,而到了业务时间段后,rac的1号节点的OLTP业务需要频繁访问该表,此时,故障发生了,由于DRM的介入,2号节点开始将内存内的40Gcache数据向1号节点传输,心跳网段千兆带宽被耗尽,RAC陷入僵死阶段,足足维持了40分钟。
事后检查网络流量图,该时段内,私有网络流量持续保持在90M/s的峰值水平。
根据metalink确认,该问题确实由DRM机制引起,最终解决方案,使用隐含参数,将DRM特性屏蔽:
_gc_affinity_time=0
_gc_undo_affinity=FALSE

修改这两个参数需要重启数据库,可以用下面两个参数实行关闭:

_gc_affinity_limit=250  

_gc_affinity_minimum=10485760 

因此,从根本上来说,drm的出现,只是在理论上的一种缓解,而并不能在实际的大型应用中发挥其作用。就类似于Oracle自己针对RAC推出的自动负载平衡一样,只是一种看起来很美的东西,如果真的有人用了,呵呵,那就只能等死吧。或许压力极小的数据库无所谓,但我没遇到过,话又说回来,压力极小,又何必上RAC呢。

reference: http://www.dbaleading.com/Article.asp?id=42

Oracle 9i 单节点文件系统 升级到 10g RAC

November 10, 2011 migration, oracle, RAC No comments

最近做了一次Oracle 9i 升级到10g RAC 的测试,现做如下简要记录:
1.查看目前数据库版本


SQL> select * from v$version;

BANNER
—————————————————————-
Oracle9i Enterprise Edition Release 9.2.0.4.0 – Production
PL/SQL Release 9.2.0.4.0 – Production
CORE 9.2.0.3.0 Production
TNS for Linux: Version 9.2.0.4.0 – Production
NLSRTL Version 9.2.0.4.0 – Production

FILE_ID FILE_NAME
———- ————————————————————
1 /u01/app/oracle/oradata/honcho/system01.dbf
2 /u01/app/oracle/oradata/honcho/undotbs01.dbf
3 /u01/app/oracle/oradata/honcho/hongzx01.dbf
4 /u01/app/oracle/oradata/honcho/drsys01.dbf
5 /u01/app/oracle/oradata/honcho/example01.dbf
9 /u01/app/oracle/oradata/honcho/users01.dbf
10 /u01/app/oracle/oradata/honcho/xdb01.dbf

FILE# NAME
———- ————————————————————
1 /u01/app/oracle/oradata/honcho/temp01.dbf

2.对当前数据库做全备

RMAN> run
2> {
3> allocate channel c1 type disk;
4> backup full database format=’/u01/backup/honcho_full_%U_%T’ tag=’honcho_full’;
5> sql ‘alter system archive log current’;
6> backup archivelog all tag=’arc_bak’ format=’/u01/backup/arch_%U_%T’ delete input;
7> backup current controlfile tag=’bak_ctlfile’ format=’/u01/backup/ctl_file_%U_%T’;
8> backup spfile tag=’spfile’ format=’/u01/backup/honcho_spfile_%U_%T’;
9> release channel c1;
10> }

RMAN> list backup summary;

List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Tag
——- — — – ———– ————— ——- ——- —
1 B F A DISK 08-NOV-11 1 1 honcho_FULL
2 B F A DISK 08-NOV-11 1 1
3 B A A DISK 08-NOV-11 1 1 ARC_BAK
4 B F A DISK 08-NOV-11 1 1 BAK_CTLFILE
5 B F A DISK 08-NOV-11 1 1 SPFILE
6 B F A DISK 08-NOV-11 1 1

3.搭建10g 的clusterware 和ASM 实例
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

3.直接拷贝原9.2.0.4的pfile到192.168.1.131相应目录,然后在两节点创建相应目录
[oracle@rac1 dbs]$ mkdir -p /u01/app/oracle/admin/honcho/bdump
[oracle@rac1 dbs]$ mkdir -p /u01/app/oracle/admin/honcho/cdump
[oracle@rac1 dbs]$ mkdir -p /u01/app/oracle/admin/honcho/udump

4.修改之后的参数如下:
[oracle@rac1 dbs]$ cat inithoncho1.ora
*.aq_tm_processes=0
*.background_dump_dest=’/u01/app/oracle/admin/honcho/bdump’
*.compatible=’10.2.0.1.0′
*.core_dump_dest=’/u01/app/oracle/admin/honcho/cdump’
*.db_block_size=8192
*.db_cache_size=33554432
*.db_domain=”
*.db_file_multiblock_read_count=16
*.db_name=’honcho’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=honchoXDB)’
*.fast_start_mttr_target=300
*.instance_name=’honcho’
*.java_pool_size=83886080
*.job_queue_processes=0
*.large_pool_size=16777216
*.open_cursors=300
*.pga_aggregate_target=25165824
*.processes=150
*.query_rewrite_enabled=’FALSE’
*.remote_login_passwordfile=’EXCLUSIVE’
*.shared_pool_size=83886080
*.sort_area_size=524288
*.star_transformation_enabled=’FALSE’
*.timed_statistics=TRUE
*.undo_management=’AUTO’
*.undo_retention=10800
*.user_dump_dest=’/u01/app/oracle/admin/honcho/udump’
*.cluster_database_instances=2
*.cluster_database=true
*.control_files=’+DATA/honcho/controlfile/control01.ctl’,’+RECOVERY/honcho/controlfile/control02.ctl’
*.db_create_file_dest=’+DATA’
*.db_recovery_file_dest=’+RECOVERY’
*.db_recovery_file_dest_size=2147483648
honcho1.instance_name=’honcho1′
honcho2.instance_name=’honcho2′
honcho1.instance_number=1
honcho2.instance_number=2
*.log_archive_dest_1=’LOCATION=+RECOVERY/honcho/archivelog’
honcho1.thread=1
honcho2.thread=2
honcho1.undo_tablespace=’UNDOTBS1′
honcho2.undo_tablespace=’UNDOTBS2′

5.使用pfile生产spfile

[oracle@rac1 dbs]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 – Production on Wed Nov 9 10:52:17 2011

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> conn / as sysdba
Connected.
SQL> create spfile=’+DATA/honcho/PARAMETERFILE/spfilehoncho.ora’ from pfile=’/u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora’;

File created.

6.重新编译pfile,让其指向spfile,在两节点都需要编译
[oracle@rac1 dbs]$ echo “SPFILE=’+DATA/honcho/PARAMETERFILE/spfilehoncho.ora'” > /u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora
[oracle@rac2 ~]$ cd $ORACLE_HOME/dbs
[oracle@rac2 dbs]$ touch inithoncho2.ora
[oracle@rac2 dbs]$ echo “SPFILE=’+DATA/honcho/PARAMETERFILE/spfilehoncho.ora'” > /u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho2.ora

7.创建密码文件:
[oracle@rac1 dbs]$ orapwd file=orapwhoncho1 password=oracle entries=20
[oracle@rac2 dbs]$ orapwd file=orapwhoncho2 password=oracle entries=20

8.开启数据库到nomount状态:
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup nomount
ORACLE instance started.

Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 184551356 bytes
Database Buffers 33554432 bytes
Redo Buffers 2973696 bytes

9.恢复controlfile

RMAN> restore controlfile from ‘/u01/backup/ctl_file_04mr66b3_1_1_20111108’;

Starting restore at 09-NOV-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=150 instance=honcho1 devtype=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
output filename=+DATA/honcho/controlfile/control01.ctl
output filename=+RECOVERY/honcho/controlfile/control02.ctl
Finished restore at 09-NOV-11

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

10.restore database

RMAN> run {
2> set newname for datafile 1 to ‘+DATA/honcho/DATAFILE/system01.dbf’;
3> set newname for datafile 2 to ‘+DATA/honcho/DATAFILE/undotbs01.dbf’;
4> set newname for datafile 3 to ‘+DATA/honcho/DATAFILE/hongzx01.dbf’;
5> set newname for datafile 4 to ‘+DATA/honcho/DATAFILE/drsys01.dbf’;
6> set newname for datafile 5 to ‘+DATA/honcho/DATAFILE/example01.dbf’;
7> set newname for datafile 9 to ‘+DATA/honcho/DATAFILE/users01.dbf’;
8> set newname for datafile 10 to ‘+DATA/honcho/DATAFILE/xdb01.dbf’;
9> restore database;
10> switch datafile all;
11> switch tempfile all;
12> }

这里并没有restore tempfile

11.recover database

RMAN> recover database;

Starting recover at 09-NOV-11
using channel ORA_DISK_1

starting media recovery

channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=5
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=6
channel ORA_DISK_1: reading from backup piece /u01/backup/arch_03mr66ar_1_1_20111108
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/backup/arch_03mr66ar_1_1_20111108 tag=ARC_BAK
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
archive log filename=+RECOVERY/honcho/archivelog/1_5_766710700.dbf thread=1 sequence=5
archive log filename=+RECOVERY/honcho/archivelog/1_6_766710700.dbf thread=1 sequence=6
unable to find archive log
archive log thread=1 sequence=7
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 11/09/2011 13:39:31
RMAN-06054: media recovery requesting unknown log: thread 1 seq 7 lowscn 196183

12.修改online logfile 到ASM

SQL> alter database rename file ‘/u01/app/oracle/oradata/honcho/redo03.log’ to ‘+DATA/honcho/ONLINELOG/redo03.log’;

Database altered.

SQL> alter database rename file ‘/u01/app/oracle/oradata/honcho/redo02.log’ to ‘+DATA/honcho/ONLINELOG/redo02.log’;

Database altered.

SQL> alter database rename file ‘/u01/app/oracle/oradata/honcho/redo01.log’ to ‘+DATA/honcho/ONLINELOG/redo01.log’;

Database altered.

13.把数据库以resetlogs的方式打开
alter database open resetlogs;

14.升级数据库到10g
SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 184551356 bytes
Database Buffers 33554432 bytes
Redo Buffers 2973696 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced

查看日志
Errors in file /u01/app/oracle/admin/honcho/udump/honcho1_ora_16126.trc:
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Error 39701 happened during db open, shutting down database
USER: terminating instance due to error 39701

解决方法:
[oracle@rac1]$ cd $ORACLE_HOME/nls/data/old
[oracle@rac1 old]$ perl cr9idata.pl
Creating directory /u01/app/oracle/product/10.0.2.1/db_1/nls/data/9idata …
Copying files to /u01/app/oracle/product/10.0.2.1/db_1/nls/data/9idata…
Copy finished.
Please reset environment variable ORA_NLS10 to /u01/app/oracle/product/10.0.2.1/db_1/nls/data/9idata!

编辑环境变量,添加:
export ORA_NLS10=/u01/app/oracle/product/10.0.2.1/db_1/nls/data/9idata

同时要将参数文件里面的与cluster 相关的参数去掉

SQL> startup upgrade;
ORACLE instance started.

Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 184551356 bytes
Database Buffers 33554432 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.
SQL> select * from v$version;

BANNER
—————————————————————-
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Prod
PL/SQL Release 10.2.0.1.0 – Production
CORE 10.2.0.1.0 Production
TNS for Linux: Version 10.2.0.1.0 – Production
NLSRTL Version 10.2.0.1.0 – Production

15.给临时表空间添加文件
SQL> alter tablespace temp add tempfile ‘+DATA’ size 100m;

16.由于在9i里面没有sysaux表空间,这里也要添加sysaux表空间:
SQL> create tablespace sysaux datafile ‘+DATA’ size 500m reuse
2 extent management local
3 segment space management auto;

Tablespace created.

17.运行升级脚本

@?/rdbms/admin/catupgrd.sql
@?/rdbms/admin/catclust.sql
@?/rdbms/admin/utlrp.sql

18.添加与RAC相关的一些参数:


SQL> alter system set cluster_database=true scope=spfile;

System altered.

SQL> alter system set cluster_database_instances=2 scope=spfile;

System altered.

SQL> alter system set instance_number=1 scope=spfile sid=’honcho1′;

System altered.

SQL> alter system set instance_number=2 scope=spfile sid=’honcho2′;

System altered.

SQL> alter system set thread=1 scope=spfile sid=’honcho1′;

System altered.

SQL> alter system set thread=2 scope=spfile sid=’honcho2′;

System altered.

19.添加节点2的undo表空间和redolog file

SQL> create undo tablespace UNDOTBS2 datafile ‘+DATA/honcho/datafile/undotbs02.dbf’ size 50m;

Tablespace created.

SQL> alter system set undo_tablespace=’UNDOTBS2′ scope=spfile sid=’honcho2′;

System altered.

SQL> alter database add logfile thread 2 group 4 ‘+DATA’ size 100m;

Database altered.

SQL> alter database add logfile thread 2 group 5 ‘+DATA’ size 100m;

Database altered.

SQL> alter database add logfile thread 2 group 6 ‘+DATA’ size 100m;

Database altered.
SQL> alter database enable public thread 2;

Database altered.

20.重新打开1节点:
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 184551356 bytes
Database Buffers 33554432 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.
SQL> show parameter cluster

NAME TYPE VALUE
———————————— ———– ——————————
cluster_database boolean TRUE
cluster_database_instances integer 2
cluster_interconnects string

在节点2上同样打开实例:
[oracle@rac2 ~]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 – Production on Wed Nov 9 18:24:02 2011

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.

Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 184551356 bytes
Database Buffers 33554432 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.

SQL> select instance_number,instance_name,host_name from gv$instance;

INSTANCE_NUMBER INSTANCE_NAME HOST_NAME
————— —————- ——————————
1 honcho1 rac1
2 honcho2 rac2

2 rows selected.

21.添加监听
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
使用netca配置
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

22.添加服务

[oracle@rac1 ~]$ srvctl add database -d honcho -o $ORACLE_HOME -p +DATA/honcho/PARAMETERFILE/spfilehoncho.ora
[oracle@rac1 ~]$ srvctl add instance -d honcho -i honcho1 -n rac1
[oracle@rac1 ~]$ srvctl add instance -d honcho -i honcho2 -n rac2
[oracle@rac1 ~]$ srvctl modify instance -d honcho -i honcho1 -s +ASM1
[oracle@rac1 ~]$ srvctl modify instance -d honcho -i honcho2 -s +ASM2
[oracle@rac1 ~]$ srvctl start database -d honcho

[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora.honcho.db application ONLINE ONLINE rac1
ora….b1.inst application ONLINE ONLINE rac1
ora….b2.inst application ONLINE ONLINE rac2
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

SQL> select * from v$version;

BANNER
—————————————————————-
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Prod
PL/SQL Release 10.2.0.1.0 – Production
CORE 10.2.0.1.0 Production
TNS for Linux: Version 10.2.0.1.0 – Production
NLSRTL Version 10.2.0.1.0 – Production

还有一些相关参数需要调整。

OK. Game over!

一次cursor: pin S wait on X的解决

November 2, 2011 oracle, RAC 3 comments

环境描述:双节点rac 数据库版本10.2.0.5 通过awr发现down机时间段产生大量的cursor: pin S wait on X等待,同时伴随大量的library latch等待,通过与相关人员沟通得知,此库在每月的月头都会有大量的报表业务,从而负载将会增长几十倍之多。通过下面
的awr可以看出负载确实很高:

DB Name DB Id Instance Inst num Release RAC Host
PORTALDB 597889515 portaldb1 1 10.2.0.5.0 YES ptdb01

Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 4115 02-Nov-11 11:00:13 209 131.6
End Snap: 4116 02-Nov-11 12:03:09 289 16.2
Elapsed:   62.93 (mins)    
DB Time:   15,687.33 (mins)    

通过对比awr与其他时间段的awr发现DBTIME 增加了几十倍 说明确实存在大量的业务

通过TOP5时间 发现最多的等待为cursor: pin S wait on X 同时伴随library cache等待:

Top 5 Timed Events

Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
cursor: pin S wait on X 77,646,738 759,238 10 80.7 Concurrency
library cache lock 200,160 95,987 480 10.2 Concurrency
enq: TX – row lock contention 70,851 34,471 487 3.7 Application
CPU time   18,501   2.0  
library cache pin 5,680 14,825 2,610 1.6 Concurrency

SELECT COMPONENT ,OPER_TYPE,FINAL_SIZE Final,to_char(start_time,’dd-mon hh24:mi:ss’) Started FROM V$SGA_RESIZE_OPS查询
短时间内发现大量的shrink,grow,由此怀疑业务高峰期间大量的shrink,grow导致shard_pool的部分cursor被刷出,从而导致hard parse的增加进而导致library cache 等待的发生,频繁的shrink,grow也会导致cursor: pin S wait on X的发生。

通过沟通,夜里将关闭ASMM,并且通过当时awr报告计算出shared_pool,buffer_cache峰值,明天将继续观察负载情况。
另外在峰值时段发现大量append操作引发enq-TX等待也将与应用人员沟通

跟进今天的情况:

Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 4139 03-Nov-11 11:00:18 118 81.5
End Snap: 4140 03-Nov-11 12:00:36 159 108.3
Elapsed:   60.31 (mins)    
DB Time:   428.83 (mins)    

从上述情况看到今天的负载小于昨天,下面是今天的等待事件:

Top 5 Timed Events

Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
CPU time   20,034   77.9  
db file sequential read 333,128 1,400 4 5.4 User I/O
direct path read temp 175,492 434 2 1.7 User I/O
db file scattered read 239,062 261 1 1.0 User I/O
gc current block 2-way 331,161 231 1 .9 Cluster

可以看到昨天的cursor等待消失

Wait Events

  • s – second
  • cs – centisecond – 100th of a second
  • ms – millisecond – 1000th of a second
  • us – microsecond – 1000000th of a second
  • ordered by wait time desc, waits desc (idle events last)
  • %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0

Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
db file sequential read 333,128   1,400 4 14.37
direct path read temp 175,492   434 2 7.57
db file scattered read 239,062   261 1 10.31
gc current block 2-way 331,161   231 1 14.28
gc cr multi block request 302,696   129 0 13.06
log file sequential read 6,144   106 17 0.27
log file parallel write 148,777   101 1 6.42
direct path write temp 186,873   88 0 8.06
gcs log flush sync 161,961 0 77 0 6.99
gc cr block 2-way 105,826   71 1 4.56
gc cr grant 2-way 127,157   63 0 5.48
log file sync 18,846 1 49 3 0.81
db file parallel read 13,520   29 2 0.58
CSS operation: query 1,230   22 18 0.05
control file sequential read 36,492   20 1 1.57
CSS initialization 410   18 44 0.02
gc current grant busy 26,892 0 17 1 1.16
gc current grant 2-way 42,689   13 0 1.84
gc buffer busy 6,259 0 9 1 0.27
control file parallel write 1,869   5 3 0.08
read by other session 846   5 6 0.04
cursor: pin S wait on X 434 100 4 10 0.02
latch: cache buffers chains 6,638   3 1 0.29
enq: TX – row lock contention 12 33 3 252 0.00
gc cr block busy 995   3 3 0.04
SQL*Net message to client 3,549,215   3 0 153.09
enq: WF – contention 14 29 2 145 0.00
DFS lock handle 2,376   1 1 0.10
direct path read 3,331   1 0 0.14
gc current block busy 199   1 6 0.01
row cache lock 4,347   1 0 0.19
cr request retry 1 100 1 977 0.00
enq: KO – fast object checkpoint 38 3 1 25 0.00
gc current multi block request 7,843   1 0 0.34
log buffer space 4   1 221 0.00
latch: KCL gc element parent latch 74   1 10 0.00
SQL*Net more data from client 7,756   1 0 0.33
gc current block congested 37   1 15 0.00
latch free 22   1 23 0.00
Log archive I/O 6,119   0 0 0.26
CGS wait for IPC msg 47,974 99 0 0 2.07
SQL*Net break/reset to client 2,210   0 0 0.10
enq: FB – contention 1,560   0 0 0.07
log file switch completion 20   0 18 0.00
CSS operation: action 807   0 0 0.03
gc cr block congested 11   0 25 0.00
gc cr grant congested 16   0 17 0.00
os thread startup 10   0 23 0.00
SQL*Net more data to client 15,643   0 0 0.67
kksfbc child completion 3 100 0 49 0.00
reliable message 306   0 0 0.01
direct path write 618   0 0 0.03
buffer busy waits 3,383   0 0 0.15
enq: PS – contention 203   0 0 0.01
LGWR wait for redo copy 8,131   0 0 0.35
ksxr poll remote instances 10,924 56 0 0 0.47
enq: CF – contention 157   0 1 0.01
library cache pin 301   0 0 0.01
PX Deq: reap credit 6,959 98 0 0 0.30
enq: TX – index contention 48   0 2 0.00
rdbms ipc reply 351   0 0 0.02
SQL*Net message from dblink 66   0 1 0.00
library cache lock 144   0 0 0.01
PX Deq Credit: send blkd 196 13 0 0 0.01
enq: HW – contention 104   0 0 0.00
enq: TM – contention 123   0 0 0.01
latch: cache buffers lru chain 112   0 0 0.00
gc current block unknown 49   0 0 0.00
IPC send completion sync 63 100 0 0 0.00
PX Deq: Signal ACK 203 39 0 0 0.01
latch: object queue header operation 29   0 1 0.00
latch: shared pool 42   0 0 0.00
gc current split 2   0 4 0.00
gc current grant congested 1   0 8 0.00
gc current retry 3   0 2 0.00
KJC: Wait for msg sends to complete 7   0 0 0.00
enq: TA – contention 11   0 0 0.00
latch: row cache objects 43   0 0 0.00
log file single write 12   0 0 0.00
ges inquiry response 6   0 0 0.00
latch: session allocation 3   0 1 0.00
enq: TD – KTF dump entries 6   0 0 0.00
latch: checkpoint queue latch 8   0 0 0.00
latch: library cache lock 7   0 0 0.00
enq: JS – job run lock – synchronize 2   0 1 0.00
enq: WL – contention 3   0 0 0.00
enq: TX – allocate ITL entry 3   0 0 0.00
enq: TT – contention 2   0 0 0.00
enq: MW – contention 2   0 0 0.00
enq: DR – contention 2   0 0 0.00
cursor: pin S 309   0 0 0.01
latch: library cache 3   0 0 0.00
SQL*Net more data to dblink 6   0 0 0.00
latch: gcs resource hash 3   0 0 0.00
latch: redo allocation 3   0 0 0.00
SQL*Net message to dblink 63   0 0 0.00
latch: ges resource hash list 1   0 0 0.00
lock escalate retry 7 86 0 0 0.00
SQL*Net more data from dblink 3   0 0 0.00
latch: messages 1   0 0 0.00
lock deadlock retry 1 100 0 0 0.00
cursor: mutex S 1   0 0 0.00
SQL*Net message from client 3,550,480   340,614 96 153.14
gcs remote message 2,022,558 19 17,483 9 87.24
virtual circuit status 121 100 3,540 29258 0.01
ges remote message 64,294 67 3,531 55 2.77
Streams AQ: qmn slave idle wait 128   3,531 27587 0.01
Streams AQ: qmn coordinator idle wait 265 48 3,531 13325 0.01
PX Idle Wait 1,476 93 3,530 2391 0.06
ASM background timer 806   3,528 4377 0.03
class slave wait 633 94 3,527 5573 0.03
Streams AQ: waiting for time management or cleanup tasks 12 100 3,464 288641 0.00
DIAG idle wait 15,062 4 2,928 194 0.65
i/o slave wait 580,651   1,423 2 25.05
PX Deq: Execute Reply 2,072 48 218 105 0.09
jobq slave wait 40 100 117 2930 0.00
PX Deq: Parse Reply 229 34 1 4 0.01
PX Deq: Execution Msg 431 8 1 2 0.02
PX Deq: Join ACK 242 38 0 1 0.01
KSV master wait 32   0 7 0.00
single-task message 3   0 32 0.00
PX Deq: Msg Fragment 95 9 0 0 0.00
PX Deq Credit: need buffer 44 11 0 0 0.00
Streams AQ: RAC qmn coordinator idle wait 265 100 0 0 0.01
PX Deq: Table Q Normal 8   0 0 0.00

Why ASMLIB and why not?

October 19, 2011 oracle, RAC 2 comments

ASMLIB是一种基于Linux module,专门为Oracle Automatic Storage Management特性设计的内核支持库
(kernel support library)。
长久以来我们对ASMLIB的认识并不全面,这里我们来具体了解一下使用ASMLIB的优缺点。

理论上我们可以从ASMLIB API中得到的以下益处:

总是使用direct,async IO
1.解决了永久性设备名的问题,即便在重启后设备名已经改变的情况下
2.解决了文件权限、拥有者的问题
3.减少了I/O期间从用户模式到内核模式的上下文切换,从而可能降低cpu使用率
4.减少了文件句柄的使用量
5.ASMLIB API提供了传递如I/O优先级等元信息到存储设备的可能
虽然从理论上我们可以从ASMLIB中得到性能收益,但实践过程中这种优势是几乎可以忽略的,没有任何性能报告显示
ASMLIB对比Linux上原生态的udev设备管理服务有任何性能上的优势。
在Oracle官方论坛上有一篇讨论ASMLIB性能收益的帖子,你可以从中看
到”asmlib wouldn’t necessarily give you much of an io performance benefit,
it’s mainly for ease of management as it will find/discover the right devices for you,
the io effect of asmlib is large the same as doing async io to raw devices.”的评论,
实际上使用ASMLIB和直接使用裸设备(raw device)在性能上没有什么差别。

ASMLIB可能带来的缺点:

1.对于多路径设备(multipathing)需要在/etc/sysconfig/oracleasm-_dev_oracleasm配置文件中设置
ORACLEASM_SCANORDER及ORACLEASM_SCANEXCLUDE,以便ASMLIB能找到正确的设备文件,
具体可以参考Metalink Note
2.因为ASM INSTANCE使用ASMLIB提供的asm disk,所以增加了额外的层面
3.每次Linux Kernel更新,都需要替换新的ASMLIB包增加了因人为错误造成宕机downtime的可能
4.使用ASMLIB意味着要花费更多时间去创建和维护
5.因为ASMLIB的存在,可能引入更多的bug,这是我们最不想看到的
6.使用ASMLIB创建的disk,其disk header并不会和普通的asm disk header有什么不同,仅仅是在头部多出了ASMLIB的属性空间。
结论:
我个人的观点是尽可能不要使用ASMLIB,当然这不是DBA个人所能决定的事情。另一方面这取决于个人习惯,在rhel 4的早期发行版本中没有提供
udev这样的设备管理服务,这导致在rhel 4中大量的ASM+RAC组合的系统使用ASMLIB。
如果是在RHEL/OEL 5中那么你有充分的理由利用udev而放弃ASMLIB。
reference:http://www.oracledatabase12g.com/archives/why-asmlib-and-why-not.html