VMCD.ORG

Oracle Real Application Clusters (RAC) allows Oracle Database to run any packaged or custom application, unchanged across a set of clustered servers. This provides high availability and flexible scalability. If a clustered server fails, then database continues running on the rest of the servers. When you need additional processing power, you can add additional servers to the cluster online.
A key hardware component of this technology is the private interconnect network between the clustered servers. The interconnect is used by the cluster for inter-node messaging. The interconnect is also used by RAC to implement cache fusion technology.
Reliable Datagram Sockets (RDS) protocol provides reliable datagram services multiplexing UDP packets over InfiniBand connection improving performance to Oracle RAC. It provides high performance cluster interconnect for Oracle 10g RAC, utilizing InfiniBand which has 10X to 30X bandwidth advantage and 10X to 30X latency reduction vs. Gigabit Ethernet.

Only the database servers are configured with a 64K MTU. Presumably this is to benefit TCP/IP traffic between the database servers, and between the database servers and any external host that is linked to the IB switch. You may be surprised to know that the IB ports on the storage cells are configured with the standard 1,500 byte MTU size. The large MTU size is not necessary on the storage cells, because I/O between the database grid and the storage grid utilizes the RDS protocol, which is much more efficient for database I/O and bypasses the TCP/IP protocol stack altogether.