Extended RAC

1. One database, many instances

A single-instance Oracle database is one set of datafiles opened by one instance: one SGA, one set of background processes, on one server. RAC keeps the database singular and makes the instance plural. Two, four or more servers each run their own instance, with their own SGA, redo threads and background processes, and all of them open the same datafiles on shared storage at the same time.

That one sentence drives everything else in RAC. Because all instances write to the same files, they must agree on who holds which block, which is what Cache Fusion does. Because an instance can die while others carry on, the survivors must detect the death quickly and recover its redo, which is what Clusterware and instance recovery do. Because clients should not care which server they land on, there are services, VIPs and the SCAN.

2. The two software stacks on every node

A RAC node runs two Oracle homes owned by two OS users. The Grid Infrastructure home (owner grid) holds Oracle Clusterware and ASM. The database home (owner oracle) holds the RDBMS software. Clusterware starts first, finds the shared storage, forms the cluster, and only then starts ASM and the database instances as cluster resources.

3. Clusterware: the processes and what each one does

Since 11g Release 2, Clusterware is organised in two stacks. The lower stack (OHASD, the "High Availability Services" stack) runs on each node on its own and needs nothing from any other node. The upper stack (CRSD, the "Cluster Ready Services" stack) runs only when the node is a member of the cluster, because it needs the OCR on shared storage.

crsctl stat res -t -init          # lower stack (OHASD-managed resources)
crsctl stat res -t                # upper stack (CRSD-managed resources)
crsctl check cluster -all         # CSS, CRS and EVM on every node
ps -ef | grep -E 'ohasd|ocssd|crsd|evmd|gpnpd|gipcd|mdnsd|octssd'

4. The registries: OCR and OLR

The OCR (Oracle Cluster Registry) is the cluster's configuration database: every resource, its attributes, dependencies and placement policy, the node list, the networks, the ASM and database configuration that srvctl shows you. It lives on shared storage, inside an ASM disk group since 11.2, and crsd is the only writer. One crsd (the OCR master) takes an automatic backup every four hours and keeps a rolling set (daily, weekly); you can also export it manually. Up to five OCR locations can be configured for redundancy, though with ASM redundancy one or two is usual.

The OLR (Oracle Local Registry) exists because of a chicken-and-egg problem: OHASD must start before the cluster is formed, so it cannot read the OCR on shared storage. The OLR is a small local copy on each node ($GRID_HOME/cdata/<hostname>.olr) holding what the lower stack needs: the GPnP profile location, the local resources, the ASM discovery details. Lose the OLR and the node cannot even start OHASD; it can be restored from its own backup or re-created with rootcrs.sh.

ocrcheck                          # integrity and size of the OCR
ocrcheck -local                   # the OLR
ocrconfig -showbackup             # automatic 4-hourly / daily / weekly backups
ocrconfig -manualbackup
ocrconfig -export /backup/ocr_$(date +%F).dmp

5. The GPnP profile

The Grid Plug and Play profile is a small signed XML file, $GRID_HOME/gpnp/<hostname>/profiles/peer/profile.xml, with a cluster-wide copy under gpnp/profiles/peer/. It carries the few facts a node needs to find the cluster, as opposed to the OCR, which describes the cluster once found:

• the cluster name and the cluster GUID;
• the network interfaces and their roles (which subnet is public, which is the cluster interconnect, and since 12.1 which carries ASM traffic);
• the ASM discovery string (where to look for voting files and the SPFILE) and the ASM SPFILE location;
• the CSS lease duration and a sequence number; the file is signed so a tampered copy is rejected.

The boot sequence shows why it matters: OHASD starts gpnpd, gpnpd reads the profile, ocssd takes the discovery string from the profile and scans those disks for voting files (the voting file location is stored in the disk header, not in any file), and ASM learns its SPFILE location the same way. This is how a node can join a cluster before ASM or the OCR is available. When you change the interconnect with oifcfg setif, it is the GPnP profile that is updated, which is why the change needs a stack restart.

gpnptool get 2>/dev/null | xmllint --format -        # read the profile
gpnptool verify -p=profile.xml -w=... -wu=peer         # verify the signature
oifcfg getif                                          # interfaces and roles (from the profile)

6. Voting files: how the cluster decides who is alive

Every node sends a network heartbeat to every other node each second over the private interconnect, and writes a disk heartbeat to every voting file each second. The voting file is therefore two things: a second heartbeat path that still works when the network is gone, and a place where nodes record who they can and cannot see.

Why an odd number. A node must be able to read and write a majority of the voting files to stay in the cluster: 2 of 3, or 3 of 5. With one voting file, losing it kills the cluster; with two, losing one still kills it (one is not a majority of two). So you configure one, three or five. In an ASM disk group the number is fixed by the redundancy: external → 1, normal → 3, high → 5, and each copy must be in a different failure group. An extended cluster uses a quorum failure group on a third site (usually a small NFS share) to hold the third vote.

Split-brain and the tie-break. If the interconnect fails between two halves of the cluster, both halves are still alive and both can still write to the voting files. Each node writes into the voting file the list of nodes it can see; by reading everyone else's list, a node learns the cohorts. The rule since 12.2 (and in effect before) is: the larger cohort survives; if the cohorts are equal, the one containing the lowest node number survives, unless css_critical tips the balance. The losers get CRS-1607 / CRS-1608 and fence themselves. Since 11.2.0.2 fencing is usually a rebootless restart: ocssd kills the I/O-capable processes and restarts the stack rather than rebooting the OS; it reboots only if that cannot be done quickly enough.

Why fencing is non-negotiable. A node that has lost the cluster but still holds datafiles open could write a stale block. Eviction is RAC's guarantee that this cannot happen: the node is removed before the survivors recover its redo and take ownership of its blocks.

crsctl query css votedisk
crsctl get css misscount ; crsctl get css disktimeout
crsctl set server css_critical yes       # 12.2+, then restart the stack on that node
# where to look after an eviction: ocssd.trc, alert.log (GI), CHM data via oclumon / ahf

7. Cache Fusion: how the buffer caches stay consistent

Before 9i, if instance 2 needed a block that instance 1 had modified, instance 1 had to write it to disk and instance 2 read it back: "disk pinging", and the reason Oracle Parallel Server was slow for anything but partitioned workloads. Cache Fusion replaced that disk round-trip with a memory-to-memory transfer over the interconnect, handled by the Global Cache Service (GCS) and tracked in the Global Resource Directory (GRD).

The GRD. For every block (strictly, every resource) in use, one instance is the master and keeps the authoritative record: which instances hold the block, in what mode (null, shared, exclusive) and in what role (local, when only one instance has it; global, when copies exist in several caches). Mastership is spread across instances by hash, and Dynamic Resource Mastering (DRM) moves the mastership of an object to the instance that uses it most, so that most requests become local.

Block transfers. When instance A needs a block it does not hold, its foreground asks the master. If the master itself holds the block, it ships it: a 2-way transfer. If a third instance holds it, the master forwards the request and that instance ships the block to A: a 3-way transfer, which is why Cache Fusion never needs more than three hops however many nodes there are. The block arrives either as a current block (the latest version, for a write) or as a consistent-read (CR) block built by the holder at the requested SCN. The LMS processes do the shipping; a sending instance that gives up a current block keeps a past image (PI) until the block is written to disk, so that recovery can rebuild it.

Enqueues (row locks, table locks, library cache locks) go through the parallel Global Enqueue Service (GES), handled by LMD0 and LCK0.

What you see in AWR. gc cr block 2-way, gc current block 3-way are normal Cache Fusion waits; healthy values are well under a millisecond on a good interconnect. gc buffer busy acquire/release means several sessions on several nodes want the same block (right-growing indexes, sequence-less keys). gc cr block lost / gc current block lost mean the interconnect dropped packets: a network problem, not a database problem. gcs log flush sync is LGWR on the sending side flushing redo before a current block can be shipped: a slow redo disk shows up as a Cache Fusion wait.

select * from gv$instance_ping;                       -- latency between instances
select * from gv$cr_block_server;                     -- CR block serving stats
select inst_id, name, value from gv$sysstat where name like 'gc%lost%';
-- AWR: RAC Statistics section, Global Cache Load Profile, Top timed events

8. Instance recovery in RAC

When an instance dies, Clusterware evicts the node if needed, then LMON on a surviving instance runs reconfiguration: the GRD entries mastered by the dead instance are remastered, and blocks it held are frozen until recovery. SMON on one surviving instance reads the dead instance's online redo thread from shared storage and applies it: first the cache recovery (roll forward, using PIs where present), then undo is rolled back as usual. The database stays open throughout; only the blocks needing recovery are unavailable, briefly. This is the difference from a failover cluster, which must mount and open the whole database on the standby node.

9. Services, VIPs and the SCAN in one paragraph

Clients never connect to an instance; they connect to a service, a named workload that runs on a chosen set of instances. The SCAN is one DNS name for the whole cluster; its listeners hand each connection to the least-loaded instance offering the requested service. Each node has a VIP that moves to another node when the node dies, so clients get an immediate error instead of a TCP timeout. The Network tab covers each of these in depth.

10. ASM in one paragraph

Automatic Storage Management is Oracle's volume manager and file system for database files, running as an instance (+ASM1) alongside the database on each node, or as three instances in the cluster with Flex ASM. It stripes every file across all disks of a disk group and mirrors at the extent level across failure groups. The voting files, OCR and ASM SPFILE live inside it. The Storage tab covers it in depth.

11. ACFS and ADVM

ADVM (ASM Dynamic Volume Manager) carves general-purpose volumes out of an ASM disk group and presents them to Linux as /dev/asm/<volume>-<n>. ACFS (ASM Cluster File System) is a POSIX cluster file system formatted on an ADVM volume and mounted on every node at once. It gives you: a shared place for anything that is not a datafile (application files, exports, GoldenGate trail files, shared Oracle homes); read-only and read-write snapshots; replication to another cluster; tagging, encryption and auditing; and it is managed as Clusterware resources (ora.<dg>.<vol>.acfs) so it mounts in the right order at startup. Database files on ACFS are supported since 11.2.0.3/12c but are uncommon; ASM directly is the norm. It needs kernel drivers (oracleacfs, oracleadvm, oracleoks) loaded by ora.drivers.acfs, which is why a kernel update can break ACFS until the matching drivers are installed.

asmcmd volcreate -G DATA -s 100G appvol
mkfs -t acfs /dev/asm/appvol-123
srvctl add filesystem -device /dev/asm/appvol-123 -path /u02/app -user oracle
acfsutil snap create -w before_release /u02/app
acfsutil info fs /u02/app

12. How the architecture changed from 10g to 26ai

Version details above are summarised from Oracle's documentation and release notes as understood at the time of writing; confirm against the release notes for your exact release update, and label anything you rely on in production as documented or inferred, as this site does.

13. Cluster startup, start to finish

1. systemd starts oracle-ohasd; ohasd reads the OLR and starts its four agents.
2. mdnsd, gpnpd, gipcd, evmd come up; the GPnP profile is available.
3. cssdmonitor and cssdagent start, then ocssd: it discovers the voting files via the ASM discovery string, starts heartbeating and joins (or forms) the cluster: CRS-1601: CSSD Reconfiguration complete.
4. octssd checks the clock; HAIP puts link-local addresses on the private interfaces.
5. ASM starts and mounts the disk groups, which exposes the OCR.
6. crsd starts, reads the OCR and spawns its agents.
7. The agents start the network resource, node VIPs, SCAN VIPs and listeners, the local listener, ONS, ACFS file systems, then the database instances and, last, the services.

Section 5's Grid & CRS lab tab plays this sequence on two nodes with the matching log lines.

1. The networks a RAC node has

A RAC node is on at least two networks and usually four. Each has a different job, a different failure behaviour and different rules.

The names "public interconnect" and "private interconnect" are sometimes used, but Oracle's terms are public network and private interconnect (or cluster interconnect). Only the private one is an interconnect in the RAC sense: it connects instances to each other.

2. The public network: host IP, VIP, SCAN

Three kinds of address live on the public subnet, and understanding why each exists is most of RAC networking.

Host IP. The server's own address (node1 → 10.20.10.11). Used for SSH, for the installer's inter-node copying, and as the HOST in the node's local listener endpoint alongside the VIP. Clients should never use it directly.

Node VIP. Each node has a virtual IP (node1-vip → 10.20.10.21) managed by Clusterware as ora.node1.vip. While the node is healthy the VIP sits on its public interface as a secondary address. When the node fails, Clusterware brings the same VIP up on a surviving node. On that node nothing listens for database connections on the failed node's VIP, so a client that tries to connect gets an immediate connection refused and moves to its next address, instead of waiting out a TCP connect timeout, which can be minutes on some platforms. That is the whole purpose of the VIP: fast failure, not failover. When the node comes back the VIP returns to it.

SCAN. The Single Client Access Name (11.2+) is one DNS name for the whole cluster, rac-scan.corp.example, resolving round-robin to three addresses on the public subnet (or served by GNS). Each address is a SCAN VIP with a SCAN listener on top, and the three pairs are spread across nodes by Clusterware: with two nodes, one node carries two. Clients connect to the SCAN name and port 1521; a SCAN listener knows every instance of every service in the cluster (because each instance registers with it through remote_listener) and redirects the client to the local listener of the best instance. The client then talks to that node's VIP. Because of the SCAN, adding or removing a node never changes a client connect string.

srvctl config scan ; srvctl config scan_listener
srvctl status scan -verbose
nslookup rac-scan.corp.example                      # must return 3 addresses
lsnrctl status LISTENER_SCAN1                        # services registered (grid user)
show parameter remote_listener                       # 'rac-scan.corp.example:1521'
show parameter local_listener                        # (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip)(PORT=1521))

Listener registration. Each instance's LREG process (PMON before 12c) registers its services with the local listener and with the three SCAN listeners every minute, or on demand with ALTER SYSTEM REGISTER. It also sends load statistics, which is what the SCAN listener uses for server-side load balancing. A service missing from lsnrctl status is almost always a remote_listener or DNS problem, not a listener problem.

Multiple public networks (12c+). You can define a second network resource (ora.net2.network) with its own VIPs, SCAN and listeners, for instance to separate application traffic from backup or replication. Each service is then bound to a network.

GNS. The Grid Naming Service lets the cluster hand out its own VIP and SCAN addresses by DHCP and answer DNS for a delegated subdomain. It removes the need to pre-register addresses in corporate DNS and is what Flex Clusters used for leaf nodes; most on-premises clusters still use static addresses in DNS.

3. The private interconnect

The interconnect carries three things: the CSS network heartbeat (tiny, once a second, latency-sensitive), Cache Fusion block transfers (8 KB blocks or multiples, bandwidth- and latency-sensitive) and GES messages (small, many). ASM also uses it for its own metadata traffic, and with Flex ASM the database-to-ASM traffic can run on it or on its own network.

Protocol. On Linux the database uses UDP over IP for Cache Fusion (Exadata uses RDS over InfiniBand/RoCE). UDP has no retransmission of its own: a dropped packet becomes a gc cr block lost wait and a retry at the Oracle layer, which is why switch buffer drops and MTU mismatches hurt RAC far more than they hurt a web server. The Clusterware daemons use TCP and UDP through GIPC.

Requirements that are not optional.

• Dedicated and non-routed. No default gateway, no other hosts, its own VLAN(s). Oracle will not support Cache Fusion over a shared or routed network.
• Same subnet on every node, interface names consistent across nodes (ens2f0 is the interconnect on every node), and the role recorded in the GPnP profile with oifcfg.
• Redundancy. Either bonding (active-backup; LACP needs switch stacking) or, preferably, HAIP with two to four separate interfaces on separate switches.
• Jumbo frames (MTU 9000) end to end: NICs, switches, and the DWDM or inter-site links in an extended cluster. A single 1500 byte hop silently fragments every block transfer.
• Multicast on 230.0.1.0/24 and 224.0.0.251 (11.2.0.2+) for mdnsd and the GPnP bootstrap.
• Latency: sub-millisecond round trip in one hall; an extended cluster should stay under roughly 1–2 ms (practically, under about 100 km of fibre) or Cache Fusion waits dominate.

HAIP. Highly Available IP (11.2.0.2+) is Clusterware's own redundancy for the interconnect. You give Clusterware up to four private interfaces; it puts one link-local address (169.254.x.x, chosen from four sub-ranges) on each, and the database and ASM bind to those 169.254 addresses, not to your 192.168 ones (gv$cluster_interconnects shows them). Traffic is load-balanced across the interfaces. If an interface or its switch fails, gipcd moves that interface's 169.254 address onto a surviving interface in under a second; the database never notices beyond a brief wait spike. HAIP and bonding can be combined, but HAIP alone with two switches is the common and simplest design.

oifcfg getif                                   # ens2f0 192.168.201.0 global cluster_interconnect,asm
ip -4 addr show | grep 169.254                 # the HAIP addresses
crsctl stat res ora.cluster_interconnect.haip -init
select * from gv$cluster_interconnects;        -- what each instance actually uses
ping -M do -s 8972 192.168.201.12              # proves MTU 9000 end to end (8972 + 28 header)
netstat -su | grep -iE 'receive errors|buffer'  # UDP drops = interconnect trouble

Parameters you should not set. cluster_interconnects overrides HAIP with fixed addresses; it disables HAIP failover and is for troubleshooting or very specific designs only. Leave the UDP buffer sizes at the values the preinstall RPM sets (net.core.rmem_max 4 MB and friends) unless Oracle Support asks.

4. Time and name resolution

Time. Clusterware will not join a node whose clock is far off. Run chronyd (or ntpd) with the slew option; octssd then stays in observer mode. Without an external time source, octssd adjusts the clocks itself, which works but leaves the cluster drifting from the rest of the estate.

DNS. Node names, VIP names and the SCAN must resolve forward and reverse, and every node must see the same answers. The SCAN must not be in /etc/hosts (only one address would ever be used); node and VIP names may be. cluvfy comp nodecon and cluvfy comp scan check all of this before an install.

5. Ports and what listens where

Firewalls between nodes on the private network are not supported; on the public network, allow the listener and ONS ports from clients and all ports between the nodes themselves.

6. Services: the thing clients really connect to

A service is a name for a workload, created with srvctl add service and stored in the OCR. It lists the instances where it prefers to run and the instances available to take over. Clients ask for the service in their connect string; the SCAN listener places them on an instance that currently offers it. Services are also where you attach the client-facing policies: load-balancing goals, TAF, Application Continuity, draining. The Application connectivity tab covers these. The network point is simple: because the connect string names a service and the SCAN, it never names a host, so the topology can change beneath it.

srvctl add service -db orcl -service sales -preferred orcl1,orcl2 -available orcl3
srvctl start service -db orcl -service sales
srvctl status service -db orcl
lsnrctl services LISTENER_SCAN1 | grep -A2 sales

7. The extended-cluster network

Two sites make three network problems harder. The public subnet must be one layer-2 domain across both sites, because a VIP from site A must be able to come up at site B with the same address; this is normally a stretched VLAN over the inter-site links, or an overlay (VXLAN/EVPN) presenting one. The interconnect must be extended too: each private VLAN continues across its own DWDM wavelength or dark-fibre pair, with jumbo frames intact and latency low enough for Cache Fusion. And the quorum network to the third site's voting file must be reachable independently of the inter-site links, or the third vote cannot break a tie when those links fail. Section 7 of this site models exactly these three paths.

8. Reading network trouble from the database side

1. What "shared storage" means

Every node must be able to read and write every datafile, every online redo log of every thread, the control files, the SPFILE, the voting files and the OCR, at the same time. The storage must therefore be presented to all nodes concurrently, and whatever sits on it must be a cluster-aware file system or volume manager. An ordinary ext4 or XFS volume mounted on two nodes would be corrupted within seconds, because each kernel caches metadata as if it were alone.

Whatever the transport, Oracle's requirement is the same: the device must support concurrent access with O_DIRECT semantics and honour write ordering, and it must be visible under the same name (or same ASM label) on every node.

2. The Linux layer: multipath and device names

A LUN presented over two fabrics appears to Linux as four block devices (/dev/sdc, /dev/sdk, …), one per path. dm-multipath folds them into one device (/dev/mapper/DATA_01) that keeps working when any path fails and spreads I/O across the live ones. The /dev/sdX names can change between reboots, so ASM must never be pointed at them; it uses the multipath alias, a udev rule, ASMLib (older) or the ASM Filter Driver (AFD, 12.1.0.2+), which also blocks any non-Oracle write to an ASM disk, including an accidental dd or mkfs. Device ownership must be grid:asmadmin, mode 0660, on every node.

multipath -ll                                  # paths per LUN, policy, ALUA state
/etc/multipath.conf: alias DATA_01 for each wwid; path_grouping_policy group_by_prio
asmcmd afd_lsdsk                               # disks labelled by the ASM Filter Driver
asmcmd afd_label DATA01 /dev/mapper/DATA_01    # label once, visible from every node
udevadm info -q property /dev/mapper/DATA_01   # udev route instead of AFD

3. ASM: the concepts

Automatic Storage Management is a volume manager and file system in one, written for Oracle files. You give it raw devices; it gives the database files. It runs as its own instance (+ASM1) with a small SGA and no datafiles of its own; its "data dictionary" is metadata written on the disks themselves.

• Disk: one LUN or device, with an ASM disk header that records its disk group, its name, its failure group and, if present, the voting file.
• Disk group: the unit of allocation and the thing you mount; +DATA, +RECO, +GRID. Every file is striped across all disks of its group.
• Failure group: a set of disks that fail together: one array, one controller, one shelf, one site. Mirroring is always between failure groups, never within one. If you do not name failure groups, each disk is its own.
• Allocation unit (AU): the chunk a disk is divided into: 1 MB default before 12.2, 4 MB since; larger for very large databases. A file is a list of AUs.
• Extent: the file-level unit; one AU for small files, growing to 4 and 16 AUs for larger files (variable extents) to keep the extent maps small.
• Striping: coarse (one AU at a time, for datafiles) or fine (128 KB stripes, for redo and control files, to spread one write across disks and cut latency).
• Mirroring: at the extent level. Each extent has a primary copy and, depending on redundancy, one or two mirror copies in other failure groups. There is no mirror disk; the mirrors are spread everywhere.
• Rebalance: whenever disks are added or dropped ASM moves extents so each disk holds a proportional share. It runs online, in the background, at a power you choose.

4. Redundancy types

The extended-cluster pattern used in section 7: +DATA normal redundancy with failure group FG_A on the site A array and FG_B on site B, so every extent exists at both sites; +GRID normal redundancy with three failure groups: one disk per site plus a quorum failure group at site C that holds only a voting file. preferred_read.enabled (12.2; asm_preferred_read_failure_groups before) makes each instance read from its own site's copy so reads do not cross the inter-site link.

Usable capacity. Normal redundancy halves it, high divides by three, and ASM additionally wants free space to re-mirror after a failure. REQUIRED_MIRROR_FREE_MB and USABLE_FILE_MB in v$asm_diskgroup tell you the real figure; a negative USABLE_FILE_MB means a disk failure could not be fully re-protected.

5. What happens when a disk fails

1. An I/O to the disk fails. ASM takes the disk offline and starts serving its extents from the mirror copies. The database sees nothing but a wait spike.
2. The disk repair timer starts: disk_repair_time (default 3.6 h before 12.2, 12 h from 12.2). While it runs ASM tracks changed extents in a staleness registry and, if the disk comes back (path failure, controller reboot), performs a fast mirror resync of only those extents (11g R1+).
3. If the timer expires, ASM drops the disk and rebalances, re-creating the lost copies on the remaining disks of other failure groups. Now a second failure is survivable again, but the rebalance costs I/O and time, and needs the free space from the previous paragraph.
4. A whole failure group going offline (an array or site) uses failgroup_repair_time (default 24 h, 12.2+) instead, giving operations a day to bring a site back before ASM gives up on it.

select name, state, mode_status, repair_timer from v$asm_disk;
alter diskgroup DATA online all;                      -- after the array/path is back
alter diskgroup DATA set attribute 'disk_repair_time'='8h';
alter diskgroup DATA set attribute 'failgroup_repair_time'='48h';
asmcmd lsdg ; asmcmd lsop                             -- operations in flight
alter diskgroup DATA rebalance power 8;               -- 1..1024 (11.2.0.2+), default asm_power_limit

6. Disk group attributes and ASM parameters

7. ASM instance processes

That last point surprises people: ASM is not in the I/O path. If every ASM instance died, running databases would keep reading and writing until they needed a new extent or hit an error. This is also why Flex ASM works.

8. Flex ASM (12.1+)

Before 12c every node ran an ASM instance, and if it died the database instances on that node died with it. Flex ASM runs a configurable number of ASM instances (cardinality, default 3) anywhere in the cluster; a database instance on a node without ASM connects to a remote ASM instance over the ASM network (which can be the interconnect or a dedicated one), and if its ASM instance fails it simply reconnects to another. The +ASM3 (remote) label on node4 in sections 6 and 7 is this. Flex ASM is the default install mode since 12.2.

srvctl config asm                         # ASM instance count: 3 (Flex) or ALL
srvctl modify asm -count 3
asmcmd showclustermode                    # Flex mode enabled
select instance_name, status from gv$asm_client;   -- which DB instance uses which ASM

9. Where each file goes

10. Design rules that save you later

• Same size, same speed disks within a disk group: ASM balances by capacity, so one small disk fills first and caps the group.
• At least four LUNs per disk group, so stripes and rebalances have somewhere to go; multiples of the failure-group count.
• Never use LVM, partitions on only some nodes, or local names: multipath alias, udev or AFD on every node identically.
• Set compatible.* deliberately at creation. Raising later is online, lowering is impossible.
• Keep 15–20 % free in normal and high redundancy groups so a failure can re-mirror.
• In an extended cluster, +DATA must be normal redundancy with one failure group per site; external redundancy there means a site loss is a data loss. Section 7's storage scenarios show this.
• Voting files in ASM: let Clusterware manage them. Do not crsctl add css votedisk on raw devices in a modern cluster.
• Test a path failure and a controller failover before go-live: disktimeout is 200 s for a reason, but an array that takes longer to fail over will evict every node.

asmcmd lsdg                                     -- state, redundancy, usable MB
asmcmd lsdsk -k                                 -- disks with failure groups and sizes
select group_number, name, type, total_mb, free_mb, required_mirror_free_mb, usable_file_mb from v$asm_diskgroup;
select name, failgroup, mode_status, path from v$asm_disk order by 1;
crsctl query css votedisk ; ocrcheck
asmcmd spget ; asmcmd pwget --asm

1. What "high availability" means to an application

A RAC database can lose a node and stay open, but the application only benefits if three things happen on its side: it finds out quickly that its connections are dead, it reconnects to a surviving instance, and, ideally, the work that was in flight completes or is replayed instead of being thrown back at the user. Oracle has built a layer for each: FAN for finding out, pools with FCF (and the older TAF) for reconnecting, and Application Continuity for replaying. Everything hangs off services.

2. Services: the control point

A service is created with srvctl and lives in the OCR. The database only knows a service by name; every HA and load-balancing policy is an attribute of the service, so you can give the batch workload and the OLTP workload different behaviour on the same database. A PDB gets its own services in a multitenant database (-pdb).

srvctl add service -db orcl -service oltp -preferred orcl1,orcl2 -available orcl3,orcl4 \
  -failovertype AUTO -commit_outcome TRUE -failover_restore AUTO -replay_init_time 600 \
  -notification TRUE -clbgoal SHORT -rlbgoal SERVICE_TIME -drain_timeout 60 -stopoption IMMEDIATE
srvctl start service -db orcl -service oltp
srvctl relocate service -db orcl -service oltp -oldinst orcl1 -newinst orcl3 -drain_timeout 120

3. Connect strings that work with RAC

The connect string is half of the client's HA. A good one names the SCAN and a service, never a host, and sets the timeouts that make failure fast:

OLTP =
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)
    (ADDRESS_LIST = (LOAD_BALANCE = ON)
      (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.corp.example)(PORT = 1521)))
    (CONNECT_DATA = (SERVICE_NAME = oltp)))

• TRANSPORT_CONNECT_TIMEOUT = 3 s: give up on a dead address quickly and move on (this is what makes the VIP's "connection refused" useful).
• RETRY_COUNT × RETRY_DELAY: keep trying through a failover window instead of failing the first attempt.
• LOAD_BALANCE = ON: client-side spread over the SCAN addresses; the SCAN listener then does the real, load-aware placement.
• With a Data Guard standby, add a second ADDRESS_LIST for the standby SCAN and let the role-based service decide which cluster answers.
• Never use SID= in RAC; it binds to one instance.

4. Load balancing: connection time and run time

Connection load balancing happens when a connection is made. The SCAN listener knows the load of every instance (LREG sends it) and places the new connection according to the service's clbgoal: SHORT uses the response-time metric, LONG uses session counts. Runtime load balancing happens continuously inside a connection pool: the Load Balancing Advisory publishes, via FAN every few seconds, a percentage for each instance (gv$servicemetric), and pools such as UCP, OCI session pool and ODP.NET hand out connections to the instance the advisory favours. A pool therefore drifts work away from a slow instance before it fails. The same mechanism is how a service drains: the advisory tells the pool to stop using an instance that is about to go down.

select service_name, inst_id, goal, cpupercall, elapsedpercall from gv$servicemetric;
select name, goal, clb_goal, aq_ha_notifications from dba_services;

5. FAN and ONS: finding out

Without notification a client learns that a node died only when its next request times out, often after the TCP keepalive, which can be minutes. Fast Application Notification is Clusterware pushing events (service down on orcl1, node down, service up on orcl3, load advisory) out through ONS (Oracle Notification Service, ports 6200 local / 6100 remote). Oracle's connection pools subscribe to ONS automatically since 12c (auto-ONS: the pool learns the ONS endpoints from the SCAN), and on a down event they immediately close the dead connections, so the application gets its error in milliseconds and the pool refills from surviving instances. That reaction is Fast Connection Failover (FCF). FAN events also reach OCI clients and, through the aq_ha_notifications service attribute, PL/SQL callbacks.

srvctl status ons ; srvctl config nodeapps -onsonly
onsctl debug                                   # ONS connections and subscribers
# JDBC thin / UCP: oracle.ucp.jdbc.PoolDataSource ... setFastConnectionFailoverEnabled(true)
# Java system property for explicit ONS: -Doracle.ons.nodes=node1-vip:6200,node2-vip:6200

6. TAF: the original failover

Transparent Application Failover (8i, OCI-based) is a reconnect feature in the Oracle client library. When the connection breaks, the OCI client itself opens a new session on a surviving instance using the same connect descriptor, re-authenticates, and, with FAILOVER_MODE TYPE=SELECT, re-executes an open SELECT and discards rows already fetched so the cursor appears to continue. It can be configured in the client's tnsnames.ora or server-side on the service (-tafpolicy BASIC), which is preferred so every client gets it.

Its limits define why Application Continuity exists:

• Only OCI clients (SQL*Plus, OCI, ODP.NET unmanaged, Pro*C). JDBC thin does not have TAF.
• Any transaction in flight is rolled back and the application receives ORA-25402 / ORA-25408; it must re-run the DML itself.
• Session state is lost: PL/SQL package variables, temporary tables, ALTER SESSION settings, unless the application re-creates them in a TAF callback.
• PRECONNECT keeps a second idle session open on the backup instance, doubling session counts; rarely used.

(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 30)(DELAY = 5))   -- client side
srvctl modify service -db orcl -service rpt -tafpolicy BASIC -failovertype SELECT   -- server side
select username, failover_type, failover_method, failed_over from gv$session where username is not null;

7. Application Continuity: replaying the work

Application Continuity (12.1) hides an outage from the application by replaying the in-flight database request on a surviving instance. A request is the unit of work between a connection being checked out of a pool and checked back in. The replay driver (oracle.jdbc.replay.OracleDataSourceImpl in JDBC, or OCI with the session pool) records every call in the request: SQL, binds, PL/SQL, the session state changes it made. If the connection breaks before the request completes, the driver opens a new connection to the service, restores the session state, asks Transaction Guard whether the last transaction committed (using the Logical Transaction ID, LTXID, embedded in every session), and if it did not, re-executes the recorded calls in order, checking that each one returns the same results (the same rows, the same row counts, the same SCN-relative outcome). If anything differs, replay is abandoned and the application gets the original error, exactly as it would have without AC. If all matches, the request continues as if nothing happened.

Requirements and realities:

• The service must have -failovertype TRANSACTION and -commit_outcome TRUE, and the application must use a supported pool/driver (UCP, WebLogic Active GridLink, the JDBC replay data source, OCI session pool, ODP.NET managed with AC, Tuxedo).
• Mutables such as SYSDATE, SYSTIMESTAMP, SYS_GUID, sequence NEXTVAL would change on replay; Oracle keeps and replays their original values when the schema owner is granted KEEP DATE TIME, KEEP SYSGUID, and the sequence has KEEP. Without that, a request using them is not replayable.
• Replay is disabled for a request that used autonomous transactions committed externally, external calls (UTL_HTTP, DBMS_PIPE, UTL_FILE), or XA, because those cannot be undone or verified.
• Session state: with -session_state DYNAMIC the driver re-creates NLS and other state; with STATIC it assumes the application sets it once.
• Replay must start within replay_init_time and is attempted once per outage.

Transparent Application Continuity (18c, -failovertype AUTO) removes the application changes: the server decides request boundaries itself by detecting when a session is in an implicit boundary (no open transaction, no open cursors, standard state) and enables and disables replay automatically, so applications that just use UCP or the OCI pool, with no explicit request marks and no code changes, get continuity. Most new deployments use AUTO.

-- grant mutable handling (schema owner)
grant keep date time to sales;  grant keep sysguid to sales;
alter sequence sales.order_seq keep;
-- see what would block replay (run the Application Continuity checker / ACCHK)
exec dbms_app_cont_report.acchk_set(...);   -- or the acchk tool shipped with the client
select * from dba_app_cont_statistics;       -- 19c+: requests, replays, protected calls
-- Transaction Guard directly
select client_id, lt_id from gv$session;     -- the LTXID
exec dbms_app_cont.get_ltxid_outcome(:ltxid, :committed, :call_completed);

8. TAF vs FCF vs Application Continuity

9. Planned maintenance: draining instead of failing

Most RAC outages seen by applications are planned: patching, node reboots, relocating a service. Since 18c the right pattern is to drain: srvctl relocate service or srvctl stop instance -drain_timeout 300 -stopoption IMMEDIATE sends a FAN service down (planned) event; FCF-enabled pools stop handing out connections to that instance and return each connection to the pool at its next check-in, moving it to a surviving instance; after the timeout, anything still connected is handled by AC if possible and otherwise disconnected. A well-configured application sees zero errors during a rolling patch. The ora.<db>.<service>.svc resource's DRAIN_TIMEOUT and the database's -stopoption interact: TRANSACTIONAL waits for transactions to end, IMMEDIATE does not.

10. Testing it, and what to watch

• Test with a kill, not a stop. srvctl stop instance is a planned event; kill -9 of PMON or pulling a cable is what you are protecting against. Section 6 and 7's scenarios and the Create & play lab are the thinking-model for that.
• dba_app_cont_statistics / gv$session columns FAILOVER_TYPE, FAILED_OVER show whether sessions actually came through protected.
• Pool metrics: connections recycled on a FAN event, requests replayed, replays failed and why (the AC checker report names the SQL or call that blocked replay).
• Session counts after failover. If two of four instances carry every session afterwards, your -preferred list or your pool's minimum size needs rebalancing; srvctl relocate service back after the node returns, or let the MANAGEMENT_POLICY be AUTOMATIC.
• Logon storms. A failed node sends every pool reconnecting at once; size PROCESSES and listener rate limits (RATE_LIMIT in listener.ora) for that moment, not for steady state.

Everything in this tab is documented behaviour in the Oracle RAC Administration and Deployment Guide and the JDBC/UCP developer guides; the recommended values (timeouts, drain windows) are practitioner defaults to start from, not Oracle mandates.