Sun cluster. SVM stale databases

SYMPTOMS

A Two node SunCluster uses disk resource from two strorage arrays. Due to array/Disk failure one cluster node panic'ed. The surviving node tried to take over the diskset but failed to come up because the metadb (replica) database has become stale.

In dmesg something like this:

Cluster.Framework: [ID 801593 daemon.error] stderr: metaset: : : stale databases
Cluster.Framework: [ID 801593 daemon.error] stderr: metaset::: 50% replicas & 50% mediator hosts available, user intervention required

CAUSE

Due to disk failure a loss of 50% of disk replica occurred. The diskset is configured with only 2 logical disks. The logical disks were configured from RAID5 volumes each from one of 2 Storage Arrays. One Storage Array had problems and causing one logical disk to be inaccessible.


In diskset configurations, copies of the metadb are distributed among the disks in the diskset. With only 2 disks, two copies of the metadb are created in each disk. To insure consistency, SVM requires more than 50% of the metadb to be available or it will not allow automatic takeover of the diskset. With one disk not being accessible, only 50% of the metadb is available, effectively making the metadb stale.


From the 2 logical disks, several mirrored metadevices were created. As a result half of the diskset metadevices are in "Needs maintenance" state.

SOLUTION

Place the cluster device group in maintenance state

    # scswitch -m -D <device-group-name>

Manually takeover the diskset

# metaset -C take -f -s <disk-set-name>

Check the diskset rdb

# metadb -i -s  ora-set01
flags           first blk       block count
a m     luo t      16              1034            /dev/did/dsk/d2s7
a       luo t      1050            1034            /dev/did/dsk/d2s7
M                  unknown         unknown         /dev/did/dsk/d3s7
M                  unknown         unknown         /dev/did/dsk/d3s7 

So half of the replica disks is down. The quickest way to fix this is to delete the faulty disk (d3s7)
from the rdb - in effect we are making the replica 100% available from
the good disk (d2s7).

# metadb -s <disk-set-name> -d /dev/did/dsk/d3s7

NOTE: To find out which disk is d3s7 use scdidadm command

Release the diskset

# metaset -C release -s<disk-set-name>

Bring the device group back online.


# scswitch -z -D <device-group-name>-h node1

The cluster should come up all the way. Do additional commands to let the other node join the cluster.
Once the array is fixed. Let LVM re-scan the disks .

# metastat -i -s <disk-set-name>

NOTE: metastat -i is available from LVM 4.2.1 and up.

Add a new rdb on d3s7.

# metadb -s <disk-set-name> -a -c 2 /dev/did/dsk/d3s7

   

Re-enable ALL the diskset volumes that are in "Needs maintenance" state:

# metareplace -s <disk-set-name> -e d1 /dev/did/rdsk/d3s0

  




IN NON CLuster enviroment




Recovery procedure if the diskset is used outside of SunCluster:

1. Force a takeover the diskset:

# metaset -s <disk-set-name> -t -f

2. Delete the faulty replica:

# metadb -s <disk-set-name> -d

3.Release the diskset:

# metaset -s <disk-set-name> -r

4. Takeover the diskset normally:

# metaset -s <disk-set-name> -t