Problem:
After the accidentally deletion of the diskgroup DATA the Oracle cluster stack does not start on any node. The Oracle Cluster Registry (OCR) and voting disks are lost. The article shows how to recover them both.
[root@rac2 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [[clsuSlosFormatDiag called with non-error slos.]
] [0]
[root@rac2 ~]# crsctl query css votedisk
Located 0 voting disk(s).
ocrcheck has a problem to connect to the disk since we deleted it. Also the voting disks are unavailabe. With the following command we can see where the voting is expected:
[root@rac2 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +DATA
First we check the backup of the OCR:
[root@rac2 ~]# ocrconfig -showbackup auto
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
rac1 2013/01/23 10:06:00 /u01/app/11.2.0/grid/cdata/rac-scan/backup00.ocr
rac1 2013/01/23 06:05:59 /u01/app/11.2.0/grid/cdata/rac-scan/backup01.ocr
rac1 2013/01/23 02:05:59 /u01/app/11.2.0/grid/cdata/rac-scan/backup02.ocr
rac2 2013/01/21 22:00:43 /u01/app/11.2.0/grid/cdata/rac-scan/day.ocr
rac2 2013/01/16 10:00:13 /u01/app/11.2.0/grid/cdata/rac-scan/week.ocr
An automatic backup is present. The action plan is to first recreate +DATA Diskgroup and then to restore the OCR to get the cluster started.
We stop the cluster and asm driver:
crsctl stop cluster -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rac1'
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
/etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
The detect the new asm disk and add it to multipath and create a parition:
iscsiadm --mode node --logoutall=all
discovery of the nodes:
iscsiadm -m discovery -t sendtargets -p 10.31.129.66
iscsiadm --mode node --loginall=all
edit the multipath file and add:
multipath {
wwid 36090a078e0728e870010d59b369e9812
alias DATA
}
restart multipath daemon
[root@rac1 ~]# /etc/init.d/multipathd restart
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]
Now start the cluster in exclusive mode (applies to 11.2.0.1):
[root@rac2 ~]# crsctl start crs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2679: Attempting to clean 'ora.diskmon' on 'rac2'
CRS-2681: Clean of 'ora.diskmon' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
Create the Linux partition:
[root@rac2 ~]# fdisk /dev/dm-12
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 1306.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
Command (m for help): p
Disk /dev/dm-12: 10.7 GB, 10742661120 bytes
255 heads, 63 sectors/track, 1306 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1306, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1306, default 1306):
Using default value 1306
Command (m for help): p
Disk /dev/dm-12: 10.7 GB, 10742661120 bytes
255 heads, 63 sectors/track, 1306 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/dm-12p1 1 1306 10490413+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
A restart was not nessecary despite fdisk requested it. We now create the diskgroup and it must have the same name as the one diskplayed by the "ocrcheck -config" command:
[oracle@rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jan 23 17:23:56 2013
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup DATA external redundancy disk 'ORCL:DATA' attribute 'COMPATIBLE.ASM' = '11.2';
Diskgroup created.
Finally we restore the OCR:
[root@rac2 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-scan/backup00.ocr
We can now start the crs daemon:
[root@rac2 ~]# crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
Recreation of the voting disk:
[root@rac2 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk 2d434a8876304f8bbfa4d17826d263fc.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
On all nodes of the cluster stop crs service, then scan asm disks and finally start crs again with the following three commands (you also have to add the iscsi lookup and the multipath config on each node):
crsctl stop crs -f
/usr/sbin/oracleasm scandisks
crsctl start crs
Finally we replace the OCR and voting disk to another diskgroup. The temporary DATA diskgroup can then be deleted.
[root@rac1 ~]# ocrconfig -add +SLICE1
[root@rac1 ~]# ocrconfig -delete +DATA
[root@rac1 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +SLICE1
[root@rac1 ~]# crsctl replace votedisk +SLICE1
Successful addition of voting disk a6febfb2c9d04feebf4e150f53206594.
Successful deletion of voting disk 2d434a8876304f8bbfa4d17826d263fc.
Successfully replaced voting disk group with +SLICE1.
CRS-4266: Voting file(s) successfully replaced
