crsd errors after downgrading the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 on Oracle Linux 6

After an upgrade from Oracle 11.2.0.4 to the version 12.1.0.1 we hit a bug described in my post "Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015)". Afterwards we tried a downgrade of the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 but the crsd service does not start.

[root@rac1 crsd]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online 


The crsd daemon has the following error in the crsd.log logfile:


2015-02-14 8:10:53.631: [ CLSE][119007008]clse_get_auth_loc: Returning default authloc: /u01/app/11.2.0/grid/auth/crs/rac1
2015-02-14 8:10:53.631: [ default][119007008] AuthLoc /u01/app/11.2.0/grid/auth/crs/rac1
[ default][119007008]Failure 3 in trying to open AV key SYSTEM.version.activeversion

Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015) - 19971324

Applying the patch on a upgraded 12.1.0.1 Linux installation gives the following opatchauto output:

Configuration Validation: Successful

Patch Location: /u01/source/grid/19971324
Grid Infrastructure Patch(es): 17077442 17303297 19769486
DB Patch(es): 17077442 19769486

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/12.1.0/grid" ...
Patch "/u01/source/oracle12/19971324/17077442" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/17303297" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/19769486" successfully applied to "/u01/app/12.1.0".

Performing postpatch operations on CRS Home... Failed
Command "/u01/app/12.1.0/perl/bin/perl -I/u01/app/12.1.0/perl/lib -I/u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/rootcrs.pl -postpatch" execution failed:
Died at /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/crspatch.pm line 885.

Oracle 11gR2 Clusterware OCR and voting disks recovery

Problem:

After the accidentally deletion of the diskgroup DATA the Oracle cluster stack does not start on any node. The Oracle Cluster Registry (OCR) and voting disks are lost. The article shows how to recover them both.

[root@rac2 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [[clsuSlosFormatDiag called with non-error slos.]
] [0]
[root@rac2 ~]# crsctl query css votedisk
Located 0 voting disk(s).