crsd errors after downgrading the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 on Oracle Linux 6

After an upgrade from Oracle 11.2.0.4 to the version 12.1.0.1 we hit a bug described in my post "Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015)". Afterwards we tried a downgrade of the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 but the crsd service does not start.

[root@rac1 crsd]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online 


The crsd daemon has the following error in the crsd.log logfile:


2015-02-14 8:10:53.631: [ CLSE][119007008]clse_get_auth_loc: Returning default authloc: /u01/app/11.2.0/grid/auth/crs/rac1
2015-02-14 8:10:53.631: [ default][119007008] AuthLoc /u01/app/11.2.0/grid/auth/crs/rac1
[ default][119007008]Failure 3 in trying to open AV key SYSTEM.version.activeversion

Oracle Database 10g HW contention in lob segemts

After a migration of a database to a faster machine the database was slow and appears to hung. The problem was a slow and big lob segment. The following bug was the reason for the behaviour:

Bug 6376915: ENQ: HW - CONTENTION WITH LOB SEGMENTS.

The bigger the lob segment, the more concurrent CPUs, the more write and read I/O => the bigger  the problem. 

Oracle LOBs + ASSM + many CPUs = enq: HW – contention

Our new hardware architecture had significant more  cores anf threads per CPU. The problem already existed with the old hardware but the new hardware made it really worse. Take a look at the statspack report:

Top 5 Timed Events                                         Avg %Total

~~~~~~~~~~~~~~~~~~                                        wait   Call

Event                                 Waits    Time (s)   (ms)   Time Wait Class

------------------------------ ------------ ----------- ------ ------ ----------

enq: HW - contention                224,968     658,323   2926   93.1 Configurat

Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015) - 19971324

Applying the patch on a upgraded 12.1.0.1 Linux installation gives the following opatchauto output:

Configuration Validation: Successful

Patch Location: /u01/source/grid/19971324
Grid Infrastructure Patch(es): 17077442 17303297 19769486
DB Patch(es): 17077442 19769486

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/12.1.0/grid" ...
Patch "/u01/source/oracle12/19971324/17077442" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/17303297" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/19769486" successfully applied to "/u01/app/12.1.0".

Performing postpatch operations on CRS Home... Failed
Command "/u01/app/12.1.0/perl/bin/perl -I/u01/app/12.1.0/perl/lib -I/u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/rootcrs.pl -postpatch" execution failed:
Died at /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/crspatch.pm line 885.