crsd errors after downgrading the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 on Oracle Linux 6

After an upgrade from Oracle 11.2.0.4 to the version 12.1.0.1 we hit a bug described in my post "Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015)". Afterwards we tried a downgrade of the Oracle clusterware from 12.1.0.1.6 to 11.2.0.4 but the crsd service does not start.

[root@rac1 crsd]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online 


The crsd daemon has the following error in the crsd.log logfile:


2015-02-14 8:10:53.631: [ CLSE][119007008]clse_get_auth_loc: Returning default authloc: /u01/app/11.2.0/grid/auth/crs/rac1
2015-02-14 8:10:53.631: [ default][119007008] AuthLoc /u01/app/11.2.0/grid/auth/crs/rac1
[ default][119007008]Failure 3 in trying to open AV key SYSTEM.version.activeversion

Oracle Database 10g HW contention in lob segemts

After a migration of a database to a faster machine the database was slow and appears to hung. The problem was a slow and big lob segment. The following bug was the reason for the behaviour:

Bug 6376915: ENQ: HW - CONTENTION WITH LOB SEGMENTS.

The bigger the lob segment, the more concurrent CPUs, the more write and read I/O => the bigger  the problem. 

Oracle LOBs + ASSM + many CPUs = enq: HW – contention

Our new hardware architecture had significant more  cores anf threads per CPU. The problem already existed with the old hardware but the new hardware made it really worse. Take a look at the statspack report:

Top 5 Timed Events                                         Avg %Total

~~~~~~~~~~~~~~~~~~                                        wait   Call

Event                                 Waits    Time (s)   (ms)   Time Wait Class

------------------------------ ------------ ----------- ------ ------ ----------

enq: HW - contention                224,968     658,323   2926   93.1 Configurat

Errors applying Grid Infrastructure PSU 12.1.0.1.6 (JAN2015) - 19971324

Applying the patch on a upgraded 12.1.0.1 Linux installation gives the following opatchauto output:

Configuration Validation: Successful

Patch Location: /u01/source/grid/19971324
Grid Infrastructure Patch(es): 17077442 17303297 19769486
DB Patch(es): 17077442 19769486

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/12.1.0/grid" ...
Patch "/u01/source/oracle12/19971324/17077442" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/17303297" successfully applied to "/u01/app/12.1.0".
Patch "/u01/source/oracle12/19971324/19769486" successfully applied to "/u01/app/12.1.0".

Performing postpatch operations on CRS Home... Failed
Command "/u01/app/12.1.0/perl/bin/perl -I/u01/app/12.1.0/perl/lib -I/u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/rootcrs.pl -postpatch" execution failed:
Died at /u01/app/12.1.0/OPatch/opatchautotemp/patchwork/crs/install/crspatch.pm line 885.

Desaster recovery in a nutshell for Oracle Database 10gR2

The article gives a brief overview how to restore and recover an Oracle database. Of course its a requirement to have a working rman backup of the database. We start at the point where the database is lost and we need to restore it from scratch using the backup files. In this scenario we use a controlfile and not a catalog database. Before you begin you need to make sure that the failed storage is available again and the directories and permissions are created and granted. 

Using direct I/O and the solaris dtrace tool

Using Solaris Dynamic Tracing (dtrace) can help to eliminate hotspots on files and give an insight to your system. Oracle ported dtrace to Oracle Linux which makes it just more interesting. This article gives a quick start using the dtrace facility and direct I/O. The detailed dtrace description can be found in its documentation.

Security aspects of an APEX-installation

The many advantages of using Oracle APEX may cause a department to consider using it for human resourc applications or for other sensitive company data. Then there is a question of security that arises. This article describes the security aspects of different types of installations in a production environments as well as important APEX security and configuration concepts.

Oracle 11gR2 Clusterware OCR and voting disks recovery

Problem:

After the accidentally deletion of the diskgroup DATA the Oracle cluster stack does not start on any node. The Oracle Cluster Registry (OCR) and voting disks are lost. The article shows how to recover them both.

[root@rac2 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [[clsuSlosFormatDiag called with non-error slos.]
] [0]
[root@rac2 ~]# crsctl query css votedisk
Located 0 voting disk(s).