Instance Hangs data gather

• If the instance is hanging and you can't log in to any instance, please do one of these 2 options instead:

1) Use "sqlplus -prelim / as sysdba" to get into sqlplus on one instance, and run the oradebug statements 2-3 times - 1 minute apart:

SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug -g all hanganalyze 3
SQL> oradebug -g all dump systemstate 267

This only needs to be done on one instance; the "-g all" will kick off collection on all the other instances.

-or-

2) Capture systemstate dumps from each node with an OS debugger (gdb):

ps -ef |grep diag
gdb $ORACLE_HOME/bin/oracle
print ksudss(266)

This must be done on each node.

RAC/CRS Troubleshooting

• Document 289690.1 Data Gathering for Troubleshooting CRS Issues 
• Document 556679.1 Data Gathering for Troubleshooting RAC Issues
• Document 135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)

• Document 399482.1 How to recreate OCR/Voting disk accidentally deleted 
• Document 265769.1 Troubleshooting CRS Reboots 
• Document 357808.1 Diagnosability for CRS / EVM / RACG 
• Document 559365.1 Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions 
• Document 341788.1 Recommendation for the Real Application Cluster Interconnect and Jumbo Frames 
• Document 279793.1 How to Restore a Lost Voting Disk in 10g 
• Document 219361.1 Troubleshooting ORA-29740 in a RAC Environment 

• Document 339939.1 Running Cluster Verification Utility to Diagnose Install 
• Document 316817.1 CLUSTER VERIFICATION UTILITY FAQ 
• Document 428681.1 OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE), including moving from RAW Devices to Block Devices 
• Document 363254.1 Applying one-off Oracle Clusterware patches in a mixed version home environment 
• Document 332257.1 Using Oracle Clusterware with Vendor Clusterware FAQ
• Document 759895.1 The ONS Daemon Explained In RAC/CRS environment
• Document 783456.1 CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage

Installing ASMLIB

Install rpm for asmlib , download from oracle or metalink.



Installing using ULN
===============

#up2date -i oracleasm-support oracleasmlib oracleasm- `uname -r`

configure oracleasm

==================
Run it as root on all RAC nodes.

#/etc/init.d/oracleasm configure
This creates following tasks
1)creates /etc/sysconfig/oracleasm config file
2)creates /dev/oracleasm mount
3)loads oracleasm kernel module


Mark DISK as ASM
==================
#/etc/init.d/oracleasm createdisk ASMDISK1 /dev/hda5
After all ASM disk created, run
#/etc/init.d/oracleasm scandisks

ASM disks misc commands

Which Disks Are Handled by ASMLib Kernel Driver?
=========================================
/sbin/blkid | grep oracleasm

/etc/init.d/oracleasm listdisks

ASMLib disks device entries are located under the /dev/oracleasm/disks directory:

# cd /dev/oracleasm/disks
# ls -l

Set below parameter for ASM , so ASM instance knows when new disk available.
===================================================
asm_diskstring                       string      ORCL:ASM*

Config File for asmlib
===============
/etc/sysconfig/oracleasm

Commands for oracleasm
=====================

/etc/init.d/oracleasm listdisk
/etc/init.d/oracleasmscandisk

Check version of ASMLIB
===================
#!rpm -qa|grep oracleasm
oracleasm-2.6.18-164.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5
oracleasm-2.6.18-164.el5-2.0.5-1.el5
oracleasmlib-2.0.4-1.el5
oracleasmlib-2.0.4-1.el5