Wikis - Page

Oracle RAC 11.2.0.1 - 2 Node Cluster Solution for ZCM

MigrationDeletedUser

0 Likes

over 3 years ago

Table of Contents:

- Introduction

- Resource Requirements

- Installation Plan

- Oracle Grid Infrastructure 11.2.0.1 Software Installation

- Post Installation Checks After Grid Installation

- ASM Disks Creation

- Oracle RAC 11.2.0.1 Software Installation

- RAC Database Creation Through DBCA

- Post Installation Checks After Oracle RAC Installation

- Troubleshooting

- ZCM (ZENworks Configuration Management) Installation With Oracle RAC

- Uninstallation Procedure

- Links and Downloads

Introduction

Oracle Real Application Clusters (RAC) is a database clustering solution that allows more than one instance to mount and open an Oracle database. In the event of a system failure, clustering ensures highest availability to users. Access to mission critical data is not lost. In simple words, it provides superior fault tolerance, load balancing, and highest scalability.

Oracle RAC is used for Active-Active cluster scenarios where more than one instance is running at any given time. In the event of a first instance server failure, the remaining server instance can continue processing the workload, and optionally failed sessions can be reconnected, thus ensuring zero downtime. This is very useful in case of an unscheduled outage.

Oracle Grid Infrastructure software provides Clusterware files(Crs voting disks) and Automatic storage management (ASM) packaged together. This is very much required for RAC operability.

Click to view.

Resource Requirements

Hardware resources

1. A blade server with 48GB RAM, 1TB HDD with SCSI, 2 CPUs, quad core

1. Oracle VM server 2.2.2

1. Oracle VM manager 2.2.0

1. Oracle Linux-Redhat5 x32 bit – 2 (node1 and node2)

1. Kernel Rpms

1. A VM on ESX with Windows/Linux to host ZCM

1. A VM on ESX with Linux-Redhat5 to host VM manager

Software resources

1. Oracle Grid infrastructure software 11.2.0.1

1. Oracle RAC DB software 11.2.0.1

1. ZCM 11.2 software

Installation Plan

Pre-installation tasks

1. Install Oracle VM server 2.2.2 on the above mentioned blade server.

1. Install Oracle VM manager 2.2.0. This is used to access the virtual machines created on VM server.

1. Download the Oracle Linux Redhat5 x32 bit iso file (OS with enhanced performance and few plugins).

1. Through VM manager, import the downloaded iso file to create virtual machines.
  
  Note: We can also use templates for the same. Refer to the VM manager documentation

1. Create two virtual machines using the Oracle Linux iso file each having 3 GB RAM, Swap memory twice as RAM, 60 HDD (approx- 10 GB for grid Rac, remaining for 5 shared disks), 2 NIC cards, 1 GB of temp space /tmp each.
  
  Root> grep MemTotal /proc/meminfo
  
  Root> grep MemTotal /proc/swapinfo
  
  These two machines will be the 2 nodes for oracle RAC.

1. Network Configuration
  
  These two machines must have:
  1. 1. A public IP address for each node, with the following characteristics:
      
      - 2 Static IP addresses with hostnames.
      
      - Resolvable by dns, entry in \etc\hosts, same subnet as virtual and SCAN addresses.
  1. 1. A virtual IP address for each node, with the following characteristics:
      
      - 2 Static IP addresses with hostnames.
      
      - Resolvable by dns, entry in \etc\hosts, same subnet mask as the public IP address and SCAN addresses.
      
      Note: VIP (virtual IP) is a public IP address that is attached to the public interface of the RAC node. VIP should be used by all clients to communicate with the database to ensure fast failover during an outage.
  1. 1. A Single Client Access Name (SCAN) for the cluster, with the following characteristics:
      
      - Three Static IP addresses with hostnames configured on the domain name server (DNS), so that the three IP addresses are associated with the name provided as the SCAN, and all three addresses are returned in random order by the DNS to the requestor. (You can request assistance from the administrator.)
      
      - Same subnet as public and virtual IP addresses.
```
Example:
bash> nslookup blr-srm-cluster
Server:         196.96.201.1
Address:        196.96.201.1#53
Name:   blr-srm-cluster.labs.blr.novell.com
Address: 196.96.94.226
Name:   blr-srm-cluster.labs.blr.novell.com
Address: 196.96.94.227
Name:   blr-srm-cluster.labs.blr.novell.com
Address: 196.96.94.228
```
      Note: Single Client Access Name (SCAN), is used to connect to databases within the cluster irrespective of which nodes they are running on. By default, the name used as the SCAN is also the name of the cluster. The default value for the SCAN is based on the local node name (where DB installed).
  1. 1. A private IP address for each node, with the following characteristics:
      
      - 2 Static IP addresses with hostnames
      
      - Need not be resolvable by dns, having its own subnet addresses, entry in \etc\hosts file
```
Example of a hosts file
127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
#node2
196.96.94.229   blr-srm-r11t.labs.blr.novell.com blr-srm-r11t   #public
196.96.94.225   blr-srm-r11p.labs.blr.novell.com blr-srm-r11p #Vip
#node1-localnode
196.96.94.221   blr-srm-r11l.labs.blr.novell.com blr-srm-r11l   #public
196.96.94.223   blr-srm-r11n.labs.blr.novell.com blr-srm-r11n  #Vip
#private ip
10.0.0.2       node2-priv.labs.blr.novell.com node2-priv
10.0.0.1       node1-priv.labs.blr.novell.com node1-priv 
```

1. Create Shared disks-SCSI:
  
  From Oracle VM manager, before switching on the machine, create 5 SCSI shared disks from local node- node1. 3 for clusterware files, 1 for fast disk recovery, 1 for RAC DB.
  1. 1. Navigate to VM manager > Resources > shared virtual disks > create.
      
      Create new virtual shared disks. Provide name (CRS1 or RAC DB) and disk space-approx 10GB each. Click Ok.
  1. 1. After creating 5 shared disks, attach the disks to the two nodes, first localnode node1 & then node2.
      
      Navigate to VM manager > Virtual machines > select a local node-node1> Configure->attach/detach shared virtual disks > add available disks > confirm.
  1. 1. Repeat the same for node2 in the same order. If we select CRS1-sda1 in node1 to add first, same should be maintained in node2.
  Note: Refer to the VM manager documentation for any clarification

1. Power On the configured machines.

1. Check if two nodes-node1 and node2 are inter pingable.

1. Check-ping /nslookup for public/private/virtual and scan IP addresses.

1. Disable firewall in both machines.

1. Disable SELinux on both machines.
  
  To check the status:
  
  Navigate to root> sestatus
  
  To disable:
  
  Run the edit /etc/selinux/config command.
  
  SELINUX=disabled
  
  reboot

Create users, user groups, directories and give permissions on both nodes from root.

create users - grid and oracle with common password across.


Example: 
user: grid, password: grid in both machines
user: oracle, password: oracle

Add user groups for 'grid' user:

groupadd -g 1000 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid


Example:
root> id grid
uid=1100(grid)gid=1000(oinstall)groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)

Add user groups for 'oracle' user:

groupadd -g 1300 dba
groupadd -g 1301 oper
useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle


Example:
root> id oracle
gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)

Create directories and give permissions:

root> mkdir -p /u01/app/oracle
                mkdir -p  /u01/app/11.2.0/grid
                chown -R grid:oinstall /u01
                chown -R oracle:oinstall /u01/app/oracle
                chmod -R 775 /u01

1. Create partitions for the added shared disks from node1-localnode.
  1. 1. from root, to list all disk
```
root@node1> fdisk -l
	
	
```
  1. 1. to create partition for each SCSI disks
```
root@node1 > fdisk /dev/sda
	
	
```
  Note: Every disk should have only 1 primary partition. After partition, we have sda1,sdb1,sdc1,sdd1,sde1.

After the partition from node1, we should inform the kernel of the new changes in all other nodes.

Go to node2 
root@node2> partprobe
root@node2>fdisk -l 
Verify that it lists all the partitioned disks in node2 as well.

1. Install and configure ASM library drivers
  
  Oracle ASM) library driver (ASMLIB) simplifies the configuration and management of the disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.
  1. 1. Download the drivers from the link below on both nodes:
      
      http://www.oracle.com/technetwork/server-storage/linux/downloads/index-088143.html
  1. 1. Install the packages from root on both nodes
```
root> rpm -oracleasm-support-2.1.7-1.el5
	oracleasmlib-2.0.4-1.el5
	oracleasm-2.6.18-274.el5xen-2.0.5-1.el5
	
	
```
  1. 1. To configure, run the /usr/sbin/oracleasm configure -i command on both nodes. Perform the following settings:
```
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
```
  1. 1. To load kernel module, run the /usr/sbin/oracleasm init on both nodes command.

Mark shared disks as ASM disks.

From node1-root, create/mark ASM disks for all 5 shared disks


Example:
root@node1> /usr/sbin/oracleasm createdisk CRS1 /dev/sda1
root@node1> /usr/sbin/oracleasm createdisk CRS2 /dev/sdb1
root@node1> /usr/sbin/oracleasm createdisk CRS3 /dev/sdc1
root@node1> /usr/sbin/oracleasm createdisk RACDB /dev/sdd1  
root@node1> /usr/sbin/oracleasm createdisk FRA /dev/sde1
root@node1> /usr/sbin/oracleasm listdisks

Note:Disk name should be in capitals, and better to keep it simple & relevant.

For the node2 to know about the change, run the following commands from root-node2:

root@node2> /usr/sbin/oracleasm scandisks
root@node2> /usr/sbin/oracleasm listdisks
All the marked disks should be visible.

Create a .bash_profile or .profile on both nodes

To create a .bash_profile for user 'grid', save and source:

grid>vi .bash_profile
PATH=$HOME/bin:/u01/app/11.2.0/grid/bin:$PATH
export PATH
umask 022
ORACLE_SID= ASM1
export ORACLE_SID
ORACLE_BASE=/u01/app/grid
export ORACLE_BASE
ORACLE_HOME=/u01/app/11.2.0/grid
export ORACLE_HOME
export TEMP=/tmp
export TMPDIR=/tmp
TNS_ADMIN=/u01/app/11.2.0/grid/network/admin
export TNS_ADMIN
LD_LIBRARY_PATH=/u01/app/11.2.0/grid/lib
export LD_LIBRARY_PATH
CLASSPATH=/u01/app/11.2.0/grid/jlib:/u01/app/11.2.0/grid/JRE:$CLASSPATH
export CLASSPATH

Note: ORACLE_SID= ASM1 for node1, ASM2 for node2 and so on.

To create a .bash_profile for user 'oracle', save and source the below

oracle>vi .bash_profile
PATH=$HOME/bin:/u01/app/oracle/product/11.2.0/dbhome_1/bin:$PATH
export PATH
ORACLE_BASE=/u01/app/oracle
export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export ORACLE_HOME
ORACLE_SID=orcl1
export ORACLE_SID
LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
export TNS_ADMIN
CLASSPATH=/u01/app/oracle/product/11.2.0/dbhome_1/jlib:$CLASSPATH
export CLASSPATH
ORACLE_UNQNAME=orcl
export ORACLE_UNQNAME
export TEMP=/tmp
export TMPDIR=/tmp

Note: ORACLE_SID= orcl1 for node1, orcl2 for node2 and so on.

1. To check the Kernel parameters:
  
  If we have missing kernel rpms or missing packges on Linux, It will be identified when the Grid installer is triggered.
  1. 1. If we have configured oracle unbreakable Linux (licensed support), then we can install it by running the following commands:
```
up2date --whatprovides libstdc  .so.5 or
./runInstaller downloadUpdate
```
  1. 1. If we do not have the support, we can use yum-server. Refer to the link below and install all the missing packages indicated by the installer.
      
      http://public-yum.oracle.com

To check the NTP service for time synchronization on both node, do the following:

Edit ntpd file to add -x flag
root>vi /etc/sysconfig/ntpd
#Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
#Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=no
# Additional options for ntpdate
NTPDATE_OPTIONS=""
root> service ntpd restart

Note: For SUSE systems, NTPD_OPTIONS="-x -u ntp"
Check the time on both nodes, should be for cluster services to operate properly.

1. To check Xterm display, do one of the following:
  1. 1. For GUI, while accessing with Putty, enable the SSH-X11 forwarding.
  1. 1. grid/oracle@node1>DISPLAY=<your local workstation>:0.0
```
export DISPLAY
TEST X CONFIGURATION BY RUNNING xterm
grid@node1>xterm &
```
      Note: The xming server should be installed and running from where putty is accessed.

Configure passwordless SSH connectivity for User equivalence on both nodes.

To configure passwordless SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (grid, oracle), as SSH ignores a private key file if it is accessible by others. In the examples that follow, the DSA key is used.

grid> mkdir ~/.ssh
grid> chmod 700 ~/.ssh
grid> /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):            [Enter]
Enter passphrase (empty for no passphrase): [Enter]
Enter same passphrase again: [Enter]
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is: 57:21:d7:d5:54:29:4c:12:40:23:36:e9:6e:2f:e6:40

Note: Repeat the above steps for all nodes.

grid@node1>touch ~/.ssh/authorized_keys
grid@node1>ls -l ~/.ssh
 grid@node2>ssh node1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'node1 (196.96.94.221)' can't be established.
RSA key fingerprint is 66:65:a6:99:5f:cb:6e:60:6a:06:18:b7:fc:c2:cc:3e
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,196.9694.221' (RSA) to the  list of known hosts.
grid@node1's password: xxxxx (grid)
grid@node1>ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'node2 (196.96.94.229)' can't be established.
RSA key fingerprint is  30:cd:90:ad:18:00:24:c5:42:49:21:b0:1d:59:2d:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2,196.96.94.229' (RSA) to the list of known hosts
grid@rode1's password: xxxxx (grid)
grid@node1>ls -l ~/.ssh
grid@node1>scp ~/.ssh/authorized_keys node2:.ssh/authorized_keys
grid@node2's password: xxxxx
authorized_keys
grid@node1>chmod 600 ~/.ssh/authorized_keys
grid@node2>chmod 600 ~/.ssh/authorized_keys

Note: Perform the above steps in all the nodes for 'grid' and 'oracle' user.

Test:
grid@node1>ssh "date:hostname" node2
grid@node2>ssh "date:hostname" node1

1. Download and copy:
  1. 1. Download Oracle Infrastructure Grid 11.2.0.1 software to node1 only from the link: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html
      
      Login with grid user, copy it to /home/grid, give permissions and extract.
  1. 1. Download Oracle RAC 11.2.0.1 software to node1 only from the link:
      http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html
      
      Login with oracle user credentials, it to /home/grid, give permissions and extract.
      
      Note: GRID infrastructure version >= Oracle RAC version.

1. Before starting the Oracle Grid installation, run a cluster requirements check with cvuqdisk utility that comes with the software.
  
  For cvuqdisk verification utility, we need to install an rpm package on both nodes by performing the following:
  1. 1. root>export CVUQDISK_GRP=oinstall
  1. 1. Copy /home/grid/grid from node1 to node2 /home/grid. The 'grid' folder contains cvuqdisk utility obtained when the software was copied.
  1. 1. rpm -iv cvuqdisk-1.0.7-1.rpm
      
      cvuqdisk-1.0.7-1
      
      Verify cvuqdisk install on both nodes by running the following command:
      
      root> ls -l /usr/sbin/cvuqdisk
      
      Execute ./runcluvfy.sh script for both nodes, by running the following command:
      
      grid>/home/grid/grid/runcluvfy.sh stage -pre crsinst -n node1,node2 -fixup –verbose

Oracle Grid Infrastructure 11.2.0.1 Software Installation

We are ready to start Oracle Grid Infrastructure 11.2.0.1 software installation.

1. From node1 with grid user using the Oracle User Installer (OUI), navigate to the location where the software is copied. For example:
```
grid@node1>cd /home/grid/
./runInstaller
```

1. Select the "Install and Configure Grid Infrastructure for a Cluster" installation option.

1. Select "Advanced Installation" installation type.

1. Select the appropriate Product Languages for your environment.

1. Grid Plug and Play Information.
  
  Click to view.
  
  SCAN name and Cluster name will the same by default. After clicking next, the OUI will validate the SCAN and cluster information.

1. Cluster node information.
  
  Click to view.
  1. 1. Add all the nodes which are to be a part of the cluster with their chosen VIP addresses.
  1. 1. Provide grid username/password.
  1. 1. Setup the SSH connectivity, test it, and click Next.

1. Specify the network interface usage.
  
  Click to view.

1. Select "Automatic storage management" as the storage information option. ( Make the options in Bold as the previous instances.)

1. Create the ASM disk group.
  
  All the previously created disks/ASM marked will be listed here:
  1. Specify a disk name.
  2. Select Normal redundancy.
  3. Select the disks ex: CRS1,CRS2,CRS3.
    
    Note: These disks are for storing clusterware files. Normal redundancy requires a minimum of 3 disks.
  Click to view.

1. Specify an ASM password
  
  "Use same password for all the accounts" ( You can make this as a Note)

1. Select "Do not use Intelligent Platform Management Interface(IPMI) "for Failure isolation support. ( Make all the GUI selections in bold to be consistent across the document).

1. Configuring Privileged Operating Systems groups:
  
  If we have created the groups correctly, it will auto populate.
  
  Provide the following settings:
```
OSDBA for ASM  : asmdba
	OSOPER for ASM : asmoper
	OSASM: asmadmin
	Refer to Section Pre-Installation tasks: step 13
	
```

1. Specify the Installation Location.
  
  Set the "Oracle Base" ($GRID_BASE) and "Software Location" ($GRID_HOME) for the Oracle Grid Infrastructure installation:
  
  Oracle Base: /u01/app/grid
  
  Software Location: /u01/app/11.2.0/grid

1. Create Inventory ( Is this a task or a GUI element. If it is a part of the procedure, you can write as To create Inventory..)
  
  For the first install on the host, create the Oracle Inventory. Use the default values provided by the OUI:
  
  Inventory Directory: /u01/app/oraInventory
  
  oraInventory Group Name: oinstall

1. Prerequisite Checks
  
  The installer will run through a series of checks to determine if both Oracle RAC nodes meet the minimum requirements for installing and configuring the Oracle Clusterware and Automatic Storage Management software.
  
  Starting with Oracle Clusterware 11g Release 2 (11.2), if any check fails, the installer (OUI) will create shell script programs called fixup scripts to resolve many incomplete system configuration requirements. If OUI detects an incomplete task that is marked "fixable", then you can easily fix the issue by generating the fixup script by clicking the [Fix & Check Again] button.
  
  The fixup script is generated during installation. You will be prompted to run the script as root in a separate terminal session. When you run the script, it raises kernel values to required inimums, if necessary, and completes other operating system configuration tasks.
  
  If all prerequisite checks pass (as was the case for my install), the OUI continues to the Summary screen.
  
  Note: If there are any missing kernel parameters, install them and re-run the installer.
  
  Refer to Section Pre-Installation tasks: step 19 for details

1. Summary: Click Finish to start the installation. Installer eprforms configuration on all nodes-first on primary (local)node, then on all other secondary nodes.

1. Execute Configuration scripts
  
  After the installation completes, you will be prompted to run the /u01/app/oraInventory/orainstRoot.sh and /u01/app/11.2.0/grid/root.sh scripts. Open a new console window on both Oracle RAC nodes in the cluster, (starting with the node you are performing the install from), as the root user account and run the scripts. After the scripts run successfully, press ok.
  
  Go back to OUI and acknowledge the "Execute Configuration scripts" dialog window.

1. The final step performed by OUI is to run the Cluster Verification Utility (CVU).

1. Finish: At the end of the installation, click the [Close] button to exit the OUI.

Post Installation Checks After Grid Installation

Verify the cluster services on all nodes by running the following command:

grid>cd /u01/app/11.2.0/grid/bin
./crsctl check cluster –all
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Check Oracle TNS listener process on all nodes as follows/by running: (Same comment for all similar sections).

grid@node1> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_SCAN2
LISTENER_SCAN3
LISTENER
grid@node2> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_SCAN1
LISTENER

Confirming oracle ASM status on both nodes

grid> cd /u01/app/11.2.0/grid/bin
./srvctl status asm -a

Check Oracle cluster registry (OCR)

grid@node1>/u01/app/11.2.0/grid/bin/ocrcheck

Check voting disk

grid@node1>cd /u01/app/11.2.0/grid/bin
./crsctl query css votedisk

1. grid@node1>cd /u01/app/11.2.0/grid/bin

1. Create a backup of the ./root.sh script from /u01/app/11.2.0/grid/bin folder.

ASM Disks Creation

We have to create ASM disks for RAC DB and Fast Recovery disk since grid installation only creates ASM disks for CRS files. ( In places where "we have to" is mentioned, change to "you need to", or just "Create ASM disks".)

1. grid@node1>cd /u01/app/11.2.0/grid/bin
  
  ./asmca
  
  Click to view.

1. Select 'Create', choose 'external redundancy', specify an appropriate disk name(FRA or RACDB), choose the listed disks and ok. Need to create it twice separately.
  
  Exit
  
  Note: Please note that the disks will be listed in this console only if they are asm marked.
  Refer to Section-Pre-installation tasks: step 17

Verify the asm disks by doing the following:

grid>cd /u01/app/11.2.0/grid/bin
./asmcmd
ls (lists all the created asm disks)

Oracle RAC 11.2.0.1 software installation

We are ready to perform Oracle RAC database installation on all the nodes.

1. From node1 with 'Oracle' user using OUI, go to the location where the software was copied.
  
  oracle@node1>cd /home/oracle
  
  ./runInstaller

1. Configure Security updates. Provide e-mail address, if required.

1. Select the Select database software only installation option.

1. Select the following grid options:
  
  Click to view.
  1. 1. All the list nodes for RAC database,
  1. 1. Provide oracle user/password.
  1. 1. Setup SSH connectivity, test SSH connectivity, and click next.
  1. 1. For Product languages, make the appropriate selection(s) for your environment.

1. Select "Enterprise Edition" as the database edition.

1. For the Installation location, specify the Oracle base and Software location (Oracle home) as follows:
  
  Oracle Base: /u01/app/oracle
  
  Software Location: /u01/app/oracle/product/11.2.0/dbhome_1

1. For the Operating system groups, select the OS groups to be used for the SYSDBA and SYSOPER privileges as follows
  
  Database Administrator (OSDBA) Group: dba
  
  Database Operator (OSOPER) Group: oper

1. Perform a Prerequisites check.
  
  The installer will run through a series of checks to determine if both Oracle RAC nodes meet the minimum requirements for installing and configuring the Oracle Database software.
  
  Starting with 11g Release 2 (11.2), if any checks fail, the installer (OUI) will create shell script programs called fixup scripts to resolve many incomplete system configuration requirements. If OUI detects an incomplete task that is marked "fixable", then you can easily fix the issue by generating the fixup script by clicking the [Fix & Check Again] button.
  
  The fixup script is generated during installation. You will be prompted to run the script as root in a separate terminal session. When you run the script, it raises kernel values to required minimums, if necessary, and completes other operating system configuration tasks.
  
  If all prerequisite checks pass (as was the case for my install), the OUI continues to the Summary screen.
2. Click Finish to start the installation. The installer performs the Oracle Database software installation process on both Oracle RAC nodes.
3. Execute configuration scripts as follows:
  
  After the installation completes, you will be prompted to run the /u01/app/oracle/product/11.2.0/dbhome_1/root.sh script on both Oracle RAC nodes. Open a new console window on both Oracle RAC nodes in the cluster, (starting with the node you are performing the install from), as the root user account.
  
  Go back to OUI and acknowledge the "Execute Configuration scripts" dialog window.
4. Finish: At the end of the installation, click the [Close] button to exit the OUI.

RAC Database Creation Through DBCA

The database creation process should only be performed from local node (node1).

1. Run the following commands
  
  oracle@node1> dbca & or
  
  oracle@node1>cd /u01/app/oracle/product/11.2.0/dbhome_1/bin
  
  ./dbca

1. Welcome Screen. Select "Oracle Real Application Clusters database".

1. Operations: Select "Create a Database"

1. Database Templates: "Custom Database"

1. Database Identification:
  
  Click to view.
  1. 1. Select "admin managed" database
  1. 1. Specify Global database name which will be the service name. It's recommended to specify full domain name.
  1. 1. Select all nodes.
  1. 1. Specify SID prefix. This is the name given to DB instance which will get appended by number of nodes. This would have already been set by us as an environment variable in .bash_profile. This is a must for database connectivity.
```
Example: 
SID of node1 = orcl1
```

1. Management options: Leave the default options here, which is to "Configure Enterprise Manager/Configure Database Control for local management".

1. Database credentials: "Use the Same Administrative Password for All Accounts". Enter the password.

Click to view.

1. Database file options:
  
  Specify storage type and locations for database files.

1. Specify "ASMSNMP Password" :Specify the ASMSNMP password for the ASM instance.

1. Recovery configuration:
  
  Check the option for Specify Fast Recovery Area.
  
  For the Fast Recovery Area, click the [Browse] button and select the disk group name FRA.

1. Database Content : Default options

1. Initialization Parameters : Default options

1. Database Storage : Default options

1. Creation options: Select "create database" and " Generate database creation scripts".Click Finish to start the database creation process. After acknowledging the database creation report and script generation dialog, the database creation will start.
  
  Click OK on the "Summary" screen.

1. At the end of the database creation, exit from the DBCA.
  
  When the DBCA has completed, you will have a fully functional Oracle RAC 11g Release 2 cluster running!

Post Installation Checks After Oracle RAC Installation

1. Log in Oracle Enterprise Manager(Database control) using DBSNMP user and explore.
  
  https://node1.labs.blr.novell.com:1158/em
  
  Click to view.
  
  We can see database and cluster status, create users, check the performance etc.

Re-compile invalid objects:

oracle@node1> sqlplus / as sysdba
sql>@?/rdbms/admin/utlrp.sql

Enabling archive log in RAC environment:

If the database is in "Archive Log Mode", Oracle will make a copy of the online redo log before it gets reused. A thread must contain at least two online redologs (or online redolog groups).

Disable the cluster instance

oracle@node1>sqlplus / as sysdba
SQL> alter system set cluster_database=false scope=spfile sid='orcl1';
System altered.

1. Shutdown all instances accessing the cluster database as the oracle user:
```
oracle@node1> srvctl stop database -d orcl
```

Using the local instance, mount the database:

oracle@node1>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Sat Nov 21 19:26:47 2009
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup mount
ORACLE instance started.
Total System Global Area 1653518336 bytes
Fixed Size                  2213896 bytes
Variable Size            1073743864 bytes
Database Buffers          570425344 bytes
Redo Buffers                7135232 bytes

Enable archiving:

SQL> alter database archivelog;
Database altered.

1. Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:
```
SQL> alter system set cluster_database=true scope=spfile 	sid='orcl1';
System altered.
```

Shutdown the local instance:

SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

1. Bring all instances back up as the oracle account using srvctl
```
oracle@node1> srvctl start database -d orcl
```

[oracle@node1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Mon Nov 8 20:07:48 2010
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     68
Next log sequence to archive   69
Current log sequence           6

Verify database status by running the following command:

oracle@node1>srvctl status database -d orcl
Instance orcl1 is running on node node1
Instance orcl2 is running on node node2

To see the configuration of database

oracle@node1>srvctl config database -d racdb –a
Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile:  RACDB/orcl/spfileorcl.ora
Domain: labs.blr.novell.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: orcl
Database instances: racdb1,racdb2
Disk Groups: RACDB,FRA,CRS
Mount point paths:
Services: orcl.labs.blr.novell.com
Type: RAC
Database is enabled
Database is administrator managed

1. To see the asm status:
```
oracle@node1> srvctl status asm
```

1. TNS listener status:
```
srvctl status listener
```

1. SCAN status:
```
oracle@node1> srvctl status scan
```

To start/stop cluster:

root>cd /u01/app/11.2.0/grid/bin
./crsctl stop cluster
./crsctl start cluster

To start/stop database

oracle> srvctl stop database -d databasename
oracle> srvctl start database -d databasename

To start only local instance or only one instance

oracle>sqlplus/ as sysdba
sql> startup mount;
sql> shutdown immediate;

Troubleshooting

1. Log files can be found in the following locations:
  1. 1. Event manager log files: $GRID_HOME/log/hostname/evmd
  1. 1. Database log files: $GRID_HOME/log/hostname/dbname
      :$ORACLE_HOME/log/hostname/dbname
  1. 1. Cluster ready services log files: $GRID_HOME/log/hostname/dbname
  1. 1. Oracle cluster registry(OCR) file logs: $GRID_HOME/log/hostname/client
  1. 1. CRS alert logs: $ORA_GRID_HOME/log/<hostname>/alert<nodename>.log
2. Error: ora-01219 database not open queries allowed on fixed tables views only
  
  This error is seen sometimes, after shutdown/startup local instance and when tried viewing tables.
  
  Solution: from localnode
```
sqlplus /as SYSBDA
SQL> select status from v$instance;
STATUS
--------
OPEN
--------
MOUNTED
```
  We still need to manually open database:
```
SQL> alter database open;
```

Error ORA-00257: archiver error. Connect internal only, until freed

This is because of less space due to archive redo logs.

Solution: from localnode

SQL> SELECT * FROM V$RECOVERY_FILE_DEST;
if used limit is equal to space_limit, then  delete the Archive Log , when it is not needed to free up some space
rman target /
RMAN>delete archivelog until time 'SYSDATE-1';

1. ORA-00603: ORACLE server session terminated by fatal error or ORA-29702: error occurred in Cluster Group Service operation
  
  If the RAC node name is listed for the loopback address, you will receive the following error during the RAC installation:" 127.0.0.1 node1 localhost.localdomain localhost"
  
  Solution: Remove the database name from loopback address.

1. ORA-01078: failure in processing system parameters ,LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/initorcl_1.ora'
  
  Though database is up and running,while trying to connect to database, it connects to an idle instance. Starting local instance gives the above error.
  
  Solution: this error is seen when ORACLE_SID is not set properly.
  
  For localnode: Export ORACLE_SID=dbname1(orcl1)

ZCM (ZENworks Configuration Management) Installation With Oracle RAC

1. Create a VM image with Windows or Linux.

1. Download ZCM 11.2 build and trigger the installation.

1. Complete the initial steps.

1. In the Database selection page, select Oracle as database for ZCM.

1. We can select new schema or existing schema. For existing schema, we should create a schema beforehand.

1. Specify the service name, hostname and port of the database.
  
  Click to view.
  
  Check the tnsname.ora file under $ORACLE_HOME/network/admin and verify the hostname and service name.
```
Example:
ORCL =
 (DESCRIPTION =
 (ADDRESS = (PROTOCOL = TCP)(HOST = blr-srm-cluster)(PORT = 1521))
    			(CONNECT_DATA =
     				 (SERVER = DEDICATED)
      				(SERVICE_NAME = orcl.labs.blr.novell.com)
    			)
		  )
		  
```
  Note: Hostname should be the Scan name and not the individual hostname as all the nodes would be registered under SCAN as a cluster. During a node failure, SCAN immediately delegates all the client connections and transactions to another available node (registered under cluster) thus ensuring zero downtime.

1. Provide administrative Oracle user and ZENworks user and proceed with successful ZENworks installation. (User credentials Specify)

1. Test by shutting down the local node.
```
Sql> shutdown immediate	
```

1. ZENworks continues to function properly without any database downtime.

Uninstallation Procedure

1. Uninstall ZENworks.

1. Delete database through dbca application.
  
  oracle@node1> dbca &

1. De-install the Oracle RAC database.
  
  oracle@node1> cd $ORACLE_HOME/deinstall/
  
  ./deinstall
  
  Specify all the required information.

1. De-install Oracle Grid software installation
  
  grid@node1> /u01/app/11.2.0/grid/deinstall
  
  ./deinstall

1. Delete ASM marked shared disks.
  
  root> /usr/sbin/oracleasm deletedisk CRS1(all the disks)

1. Check for oracle processes using ps -ef. Stop the processes, if any, and the machine is as good as a fresh one.

Links and Downloads

1. Oracle VM server 2.2.2
  
  http://www.oracle.com/virtualization

1. Oracle VM manager 2.2.0
  
  http://www.oracle.com/virtualization

1. Oracle Linux-Redhat5
  
  http://www.oracle.com/linux

1. Oracle Grid Infrastructure software 11.2.0.1.0
  
  http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

1. Oracle RAC Database 1.2.0.1.0
  
  http://www.oracle.com/technetwork/database/enterprise- edition/downloads/112010-linuxsoft-085393.html

1. Kernel Rpms:
  
  http://public-yum.oracle.com

IMPORTANT NOTE:

Please note that if ZENworks Reporting Server(ZRS) has been configured in the setup, Oracle RAC would not work.

oracle_rac_for_zcm.pdf

Tags:

Oracle RAC

clustering

Labels:

Collateral

How To-Best Practice

Comment List

MigrationDeletedUser over 11 years ago in reply to MigrationDeletedUser$emplogoPC

Hi Peterhine,
During ZCM installation with Oracle database, we need to specify Oracle server credentials like server ip or DNS name, Port and Service name(SID) based on which a JDBC connection (using JDBC driver) call is made to Oracle server for validation. Once successful, the information will be stored in the database in encrpyted format for subsequent transactions.

For more information, refer: www.novell.com/.../

Hope it helps!
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel
MigrationDeletedUser over 11 years ago

Where have you put "export ORACLE_HOME" so it can be seen by the zcm services on the zcm server ?
or how do the ZCM services know where oracle client is installed ?

p
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel
MigrationDeletedUser over 12 years ago

Hello,
As mentioned in the document, Zenworks Reporting Serve(ZRS)r doesn't work with Oracle RAC by default.

Please follow the below steps to make ZRS work
Steps:
1. Install ZRS on top of ZCM
2. Create an empty 'boe_jdbc_url.txt' under
/etc/opt/novell/zenworks/datamodel/ if ZRS is running on linux OS.
or
ZENWORKS_HOME/conf/datamodel/ if ZRS is running on Windows OS.
3. Add JDBC URL for RAC server in file.
Below is eg url for the same:
jdbc:oracle:thin:@(DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=Ipaddress1)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=IpAddress2)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=OracleServiceName)))
Note: DbserverIP, dbname, dbinstance should be changed if the database is moved.
4. Run the command: novell-zenworks-configure -c UpdateBOE .
5. Launch ZRS.
6. Run or create reports successfully in an Oracle RAC environment

Also refer to www.novell.com/.../doc.php for more information

Best Regards,
Megha
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel

Recommended

Menu

ZENworks Tips & Information

Oracle RAC 11.2.0.1 - 2 Node Cluster Solution for ZCM

Introduction

Resource Requirements

Installation Plan

Oracle Grid Infrastructure 11.2.0.1 Software Installation

Post Installation Checks After Grid Installation

ASM Disks Creation

Oracle RAC 11.2.0.1 software installation

RAC Database Creation Through DBCA

Post Installation Checks After Oracle RAC Installation

Troubleshooting

ZCM (ZENworks Configuration Management) Installation With Oracle RAC

Uninstallation Procedure

Links and Downloads

IMPORTANT NOTE:

Tags:

Labels:

Resources