Modified:Feb 7, 2012Type:HOWTOStatus:PUBLISHEDPriority:3
Comments (0)

In this Document
Goal
Solution
References

Applies to:

Oracle Server – Enterprise Edition – Version: 10.2.0.1 to 11.2.0.1.0 – Release: 10.2 to 11.2
Information in this document applies to any platform.
Goal

Sometimes it can happen that the ASM contains empty system directories which cannot be removed. According to Note 3751057.8(ALTER DISKGROUP DROP DIRECTORY FORCE removes system aliases) the system generated directories should be removed automatically as soon as all it’s files are removed. However this does not happen sometimes and a manual directory removal will fail :

SQL> set pages 2000
SQL> SELECT level, dir, sys, substr(lpad(‘ ‘,2*level,’ ‘)||CONCAT(‘+’||gname,
SYS_CONNECT_BY_PATH(aname,’/')),1,60) full_path
FROM ( SELECT g.name gname, a.parent_index pindex, a.name aname,
a.reference_index rindex, a.ALIAS_DIRECTORY dir, a.SYSTEM_CREATED sys
FROM v$asm_alias a, v$asm_diskgroup g
WHERE a.group_number = g.group_number)
START WITH (MOD(pindex, POWER(2, 24))) = 0
CONNECT BY PRIOR rindex = pindex
ORDER BY rtrim(ltrim(full_path))desc, level asc;

3 N Y +DATA/UDWH/DATAFILE/ADASTRA.376.624105133

2 Y Y +DATA/UDWH/DATAFILE

3 N Y +DATA/UDWH/CONTROLFILE/Current.486.624103637

3 N Y +DATA/UDWH/CONTROLFILE/Current.479.624103637

3 N Y +DATA/UDWH/CONTROLFILE/Current.434.624103637

2 Y Y +DATA/UDWH/CONTROLFILE

1 Y Y +DATA/UDWH

1 Y Y +DATA/TDWH

2 N N +DATA/IDWH/spfileIDWH.ora

2 N N +DATA/IDWH/control03.ctl

2 N N +DATA/IDWH/control02.ctl

2 N N +DATA/IDWH/control01.ctl

The TDWH directory from the above list is empty but cannot be removed :

ASMCMD> cd tdwh
ASMCMD> ls
ASMCMD> cd ..
ASMCMD> rm -rf tdwh
ORA-15032: not all alterations performed
ORA-15177: cannot operate on system aliases (DBD ERROR: OCIStmtExecute)

alter diskgroup DATA drop directory ‘+DATA/TDWH’ FORCE;
ORA-15032: not all alterations performed
ORA-15177: cannot operate on system aliases (DBD ERROR: OCIStmtExecute)

A solution to remove these directories is to recreate the ASM diskgroups but this is not something one would do very often especially when the ASM is used by production databases.

Solution

Either:
Create a new database with the DBCA having the same name as the old directory and subsequently drop the database with the DBCA.

Or:
Create a dummy tablespace having a datafile within the directory to be dropped, drop the tablespace and then remove the datafile using the ASMCMD tool. The old directory will be removed automatically.

There is a scenario when there are left behind directories after the database is dropped using RMAN’s command “drop database including backups [noprompt];”. When one falls into this there is no way out but to recreate the diskgroups. This problem is being investigated in Bug 6606689. To avoid this issue one should run the following two commands instead of “DROP DATABASE INCLUDING BACKUPS;”:

RMAN> delete backupset;

RMAN> drop database;

References

BUG:6110998 – UNABLE TO DELETE SYSTEM CREATED DIRECTORY
NOTE:3751057.8 – Bug 3751057 – ALTER DISKGROUP DROP DIRECTORY FORCE removes system aliases
BUG:6606689 – RMAN LEAVES BEHIND DIRECTORIES IN ASM THAT CANNOT BE REMOVED

Modified:Aug 22, 2012Type:PROBLEMStatus:PUBLISHEDPriority:3
Comments (0)

In this Document
Symptoms
Cause
Solution
Applies to:

Oracle Server – Enterprise Edition – Version 10.2.0.3 and later
Information in this document applies to any platform.
***Checked for relevance on 27-Feb-2011***

Symptoms

SQL> alter diskgroup TEMP_GROUP drop alias
‘+TEMP_GROUP/PDWH/DUMPSET/SYSTEMSYS_EXPORT_SCHEMA_01_1309498_1.259.6324066′;

alter diskgroup TEMP_GROUP drop alias
‘+TEMP_GROUP/PDWH/DUMPSET/SYSTEMSYS_EXPORT_SCHEMA_01_1309498_1.259.6324066′
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15177: cannot operate on system aliases

Cause

When the file was dropped from ASM/sqlplus, then it should also remove the System ASM file.

During the file drop through asmcmd , it is necessary to provide the untruncated file name.Since the NAME column in the fixed views like v$asm_alias is VARCHAR2(48), the fix would be to get the untruncated name and then drop the file explicitly.

No queries currently available to get the Actual file name and this is because of the way rows are present in V$ASM_ALIAS . This has to be achieved manually , as below .

Solution

Alternatively, use the below ” +diskgroup.fnum.incarn ” to drop the file like:

SQL > ALTER DISKGROUP DROP FILE ”;

example :

SQL > ALTER DISKGROUP TEMP_GROUP DROP FILE ‘+TEMP_GROUP.259.632406635′;

The “+diskgroup.fnum.incarn ” information , can be retrieved from the below script output , ASM_FILENAME ::

SQL> select f.file_number, a.name , (‘+’ || g.name||’.'||f.file_number||’.'||f.incarnation)
asm_filename,
2 f.type from v$asm_diskgroup g,
3 v$asm_file f,
4 v$asm_alias a
5 where g.name = ”
6 and g.group_number = f.group_number
7 and g.group_number = a.group_number
8 and f.file_number = a.file_number
9 order by f.file_number;

Oracle highly recommends deploying EM in a HA environment. The best practices for HA deployments, backup and housekeeping of your Enterprise Manager environment are documented in the Oracle Enterprise Manager Advanced Configuration Guide. It is imperative that there is a good disaster recovery plan in place for your EM deployment. In this post I want to talk about a customer who failed to do the correct planning and housekeeping for EM and landed in a situation where we the all the OMSes were nearly blown away had we not jumped to help.

We recently hit an issue at a customer site where we had a two node OMS setup of the Enterprise Manager and a RAC Database being used as the EM repository. An accidental delete of the OMS oracle home left us with a single node deployment. While we were trying to figure out a possible path to recover the first node, the second node was rebooted under a maintenance window. What followed was a complete site outage as the Admin and managed servers would not start on either of the nodes.

In my situation there were

- No backups of the Oracle Homes from any node

- No OMS Configuration snapshots (created using the “emctl exportconfig oms” command) and the instance home was completely lost on node 1 which also had the Admin Server

We did however have:

- A copy of the emkey.ora that I found under the OMS_ORACLE_HOME/ of the second node (NOTE: it is a bad practice to have your emkey present under the OMS Oracle home directory on the same server as the OMS. The backup of the emkey should be maintained on some other server. In this case however it was a savior in my situation since there were no backups

- The oms oracle home on the second node but missing a number of files and had a number of changes done to the files in the home. There were a number of attempts to start the server by modifying various files based on the Weblogic server logs to have atleast node up and running but all of them failed.

Here is how you can recover from this scenario:

Follow these steps:

STEP 1: Check status of emkey.ora

Check whether the emkey exists is present in the EM repository or not. Run the following command:

$OMS_ORACLE_HOME/bin/emctl status emkey

If the output is something like this below then you are good to go and the key is present in the repository

./emctl status emkey

Oracle Enterprise Manager 11g Release 1 Grid Control

Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey is configured properly.

Here are the messages that you might see as the emctl status emkey output depending upon whether the EM Admin Server is up and if the key is configured properly:

Case1: AdminServer is up, emkey is proper in CredStore & not in repos. This is same as the output of the command shown above:

The EMKey is configured properly

Case 2: AdminServer is up, emkey is proper in CredStore & exists in repos:

The EMKey is configured properly, but is not secure.
Secure the EMKey by running “emctl config emkey -remove_from_repos”.

Case 3: AdminServer is down or emkey is corrupted in CredStore) & (emkey exists in repos):

The EMKey exists in the Management Repository, but is not configured properly or is corrupted in the credential store.
Configure the EMKey by running “emctl config emkey -copy_to_credstore”.

Case 4: (AdminServer is down or emkey is corrupted in CredStore) & (emkey does not exist in repos):

The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository. To correct the problem:
1) Get the backed up emkey.ora file.
2) Configure the emkey by running “emctl config emkey -copy_to_credstore_from_file”.

If not the key was not secured properly, we will have to be put in the repository before proceeding. Look at the next step 2 for doing this

There may be cases (like mine) where running emctl may give errors like the following:

$OMS_ORACLE_HOME/bin/emctl status emkey

Exception in thread “Main Thread” java.lang.NoClassDefFoundError: oracle/security/pki/OracleWallet

At oracle.sysman.emctl.config.oms.EMKeyCmds.main (EMKeyCmds.java:658)

Just move to the next step to put the key back in the repository

STEP 2: Put emkey.ora back in the repository

Skip this step if your emkey.ora is present in the repository. If not, you need to put the key back in the repository See if you can run the following command (with sample output):

$OMS_ORACLE_HOME/bin/emctl config emkey –copy_to_repos

Oracle Enterprise Manager 11g Release 1 Grid Control

Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.

The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.

After the required operation has been completed, secure the EMKey by running

“emctl config emkey -remove_from_repos”.

Typically the key is present under $OMS_ORACLE_HOME/sysman/config directory before being removed after the install as a best practice. If you hit any errors while running emctl commands like the one mentioned in step 1, jump to step 3 and we will take care of the emkey.ora in Step 5

STEP 3: Get the port information

Check for the existing port information in the emd.properties file under EM_INSTANCE_DIRECTORY (typically gc_inst directory right above the Middleware home where you have deployed em. For eg. /u01/app/oracle/product/gc_inst in case your oms home is /u01/app/oracle/product/Middleware/oms11g)

In my case I got the information from the emgc.properties present in the gc_inst on the second node. If you can run emctl you may want to try the following command as well

$OMS_ORACLE_HOME/bin/emctl status oms –details

Note this information as this will be used in the next step

STEP 4: Perform cleanup on Node 1

Note the oracle home of the Weblogic and OMS, get the list of applied patches in the homes (using opatch lsinventory command), take a backup copy of the home just in case we need it and then de-install/remove oracle homes, update inventory and cleanup processes on the first node

STEP 5: Perform Software Only Installation of OMS on Node 1

Perform Weblogic 10.3.2 installation exactly under the same location as present in the earlier installation. Perform software only installation of the OMS using the following command. This will not run any configuration assistants and bypass all user interface validations

runInstaller –noconfig -validationaswarnings

Select the “Additional OMS” option while performing the installation. Provide the same path for OMS and Instance directories like the previous installation

Use the port information collected in Step 3 while performing the installation. Once the installation is complete run the allroot.sh script to complete the binary deployment

STEP 6: Apply one-off patches

At this point you can apply any patches to the OMS Oracle Home previously. You only need to run opatch to install the patch in the home and not required to run the SQLs

STEP 7: Copy EM key

This step is only required if you were not able to use emctl command to put the emkey back into the EM repository in STEP 2

Copy the emkey.ora file of the old installation you have under $OMS_ORACLE_HOME/sysman/config directory of the newly installed OMS

STEP 8: Configure Grid Control Domain

Run the following command to configure the EM domain and OMS. Note that you need to use a different GC Domain name than what you used earlier. For example I have used GCDOMAIN11 as the new domain name when my previous domain name was GCDOMAIN

$OMS_ORACLE_HOME/bin/omsca new –AS_USERNAME weblogic –EM_DOMAIN_NAME GCDOMAIN11 –NM_USER nodemanager -nostart

This command as shown below will prompt for a number of inputs like Admin Server hostname, port, password, etc. Verify if the defaults shown are correct by pressing enter or provide a new value

STEP 9: Run Add-ON Configuration Assistant

After this step run the following add-on configuration assistant. This was used in my case to configure the virtualization add-on

$OMS_ORACLE_HOME/addonca -oui -omsonly -name vt -install gc

STEP 10: Start the OMS

Now start the OMS using

$OMS_ORACLE_HOME/bin/emctl start oms

In a multi-node setup like mine you would either have a software load balancer or DNS round robin (using a virtual host name that resolves to one of multiple OMS hostnames) being used for load balancing. Secure the OMS against the SLB or DNS virtual hostname using the following

$ OMS_HOME/bin/emctl secure oms -host slb.example.com -secure_port 1159 -slb_port 1159 -slb_console_port 443

STEP 11: Configure the Agent

From the $AGENT_ORACLE_HOME/bin run the ./agentca –f

At this point you should have your OMS on node 1 fully re-covered. Clean up node 2 and use the normal Additional OMS installation process documented in the official installation guide to add the additional OMS on node 2

Summary

It took us nearly a little over two days to completely recover the environment with some other non-EM related issues that hit us along the way as well. In the end a situation like this could have been completely avoided had the proper housekeeping and backup of the Enterprise Manager Deployment been done in the first place. This is going to a topic that we cover in the next post. In the meantime please do refer to the Oracle Enterprise Manager Advanced Configuration Guide for planning your EM installation, backup and housekeeping procedures. This can be found here:

http://download.oracle.com/docs/cd/E11857_01/index.htm

Thanks

This post would not have been possible without Raj Aggarwal, Prasad Chebrolu and Ravikumar Basa who helped to recover the environment and provided all the support we needed

This note describes the process of cloning an Oracle Home – in this case we are cloning a 11g Release 2 Oracle Home from source machine devastator to target machine vixen.

On source (devastator)

ORACLE_HOME location is : /u01/app/oracle/product/11.2.0/dbhome_1

Go to directory on source machine one level higher and tar the db_home1 directory and all sub-directories

devastator:$ pwd

devastator:$ /u01/app/oracle/product/11.2.0

devastator:$ tar -cvf 11gr2.tar dbhome_1

On Target (vixen)

Create the directory /u01/app/oracle/product/11.2.0 if it does not exist

On Source

scp or ftp the tar file to this location on the target machine

devastator:$ scp -rp 11gr2.tar oracle@vixen:/u01/app/oracle/product/11.2.0

On target

Uncompress the tar file

vixen:$ cd /u01/app/oracle/product/11.2.0

vixen:$ tar -xvf 11gr2.tar

Clone the Oracle Home using the perl script clone.pl which is located under the $ORACLE_HOME/clone/bin

vixen:$ cd /u01/app/oracle/product/11.2.0/dbhome_1/clone/bin

vixen:$ perl clone.pl ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 ORACLE_HOME_NAME=11GR2_HOME

./runInstaller -clone -waitForCompletion “ORACLE_BASE=/u01/app/oracle” “ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1″ “ORACLE_HOME_NAME=11GR2_HOME” -silent -noConfig -nowait
Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 13044 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-06-28_12-19-47PM. Please wait …Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.

You can find the log of this install session at:
/u01/app/oracle/oraInventory/logs/cloneActions2010-06-28_12-19-47PM.log
………………………………………………………………………………………. 100% Done.

Installation in progress (Monday, June 28, 2010 12:20:29 PM WST)
…………………………………………………………………… 78% Done.
Install successful

Linking in progress (Monday, June 28, 2010 12:21:06 PM WST)
Link successful

Setup in progress (Monday, June 28, 2010 12:23:34 PM WST)
Setup successful

End of install phases.(Monday, June 28, 2010 12:34:05 PM WST)
Starting to execute configuration assistants
The following configuration assistants have not been run. This can happen because Oracle Universal Installer was invoked with the -noConfig option.
————————————–
The “/u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/configToolFailedCommands” script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.
The “/u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/configToolAllCommands” script contains all commands to be executed by the configuration assistants. This file may be used to run the configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.

————————————–
WARNING:
The following configuration scripts need to be executed as the “root” user.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh
To execute the configuration scripts:
1. Open a terminal window
2. Log in as “root”
3. Run the scripts

The cloning of 11GR2_HOME was successful.
Please check ‘/u01/app/oracle/oraInventory/logs/cloneActions2010-06-28_12-19-47PM.log’ for more details.

Run the root.sh script as root

# pwd
/u01/app/oracle/product/11.2.0/dbhome_1

# ./root.sh
Check /u01/app/oracle/product/11.2.0/dbhome_1/install/root_devastator_2010-06-29_07-42-31.log for the output of root script

vixen:$cat /u01/app/oracle/product/11.2.0/dbhome_1/install/root_vixen_2010-06-29_07-42-31.log

Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

We can now check that the OraInventory has been updated with the 11g Release 2 Oracle Home details.

On this machine, earlier there was only a 10g Oracle Home installed. When we check the contents of the inventory.xml file, we will see that it has information about the 11g Oracle Home – 11GR2_HOME

vixen:$ cat /var/opt/oracle/oraInst.loc
inventory_loc=/u01/app/oracle/oraInventory

We can identified inventory location by myself using INVENTORY_LOCATION parameter

Like:

perl clone.pl ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=Oracle_home_name ‘-O”sl_tableList={new_node:new_node-priv:new_node-vip}”‘ ‘-O-noConfig”-O”INVENTORY_LOCATION=central_inventory_location”‘

(note:this is on Rac env)

inst_group=dba

vixen:$cd /u01/app/oracle/oraInventory

vixen:$ ls
backup ContentsXML logs oraInstaller.properties oui
Contents install.platform oraInst.loc orainstRoot.sh sessionContext.se

vixen:$cd ContentsXML

vixen:$ ls
comps.xml inventory.xml libs.xml

vixen:$ vi inventory.xml

11.2.0.1.0
2.1.0.6.0