Facts
I created a DB on ASM. Also, I use the ACFS (ASM Cluster File System) in order to store archived log and others logs like the alert.log
This DB was created on a clone environment. The day after I created the DB on production. So I had two independent installations and configurations but, one is the clone of the other.
Usually, the correct steps are:
- Install and configure the DB in production
- Clone the DB
- Start the clone
In this case instead:
- Created a DB on the clone
- Created a DB in production
- Synchronized the DB clone starting from DB in production
Following picture try to explain what I mean
Symptoms
Because of this order, the Logical Volume on prod and clone are different.
What the CRS knows is:
[clone01.+ASM1]/sbRdbms/oracle-$acfsutil registry –l |grep -i llvm
Device : /dev/asm/lvllvmpak-486 : Mount Point : /dbLLVMp/akLLVMp01 : Options : none : Nodes : all : Disk Group : DGLLVMPFS : Volume : LVLLVMPAK
Device : /dev/asm/lvllvmpfs-486 : Mount Point : /dbLLVMp/fsLLVMp01 : Options : none : Nodes : all : Disk Group : DGLLVMPFS : Volume : LVLLVMPFS
On the DGLLVMPFS Disk Group, there two Logical Volume: LBLLVMPAK for the archived log with /dev/asm/lvllvmpak-486 device and LBLLVMPFS for the log files with /dev/asm/lvllvmpfs-486 device.
But because the DG are imported from the production, they had different pieces of information
[clone01.+ASM1]/sbRdbms/oracle-$asmcmd volinfo -G DGLLVMPFS -a
Diskgroup Name: DGLLVMPFS
Volume Name: LVLLVMPAK
Volume Device: /dev/asm/lvllvmpak-34
State: DISABLED
Size (MB): 102400
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage: ACFS
Mountpath: /dbLLVMp/akLLVMp01
Volume Name: LVLLVMPFS
Volume Device: /dev/asm/lvllvmpfs-34
State: DISABLED
Size (MB): 51008
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage: ACFS
Mountpath: /dbLLVMp/fsLLVMp01
That is, there is always a DGLLVMPFS Disk Group with the LBLLVMPAK and LBLLVMPFS Logical Volume, but in this case, the former device is /dev/asm/lvllvmpak-34 and the latter is /dev/asm/lvllvmpfs-34.
The status of the resources are
[clone01.+ASM1]/sbRdbms/oracle-$cstat |grep -i llvm
===================================================================================================
HA Resource Resource Type Target State
===================================================================================================
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type ONLINE UNKNOWN on clone01
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type ONLINE UNKNOWN on clone02
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type ONLINE UNKNOWN on clone01
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type ONLINE UNKNOWN on clone02
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type OFFLINE OFFLINE
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.llvmp_batch.svc ora.service.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpfs.acfs ora.acfs.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpfs.acfs ora.acfs.type ONLINE OFFLINE
This outptu is reformatted using the script you can find here. (the script in the post is updated. The one I used here is an old version, so it will be a little different).
As you can see, the STATE of the resources is UNKNOW.
Solution
There is a Doc ID on MOS, that I followed in order to resolve the problem: "ODA: How to Remove an ADVM CRS Resource When It has not Any Associated ADVM Volume in /dev/asm/* (Doc ID 2174600.1) (PDF)". Also there are a couple of command I used from the "Automatic Storage Management Administrator's Guide"
So, the steps I made was:
- Disable the Logical Volume
- Deregister the file system
- Delete the Resource from crs
- Enable the Logical Volume
- Register the file system
1) Disable the Logical Volume
I run this step from [1]:
[clone01.+ASM1]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/asmcmd voldisable -G DGLLVMPAK -a
[clone01.+ASM1]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/asmcmd voldisable -G DGLLVMPFS -a
[clone02.+ASM2]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/asmcmd voldisable -G DGLLVMPAK -a
[clone02.+ASM2]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/asmcmd voldisable -G DGLLVMPFS -a
Please, pay attention. I run the commands on all nodes of the RAC.
2) Deregister the file system
This command come from [1]
[root@clone01 ~]#/sbin/acfsutil registry -d /dev/asm/lvllvmpfs-486
I run the command as root user. After this, the ADVM (ASM Dynamic Volume Manager) for LVLLVMFS, disappear. It only exists the ADVM resource for the AK Logical Volume.
[mdapora2d01-TP.+ASM1]/sbRdbms/oracle-$cstat |grep -i llvm
===================================================================================================
HA Resource Resource Type Target State
===================================================================================================
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type OFFLINE UNKNOWN on clone01
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type OFFLINE UNKNOWN on clone02
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type OFFLINE UNKNOWN on clone01
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type OFFLINE UNKNOWN on clone02
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type OFFLINE OFFLINE
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.llvmp_batch.svc ora.service.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type OFFLINE OFFLINE
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type OFFLINE OFFLINE
With following command, I remove/deregister the last resource.
[root@clone01 ~]#/sbin/acfsutil registry -d /dev/asm/lvllvmpak-486
3) Delete the Resource from crs
These commands come from [2]:
[clone01.+ASM1]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/crsctl delete resource ora.DGLLVMPFS.LVLLVMPFS.advm -f -unsupported
[clone01.+ASM1]/sbRdbms/oracle-$cstat |grep -i llvm
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type OFFLINE UNKNOWN on clone01
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type OFFLINE UNKNOWN on clone02
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type OFFLINE OFFLINE
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.llvmp_batch.svc ora.service.type OFFLINE OFFLINE
As you can see, because I removed the DGLLVMPFS Logical Volume, it disappeared. Only the DGLLVMPAK is still present. I remove it running following command
[clone01.+ASM1]/sbRdbms/oracle-$/sbOraGrid/12.1.0.2/grid/bin/crsctl delete resource ora.DGLLVMPFS.LVLLVMPFS.advm -f -unsupported
4) Enable the Logical Volume
Really this step should be the last, but Oracle permitted me so I enabled the Logical Volume before to register the ASCF. This is because an LV is just a container and it could be empty
This is the picture until now. The ADVM and ACFS resources are missing:
[mdapora2d01-TP.+ASM1]/sbRdbms/oracle-$cstat |grep -i llvm
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type OFFLINE OFFLINE
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.llvmp_batch.svc ora.service.type OFFLINE OFFLINE
[clone01.+ASM1]/sbRdbms/oracle-$asmcmd volenable -G DGLLVMPFS –a
[clone01.+ASM1]/sbRdbms/oracle-$asmcmd volenable -G DGLLVMPAK –a
[mdapora2d01-TP.+ASM1]/sbRdbms/oracle-$cstat |grep -i llvm
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type OFFLINE OFFLINE
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type OFFLINE OFFLINE
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type OFFLINE OFFLINE
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.db ora.database.type OFFLINE OFFLINE
ora.llvmp.llvmp_batch.svc ora.service.type OFFLINE OFFLINE
Here you can see that only the ADVM on the first node is ONLINE. They will be ONLINE on the second node after the following command:
[clone02.+ASM2]/sbRdbms/oracle-$asmcmd volenable -G DGLLVMPFS –a
[clone02.+ASM2]/sbRdbms/oracle-$asmcmd volenable -G DGLLVMPFS –a
5) Register the file system
There is just this last step. I run the acfsutil registry again
[root@clone01 ~]# /sbin/acfsutil registry -a /dev/asm/lvllvmpak-34 /dbLLVMp/akLLVMp01
acfsutil registry: mount point /dbLLVMp/akLLVMp01 successfully added to Oracle Registry
[root@clone01 ~]# /sbin/acfsutil registry -a /dev/asm/lvllvmpfs-34 /dbLLVMp/fsLLVMp01
acfsutil registry: mount point /dbLLVMp/fsLLVMp01 successfully added to Oracle Registry
Remember that you need the root privileges to run acfsutil. Otherwise, you have following error (yes I run it with oracle user, sigh)
[clone02.+ASM2]/sbRdbms/oracle-$/sbin/acfsutil registry -a /dev/asm/lvllvmpfs-34 /dbLLVMp/fsLLVMp01
PRCN-2018 : Current user oracle is not a privileged user
acfsutil registry: ACFS-03111: unable to add ACFS mount /dbLLVMp/fsLLVMp01 within Oracle Registry
At this point all resources are ONLINE
[mdapora2d02-TP.+ASM2]/sbRdbms/oracle-$cstat |grep -i llvm
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB01.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPDB02.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.LVLLVMPAK.advm ora.volume.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.LVLLVMPFS.advm ora.volume.type ONLINE ONLINE on clone02
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone01
ora.DGLLVMPFS.dg ora.diskgroup.type ONLINE ONLINE on clone02
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone01
ora.LISTENER_LLVMP.lsnr ora.listener.type ONLINE ONLINE on clone02
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type ONLINE ONLINE on clone01
ora.dgllvmpfs.lvllvmpak.acfs ora.acfs.type ONLINE ONLINE on clone02
ora.dgllvmpfs.lvllvmpfs.acfs ora.acfs.type ONLINE ONLINE on clone01
ora.dgllvmpfs.lvllvmpfs.acfs ora.acfs.type ONLINE ONLINE on clone02
ora.llvmp.db ora.database.type ONLINE ONLINE on clone01
ora.llvmp.db ora.database.type ONLINE ONLINE on clone02
ora.llvmp.llvmp_batch.svc ora.service.type ONLINE ONLINE on clone01
Acknowledgments
Special thanks to Roberto (my colleague) who supported me during this activity
References
[1] Automatic Storage Management Administrator's Guide
[2] ODA: How to Remove an ADVM CRS Resource When It has not Any Associated ADVM Volume in /dev/asm/* (Doc ID 2174600.1)
[3] Ch.11 Introducing Oracle ACFS and Oracle ADVM
[4] acfsutil registry command
[5] asmcmd volenable/disable
[6] crsctl command
Nessun commento:
Posta un commento