STEP BY STEP RMAN ACTIVE DUPLICATE Database From RAC TO RAC 19c
In this article we will be using ACTIVE DUPLICATION method to prepare Auxiliary Instance on RAC 19c.
1=> Environment. We are having below envionment. Currently only Source side is having completely ready database. Target Side only have Cluster_Home and Oracle_Home installed, NO DB installed yetOn Source Side [oracle@racsetn1 dbs]$ /grid_home/app/19c/grid/bin/crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE racsetn1 STABLE ONLINE ONLINE racsetn2 STABLE ora.chad ONLINE ONLINE racsetn1 STABLE ONLINE ONLINE racsetn2 STABLE ora.net1.network ONLINE ONLINE racsetn1 STABLE ONLINE ONLINE racsetn2 STABLE ora.ons ONLINE ONLINE racsetn1 STABLE ONLINE ONLINE racsetn2 STABLE ora.proxy_advm OFFLINE OFFLINE racsetn1 STABLE OFFLINE OFFLINE racsetn2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup) 1 ONLINE ONLINE racsetn1 STABLE 2 ONLINE ONLINE racsetn2 STABLE 3 ONLINE OFFLINE STABLE ora.DATA.dg(ora.asmgroup) 1 ONLINE ONLINE racsetn1 STABLE 2 ONLINE ONLINE racsetn2 STABLE 3 OFFLINE OFFLINE STABLE ora.FRA.dg(ora.asmgroup) 1 ONLINE ONLINE racsetn1 STABLE 2 ONLINE ONLINE racsetn2 STABLE 3 OFFLINE OFFLINE STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE racsetn1 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE racsetn2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE racsetn2 STABLE ora.OCR.dg(ora.asmgroup) 1 ONLINE ONLINE racsetn1 STABLE 2 ONLINE ONLINE racsetn2 STABLE 3 OFFLINE OFFLINE STABLE ora.asm(ora.asmgroup) 1 ONLINE ONLINE racsetn1 Started,STABLE 2 ONLINE ONLINE racsetn2 Started,STABLE 3 OFFLINE OFFLINE STABLE ora.asmnet1.asmnetwork(ora.asmgroup) 1 ONLINE ONLINE racsetn1 STABLE 2 ONLINE ONLINE racsetn2 STABLE 3 OFFLINE OFFLINE STABLE ora.cvu 1 ONLINE ONLINE racsetn2 STABLE ora.ntnt.db 1 ONLINE ONLINE racsetn1 Open,HOME=/ora_home/ app/19c/db,STABLE 2 ONLINE ONLINE racsetn2 Open,HOME=/ora_home/ app/19c/db,STABLE ora.qosmserver 1 ONLINE ONLINE racsetn2 STABLE ora.racsetn1.vip 1 ONLINE ONLINE racsetn1 STABLE ora.racsetn2.vip 1 ONLINE ONLINE racsetn2 STABLE ora.scan1.vip 1 ONLINE ONLINE racsetn1 STABLE ora.scan2.vip 1 ONLINE ONLINE racsetn2 STABLE ora.scan3.vip 1 ONLINE ONLINE racsetn2 STABLE -------------------------------------------------------------------------------- [oracle@racsetn1 dbs]$ [oracle@racsetn1 dbs]$ /grid_home/app/19c/grid/bin/crsctl check cluster -all ************************************************************** racsetn1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** racsetn2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** On Target Side (Where DB will be Duplicated). [oracle@racnoden1 dbs]$ [oracle@racnoden1 dbs]$ /grid_home/app/19c/grid/bin/crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.chad ONLINE OFFLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.net1.network ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.ons ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.proxy_advm OFFLINE OFFLINE racnoden1 STABLE OFFLINE OFFLINE racnoden2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 ONLINE OFFLINE STABLE ora.DATA_REP.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.FRA_RECO.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE racnoden1 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE racnoden2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE racnoden2 STABLE ora.OCR.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.asm(ora.asmgroup) 1 ONLINE ONLINE racnoden1 Started,STABLE 2 ONLINE ONLINE racnoden2 Started,STABLE 3 OFFLINE OFFLINE STABLE ora.asmnet1.asmnetwork(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.cvu 1 ONLINE ONLINE racnoden2 STABLE ora.qosmserver 1 ONLINE ONLINE racnoden2 STABLE ora.racnoden1.vip 1 ONLINE ONLINE racnoden1 STABLE ora.racnoden2.vip 1 ONLINE ONLINE racnoden2 STABLE ora.scan1.vip 1 ONLINE ONLINE racnoden1 STABLE ora.scan2.vip 1 ONLINE ONLINE racnoden2 STABLE ora.scan3.vip 1 ONLINE ONLINE racnoden2 STABLE -------------------------------------------------------------------------------- [oracle@racnoden1 dbs]$ /grid_home/app/19c/grid/bin/crsctl check cluster -all ************************************************************** racnoden1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** racnoden2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [oracle@racnoden1 dbs]$ 2=> Setting Up tnsnames.ora file on Source & Target Side. We need to create tnsnames.ora file on Source and Target side database hosts as below. -> Source Side - All Nodes [oracle@racsetn1 tmp]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /ora_home/app/19c/db/network/admin/tnsnames.ora # Generated by Oracle configuration tools. NTNT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab) ) ) TART = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.103)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.104)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab) ) ) [oracle@racsetn1 tmp]$ [oracle@racsetn2 ~]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora.racsetn2 Network Configuration File: /ora_home/app/19c/db/network/admin/tnsnames.ora.racsetn2 # Generated by Oracle configuration tools. NTNT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab) ) ) TART = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.103)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.104)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab) ) ) [oracle@racsetn2 ~]$ -> Target Side - All Nodes [oracle@racnoden1 dbs]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /ora_home/app/19c/db/network/admin/tnsnames.ora # Generated by Oracle configuration tools. TART = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racnode-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab)(UR=A) ) ) NTNT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab) ) ) [oracle@racnoden1 dbs]$ [oracle@racnoden2 ~]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /ora_home/app/19c/db/network/admin/tnsnames.ora # Generated by Oracle configuration tools. TART = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racnode-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab)(UR=A) ) ) NTNT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab) ) ) [oracle@racnoden2 ~]$ 3=> Setting Up Static Listener on Target Side Cluster_Home on NODE-1 only as below. [oracle@racnoden1 dbs]$ su - grid Password: [grid@racnoden1 ~]$ cat $ORACLE_HOME/network/admin/listener.ora LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET # line added by Agent SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = TART.ace2oracle.lab) (ORACLE_HOME = /ora_home/app/19c/db) (SID_NAME = TART1) ) ) [grid@racnoden1 ~]$ lsnrctl status Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 25-JAN-2023 20:23:30 Uptime 0 days 5 hr. 32 min. 56 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /grid_home/app/19c/grid/network/admin/listener.ora Listener Log File /grid_home/app/oracle/diag/tnslsnr/racnoden1/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.101)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.103)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_DATA_REP" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_FRA_RECO" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_OCR" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "TART.ace2oracle.lab" has 1 instance(s). Instance "TART1", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully [grid@racnoden1 ~]$ [grid@racnoden1 ~]$ lsnrctl reload LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 01:56:25 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) The command completed successfully [grid@racnoden1 ~]$ lsnrctl service LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 01:56:30 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) Services Summary... Service "TART.ace2oracle.lab" has 1 instance(s). Instance "TART1", status UNKNOWN, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 LOCAL SERVER The command completed successfully [grid@racnoden1 ~]$ lsnrctl status LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 01:56:34 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 25-JAN-2023 20:23:30 Uptime 0 days 5 hr. 33 min. 9 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /grid_home/app/19c/grid/network/admin/listener.ora Listener Log File /grid_home/app/oracle/diag/tnslsnr/racnoden1/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) Services Summary... Service "TART.ace2oracle.lab" has 1 instance(s). Instance "TART1", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully [grid@racnoden1 ~]$ ssh racnoden2 [grid@racnoden2 ~]$ [grid@racnoden2 ~]$ cat $ORACLE_HOME/network/admin/listener.ora LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set [grid@racnoden2 ~]$ [grid@racnoden2 ~]$ lsnrctl status Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 25-JAN-2023 20:22:34 Uptime 0 days 5 hr. 34 min. 21 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /grid_home/app/19c/grid/network/admin/listener.ora Listener Log File /grid_home/app/oracle/diag/tnslsnr/racnoden2/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.102)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.104)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "+ASM_DATA_REP" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "+ASM_FRA_RECO" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "+ASM_OCR" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... The command completed successfully [grid@racnoden2 ~]$ lsnrctl services LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 01:57:00 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER Service "+ASM_DATA_REP" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER Service "+ASM_FRA_RECO" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER Service "+ASM_OCR" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER The command completed successfully 4=> Copying Password file from Source to Destination on ALL NODES. -> Get password file from Source as below. [oracle@racsetn1 tmp]$ su - grid Password: [grid@racsetn1 ~]$ asmcmd ASMCMD> pwget --dbuniquename NTNT +DATA/NTNT/PASSWORD/pwdntnt.365.1123698383 ASMCMD> ASMCMD> pwcopy +DATA/NTNT/PASSWORD/pwdntnt.365.1123698383 /tmp/pwfile copying +DATA/NTNT/PASSWORD/pwdntnt.365.1123698383 -> /tmp/pwfile ASMCMD> exit [grid@racsetn1 ~]$ ls -lrt /tmp/pwfile -rw-r-----. 1 grid oinstall 2048 Jan 26 02:01 /tmp/pwfile [grid@racsetn1 ~]$ chmod 777 /tmp/pwfile [grid@racsetn1 ~]$ logout [oracle@racsetn1 tmp]$ -> Send password file from Source to TARGET ALL NODES in $ORACLE_HOME/dbs as below , per instancwise. [oracle@racsetn1 tmp]$ scp /tmp/pwfile racnoden1:/ora_home/app/19c/db/dbs/orapwTART1 oracle@racnoden1's password: pwfile 100% 2048 3.6MB/s 00:00 [oracle@racsetn1 tmp]$ [oracle@racsetn1 tmp]$ scp /tmp/pwfile racnoden2:/ora_home/app/19c/db/dbs/orapwTART2 oracle@racnoden2's password: pwfile 100% 2048 4.8MB/s 00:00 [oracle@racsetn1 tmp]$ 5=> Test TNSPING and connections as below. -> TNSPING from Source - ALL Nodes [oracle@racsetn1 tmp]$ [oracle@racsetn1 tmp]$ tnsping TART TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:05:59 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.103)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.104)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab))) OK (0 msec) [oracle@racsetn1 tmp]$ [oracle@racsetn1 tmp]$ tnsping NTNT TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:02 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab))) OK (10 msec) [oracle@racsetn1 tmp]$ ssh racsetn2 Last login: Thu Jan 26 01:53:45 2023 from racsetn1.ace2oracle.lab [oracle@racsetn2 ~]$ tnsping TART TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:20 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racnode-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab))) OK (0 msec) [oracle@racsetn2 ~]$ [oracle@racsetn2 ~]$ [oracle@racsetn2 ~]$ tnsping NTNT TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:23 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab))) OK (0 msec) [oracle@racsetn2 ~]$ logout Connection to racsetn2 closed. [oracle@racsetn1 tmp]$ -> TNSPING from Target - ALL Nodes [oracle@racnoden1 dbs]$ tnsping TART TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:35 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racnode-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab)(UR=A))) OK (0 msec) [oracle@racnoden1 dbs]$ tnsping NTNT TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:37 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab))) OK (0 msec) [oracle@racnoden1 dbs]$ ssh racnoden2 [oracle@racnoden2 ~]$ [oracle@racnoden2 ~]$ tnsping NTNT TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:50 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racset-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = NTNT.ace2oracle.lab))) OK (0 msec) [oracle@racnoden2 ~]$ tnsping TART TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 26-JAN-2023 02:06:53 Copyright (c) 1997, 2019, Oracle. All rights reserved. Used parameter files: /ora_home/app/19c/db/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racnode-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TART.ace2oracle.lab)(UR=A))) OK (0 msec) [oracle@racnoden2 ~]$ 6=> Check connection as Auxiliary and Target as below. -> From Source DB [oracle@racsetn1 dbs]$ rman target sys/Oracle_4U@NTNT Recovery Manager: Release 19.0.0.0.0 - Production on Thu Jan 26 02:09:51 2023 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. connected to target database: NTNT (DBID=3125572952) RMAN> Recovery Manager complete. [oracle@racsetn1 dbs]$ rman auxiliary sys/Oracle_4U@TART Recovery Manager: Release 19.0.0.0.0 - Production on Thu Jan 26 02:10:04 2023 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. connected to auxiliary database (not started) RMAN> 7=> Prepare Parameter file copy to Target side as below. [oracle@racsetn1 tmp]$ sqlplus "/as sysdba" Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> create pfile='/tmp/pfile_s.ora' from spfile; created. -> Get DB_FILE and LOG_FIEL path from Source DB as below. [oracle@racsetn1 dbs]$ sqlplus "/as sysdba" SQL> select member from v$logfile; MEMBER -------------------------------------------------------------------------------- +DATA/NTNT/ONLINELOG/group_2.272.1123698529 +FRA/NTNT/ONLINELOG/group_2.789.1123698531 +DATA/NTNT/ONLINELOG/group_1.280.1123698529 +FRA/NTNT/ONLINELOG/group_1.783.1123698531 +DATA/NTNT/ONLINELOG/group_3.279.1123699801 +FRA/NTNT/ONLINELOG/group_3.821.1123699803 +DATA/NTNT/ONLINELOG/group_4.278.1123699813 +FRA/NTNT/ONLINELOG/group_4.851.1123699813 8 rows selected. SQL> select file_name from dba_data_files; FILE_NAME -------------------------------------------------------------------------------- +DATA/NTNT/DATAFILE/system.366.1123698405 +DATA/NTNT/DATAFILE/sysaux.358.1123698431 +DATA/NTNT/DATAFILE/undotbs1.363.1123698445 +DATA/NTNT/DATAFILE/users.347.1123698447 +DATA/NTNT/DATAFILE/undotbs2.262.1123699645 +DATA/NTNT/DATAFILE/rman_tspitr.257.1126727007 6 rows selected. [oracle@racsetn1 dbs]$ 8=> Edit parameters as per TARGET DB as below. [oracle@racnoden1 dbs]$ cat pfile_tart.ora *.audit_file_dest='/ora_home/app/oracle/admin/TART/adump' *.audit_trail='db' *.cluster_database=FALSE *.compatible='19.0.0' *.control_files='+DATA_REP/TART/CONTROLFILE/cont01.ctl','+DATA_REP/TART/CONTROLFILE/cont02.ctl' *.db_block_size=8192 *.db_domain='ace2oracle.lab' *.db_name='TART' *.db_recovery_file_dest='+FRA_RECO' *.db_recovery_file_dest_size=8256m *.diagnostic_dest='/ora_home/app/oracle' *.log_archive_format='%t_%s_%r.dbf' *.nls_language='AMERICAN' *.nls_territory='AMERICA' *.open_cursors=300 *.pga_aggregate_target=871m *.processes=320 *.remote_login_passwordfile='exclusive' *.sga_target=2612m TART1.thread=1 TART1.undo_tablespace='UNDOTBS1' *.log_file_name_convert='+DATA/NTNT','+DATA_REP/TART','+FRA/NTNT','+FRA_RECO/TART' *.db_file_name_convert='+DATA/NTNT','+DATA_REP/TART' [oracle@racnoden1 dbs]$ 9=>Create Required Directories on Target Side DB as below. -> Create File system on both nodes [oracle@racnoden1 dbs]$ mkdir -p /ora_home/app/oracle/admin/TART/adump [oracle@racnoden1 dbs]$ ssh racnoden2 Last login: Thu Jan 26 02:06:43 2023 from racnoden1.ace2oracle.lab [oracle@racnoden2 ~]$ mkdir -p /ora_home/app/oracle/admin/TART/adump -> Create ASM directories as below [oracle@racnoden1 dbs]$ su - grid [grid@racnoden1 ~]$ asmcmd ASMCMD> ls DATA_REP/ FRA_RECO/ OCR/ ASMCMD> cd DATA_REP ASMCMD> mkdir TART ASMCMD> cd TART ASMCMD> pwd +DATA_REP/TART ASMCMD> cd ../.. ASMCMD> ls DATA_REP/ FRA_RECO/ OCR/ ASMCMD> cd FRA_RECO ASMCMD> mkdir TART ASMCMD> cd TART ASMCMD> pwd +FRA_RECO/TART ASMCMD> 10=> Export ORACLE_SID and start database on TARGET Side as NOMOUNT. [oracle@racnoden1 dbs]$ echo $ORACLE_SID TART1 [oracle@racnoden1 dbs]$ sqlplus "/as sysdba" SQL> startup nomount pfile='pfile_tart.ora'; ORACLE instance started. Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes SQL> 11=> Prepare rman script as below on Source DB [oracle@racsetn1 dbs]$ cat rman_clone.cmd run { allocate channel src1 type disk; allocate channel src2 type disk; allocate channel src3 type disk; allocate channel src4 type disk; allocate auxiliary channel aux1 type disk; allocate auxiliary channel aux2 type disk; allocate auxiliary channel aux3 type disk; allocate auxiliary channel aux4 type disk; allocate auxiliary channel aux5 type disk; allocate auxiliary channel aux6 type disk; duplicate target database to TART from active database USING BACKUPSET ; } [oracle@racsetn1 dbs]$ 12=> Start DUPLICATING Source database as below . nohup rman target sys/Oracle_4U@NTNT auxiliary sys/Oracle_4U@TART msglog /ora_home/app/19c/db/dbs/rman_clone2_log cmdfile=/ora_home/app/19c/db/dbs/rman_clone.cmd & [oracle@racsetn1 dbs]$ [oracle@racsetn1 dbs]$ nohup rman target sys/Oracle_4U@NTNT auxiliary sys/Oracle_4U@TART msglog /ora_home/app/19c/db/dbs/rman_clone2_log cmdfile=/ora_home/app/19c/db/dbs/rman_clone.cmd & [1] 24234 [oracle@racsetn1 dbs]$ nohup: ignoring input and appending output to ‘nohup.out’ [oracle@racsetn1 dbs]$ tail -100f /ora_home/app/19c/db/dbs/rman_clone2_log Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. connected to target database: NTNT (DBID=3125572952) connected to auxiliary database: TART (not mounted) RMAN> run 2> { 3> allocate channel src1 type disk; 4> allocate channel src2 type disk; 5> allocate channel src3 type disk; 6> allocate channel src4 type disk; 7> allocate auxiliary channel aux1 type disk; 8> allocate auxiliary channel aux2 type disk; 9> allocate auxiliary channel aux3 type disk; 10> allocate auxiliary channel aux4 type disk; 11> allocate auxiliary channel aux5 type disk; 12> allocate auxiliary channel aux6 type disk; 13> duplicate target database to TART from active database USING BACKUPSET ; 14> } 15> using target database control file instead of recovery catalog allocated channel: src1 channel src1: SID=27 instance=NTNT1 device type=DISK allocated channel: src2 channel src2: SID=28 instance=NTNT2 device type=DISK allocated channel: src3 channel src3: SID=409 instance=NTNT1 device type=DISK allocated channel: src4 channel src4: SID=160 instance=NTNT2 device type=DISK allocated channel: aux1 channel aux1: SID=15 device type=DISK allocated channel: aux2 channel aux2: SID=140 device type=DISK allocated channel: aux3 channel aux3: SID=268 device type=DISK allocated channel: aux4 channel aux4: SID=393 device type=DISK allocated channel: aux5 channel aux5: SID=16 device type=DISK allocated channel: aux6 channel aux6: SID=141 device type=DISK Starting Duplicate Db at 26-JAN-23 current log archived contents of Memory Script: { sql clone "create spfile from memory"; } executing Memory Script sql statement: create spfile from memory contents of Memory Script: { shutdown clone immediate; startup clone nomount; } executing Memory Script ... ... Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes allocated channel: aux1 channel aux1: SID=379 device type=DISK allocated channel: aux2 channel aux2: SID=13 device type=DISK allocated channel: aux3 channel aux3: SID=138 device type=DISK allocated channel: aux4 channel aux4: SID=266 device type=DISK allocated channel: aux5 channel aux5: SID=267 device type=DISK allocated channel: aux6 channel aux6: SID=394 device type=DISK contents of Memory Script: { sql clone "alter system set db_name = ''NTNT'' comment= ''Modified by RMAN duplicate'' scope=spfile"; sql clone "alter system set db_unique_name = ''TART'' comment= ''Modified by RMAN duplicate'' scope=spfile"; shutdown clone immediate; startup clone force nomount restore clone from service 'NTNT' primary controlfile; alter clone database mount; } executing Memory Script sql statement: alter system set db_name = ''NTNT'' comment= ''Modified by RMAN duplicate'' scope=spfile sql statement: alter system set db_unique_name = ''TART'' comment= ''Modified by RMAN duplicate'' scope=spfile Oracle instance shut down Oracle instance started Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes allocated channel: aux1 channel aux1: SID=379 device type=DISK allocated channel: aux2 channel aux2: SID=13 device type=DISK allocated channel: aux3 channel aux3: SID=138 device type=DISK allocated channel: aux4 channel aux4: SID=266 device type=DISK allocated channel: aux5 channel aux5: SID=140 device type=DISK allocated channel: aux6 channel aux6: SID=15 device type=DISK Starting restore at 26-JAN-23 channel aux1: starting datafile backup set restore channel aux1: using network backup set from service NTNT channel aux1: restoring control file channel aux1: restore complete, elapsed time: 00:00:05 output file name=+DATA_REP/TART/CONTROLFILE/cont01.ctl output file name=+DATA_REP/TART/CONTROLFILE/cont02.ctl Finished restore at 26-JAN-23 database mounted RMAN-05529: warning: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only. contents of Memory Script: { sql clone 'alter database flashback off'; set newname for datafile 1 to "+DATA_REP"; set newname for datafile 2 to "+DATA_REP"; set newname for datafile 3 to "+DATA_REP"; set newname for datafile 4 to "+DATA_REP"; set newname for datafile 5 to "+DATA_REP"; set newname for datafile 7 to "+DATA_REP"; restore from nonsparse from service 'NTNT' clone database ; sql 'alter system archive log current'; } executing Memory Script sql statement: alter database flashback off executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME Starting restore at 26-JAN-23 channel aux1: starting datafile backup set restore channel aux1: using network backup set from service NTNT channel aux1: specifying datafile(s) to restore from backup set channel aux1: restoring datafile 00001 to +DATA_REP channel aux2: starting datafile backup set restore channel aux2: using network backup set from service NTNT channel aux2: specifying datafile(s) to restore from backup set channel aux2: restoring datafile 00002 to +DATA_REP channel aux3: starting datafile backup set restore channel aux3: using network backup set from service NTNT channel aux3: specifying datafile(s) to restore from backup set channel aux3: restoring datafile 00003 to +DATA_REP channel aux4: starting datafile backup set restore channel aux4: using network backup set from service NTNT channel aux4: specifying datafile(s) to restore from backup set channel aux4: restoring datafile 00004 to +DATA_REP channel aux5: starting datafile backup set restore channel aux5: using network backup set from service NTNT channel aux5: specifying datafile(s) to restore from backup set channel aux5: restoring datafile 00005 to +DATA_REP channel aux6: starting datafile backup set restore channel aux6: using network backup set from service NTNT channel aux6: specifying datafile(s) to restore from backup set channel aux6: restoring datafile 00007 to +DATA_REP channel aux2: restore complete, elapsed time: 00:00:06 channel aux4: restore complete, elapsed time: 00:00:04 channel aux5: restore complete, elapsed time: 00:00:05 channel aux6: restore complete, elapsed time: 00:00:03 channel aux1: restore complete, elapsed time: 00:00:10 channel aux3: restore complete, elapsed time: 00:00:09 Finished restore at 26-JAN-23 sql statement: alter system archive log current current log archived contents of Memory Script: { restore clone force from service 'NTNT' archivelog from scn 2987129; switch clone datafile all; } executing Memory Script Starting restore at 26-JAN-23 channel aux1: starting archived log restore to default destination channel aux1: using network backup set from service NTNT channel aux1: restoring archived log archived log thread=1 sequence=61 channel aux2: starting archived log restore to default destination channel aux2: using network backup set from service NTNT channel aux2: restoring archived log archived log thread=1 sequence=62 channel aux3: starting archived log restore to default destination channel aux3: using network backup set from service NTNT channel aux3: restoring archived log archived log thread=1 sequence=63 channel aux4: starting archived log restore to default destination channel aux4: using network backup set from service NTNT channel aux4: restoring archived log archived log thread=2 sequence=65 channel aux5: starting archived log restore to default destination channel aux5: using network backup set from service NTNT channel aux5: restoring archived log archived log thread=2 sequence=66 channel aux1: restore complete, elapsed time: 00:00:06 channel aux2: restore complete, elapsed time: 00:00:06 channel aux3: restore complete, elapsed time: 00:00:07 channel aux4: restore complete, elapsed time: 00:00:06 channel aux5: restore complete, elapsed time: 00:00:04 Finished restore at 26-JAN-23 datafile 1 switched to datafile copy input datafile copy RECID=13 STAMP=1127096670 file name=+DATA_REP/TART/DATAFILE/system.278.1127096633 datafile 2 switched to datafile copy input datafile copy RECID=14 STAMP=1127096670 file name=+DATA_REP/TART/DATAFILE/rman_tspitr.280.1127096635 datafile 3 switched to datafile copy input datafile copy RECID=15 STAMP=1127096670 file name=+DATA_REP/TART/DATAFILE/sysaux.261.1127096635 datafile 4 switched to datafile copy input datafile copy RECID=16 STAMP=1127096670 file name=+DATA_REP/TART/DATAFILE/undotbs1.257.1127096637 datafile 5 switched to datafile copy input datafile copy RECID=17 STAMP=1127096670 file name=+DATA_REP/TART/DATAFILE/undotbs2.258.1127096639 datafile 7 switched to datafile copy input datafile copy RECID=18 STAMP=1127096671 file name=+DATA_REP/TART/DATAFILE/users.259.1127096639 contents of Memory Script: { set until scn 2987359; recover clone database delete archivelog ; } executing Memory Script executing command: SET until clause Starting recover at 26-JAN-23 starting media recovery archived log for thread 1 with sequence 62 is already on disk as file +FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_62.258.1127096663 archived log for thread 1 with sequence 63 is already on disk as file +FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_63.257.1127096667 archived log for thread 2 with sequence 65 is already on disk as file +FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_65.690.1127096669 archived log for thread 2 with sequence 66 is already on disk as file +FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_66.654.1127096669 archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_62.258.1127096663 thread=1 sequence=62 archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_65.690.1127096669 thread=2 sequence=65 archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_63.257.1127096667 thread=1 sequence=63 archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_66.654.1127096669 thread=2 sequence=66 media recovery complete, elapsed time: 00:00:04 Finished recover at 26-JAN-23 contents of Memory Script: { delete clone force archivelog all; } executing Memory Script deleted archived log archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_62.258.1127096663 RECID=2 STAMP=1127096668 Deleted 1 objects deleted archived log archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_61.259.1127096663 RECID=1 STAMP=1127096667 Deleted 1 objects deleted archived log archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_1_seq_63.257.1127096667 RECID=3 STAMP=1127096669 Deleted 1 objects deleted archived log archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_66.654.1127096669 RECID=5 STAMP=1127096670 Deleted 1 objects deleted archived log archived log file name=+FRA_RECO/TART/ARCHIVELOG/2023_01_26/thread_2_seq_65.690.1127096669 RECID=4 STAMP=1127096669 Deleted 1 objects released channel: src1 released channel: src2 released channel: src3 released channel: src4 released channel: aux1 released channel: aux2 released channel: aux3 released channel: aux4 released channel: aux5 released channel: aux6 Oracle instance started Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes contents of Memory Script: { sql clone "alter system set db_name = ''TART'' comment= ''Reset to original value by RMAN'' scope=spfile"; sql clone "alter system reset db_unique_name scope=spfile"; } executing Memory Script sql statement: alter system set db_name = ''TART'' comment= ''Reset to original value by RMAN'' scope=spfile sql statement: alter system reset db_unique_name scope=spfile Oracle instance started Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes sql statement: CREATE CONTROLFILE REUSE SET DATABASE "TART" RESETLOGS ARCHIVELOG MAXLOGFILES 192 MAXLOGMEMBERS 3 MAXDATAFILES 1024 MAXINSTANCES 32 MAXLOGHISTORY 292 LOGFILE GROUP 1 ( '+FRA_RECO', '+DATA_REP' ) SIZE 200 M REUSE, GROUP 2 ( '+FRA_RECO', '+DATA_REP' ) SIZE 200 M REUSE DATAFILE '+DATA_REP/TART/DATAFILE/system.278.1127096633' CHARACTER SET AL32UTF8 sql statement: ALTER DATABASE ADD LOGFILE INSTANCE 'i2' GROUP 3 ( '+DATA_REP', '+FRA_RECO' ) SIZE 200 M REUSE, GROUP 4 ( '+DATA_REP', '+FRA_RECO' ) SIZE 200 M REUSE contents of Memory Script: { set newname for tempfile 1 to "+DATA_REP"; switch clone tempfile all; catalog clone datafilecopy "+DATA_REP/TART/DATAFILE/rman_tspitr.280.1127096635", "+DATA_REP/TART/DATAFILE/sysaux.261.1127096635", "+DATA_REP/TART/DATAFILE/undotbs1.257.1127096637", "+DATA_REP/TART/DATAFILE/undotbs2.258.1127096639", "+DATA_REP/TART/DATAFILE/users.259.1127096639"; switch clone datafile all; } executing Memory Script executing command: SET NEWNAME renamed tempfile 1 to +DATA_REP in control file cataloged datafile copy datafile copy file name=+DATA_REP/TART/DATAFILE/rman_tspitr.280.1127096635 RECID=1 STAMP=1127096783 cataloged datafile copy datafile copy file name=+DATA_REP/TART/DATAFILE/sysaux.261.1127096635 RECID=2 STAMP=1127096783 cataloged datafile copy datafile copy file name=+DATA_REP/TART/DATAFILE/undotbs1.257.1127096637 RECID=3 STAMP=1127096783 cataloged datafile copy datafile copy file name=+DATA_REP/TART/DATAFILE/undotbs2.258.1127096639 RECID=4 STAMP=1127096783 cataloged datafile copy datafile copy file name=+DATA_REP/TART/DATAFILE/users.259.1127096639 RECID=5 STAMP=1127096783 datafile 2 switched to datafile copy input datafile copy RECID=1 STAMP=1127096783 file name=+DATA_REP/TART/DATAFILE/rman_tspitr.280.1127096635 datafile 3 switched to datafile copy input datafile copy RECID=2 STAMP=1127096783 file name=+DATA_REP/TART/DATAFILE/sysaux.261.1127096635 datafile 4 switched to datafile copy input datafile copy RECID=3 STAMP=1127096783 file name=+DATA_REP/TART/DATAFILE/undotbs1.257.1127096637 datafile 5 switched to datafile copy input datafile copy RECID=4 STAMP=1127096783 file name=+DATA_REP/TART/DATAFILE/undotbs2.258.1127096639 datafile 7 switched to datafile copy input datafile copy RECID=5 STAMP=1127096783 file name=+DATA_REP/TART/DATAFILE/users.259.1127096639 contents of Memory Script: { Alter clone database open resetlogs; } executing Memory Script database opened Reenabling controlfile options for auxiliary database Executing: alter database flashback on Cannot remove created server parameter file Finished Duplicate Db at 26-JAN-23 Recovery Manager complete. Done nohup rman target sys/Oracle_4U@NTNT auxiliary sys/Oracle_4U@TART msglog /ora_home/app/19c/db/dbs/rman_clone2_log cmdfile=/ora_home/app/19c/db/dbs/rman_clone.cmd 13=>As OUR DB is cloned now. Let we create spfile to ASM DG as below. SQL> create spfile='+DATA_REP/TART/PARAMETERFILE/spfiletnt.ora' from memory; File created. SQL> exit -> EDIT Below entry on both nodes as below. [oracle@racnoden1 dbs]$ [oracle@racnoden1 dbs]$ cat init init.ora initTATRT1.ora [oracle@racnoden1 dbs]$ cat initTATRT1.ora spfile='+DATA_REP/TART/PARAMETERFILE/spfiletnt.ora' [oracle@racnoden1 dbs]$ [oracle@racnoden1 dbs]$ ssh racnoden2 Last login: Fri Jan 27 02:24:43 2023 from racnoden1.ace2oracle.lab [oracle@racnoden2 ~]$ [oracle@racnoden2 ~]$ cd $ORACLE_HOME/dbs [oracle@racnoden2 dbs]$ cat initTATRT2.ora spfile='+DATA_REP/TART/PARAMETERFILE/spfiletnt.ora' [oracle@racnoden2 dbs]$ -> Recycle database as below post removing spfileTART1.ora from NODE-1 $ORACLE_HOME/dbs location. [oracle@racnoden1 dbs]$ cd $ORACLE_HOME/dbs [oracle@racnoden1 dbs]$ rm spfileTART1.ora -> Recycle as below. [oracle@racnoden1 dbs]$ sqlplus "/as sysdba" SQL> shut immediate Database closed. Database dismounted. ORACLE instance shut down. [oracle@racnoden1 dbs]$ sqlplus "/as sysdba" SQL> startup ORACLE instance started. Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes Database mounted. Database opened. 14=> Set RAC parameters to ENABLE NODE-2 as below and start DATABASE as Clustered on ALL NODES [oracle@racnoden1 dbs]$ sqlplus "/as sysdba" SQL> show parameter pfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string +DATA_REP/TART/PARAMETERFILE/spfiletnt.ora SQL> alter system set undo_tablespace=UNDOTBS2 sid='TART2' scope=spfile; SQL> alter system set instance_number=1 sid='TART1' scope=spfile; SQL> alter system set instance_number=2 sid='TART2' scope=spfile; SQL> alter system set instance_name='TART1' sid='TART1' scope=spfile; SQL> alter system set instance_name='TART2' sid='TART2' scope=spfile; SQL> alter system set thread=1 sid='TART1' scope=spfile; SQL> alter system set thread=2 sid='TART2' scope=spfile; SQL> alter system set cluster_database=TRUE scope=spfile; SQL> alter system set remote_listener='racnode-scan:1521' scope=spfile; System altered. SQL> shut immediate Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 2751463160 bytes Fixed Size 8900344 bytes Variable Size 570425344 bytes Database Buffers 2164260864 bytes Redo Buffers 7876608 bytes Database mounted. Database opened. -> Start database on ALL NOES as . [oracle@racnoden1 dbs]$ srvctl start database -d TART [oracle@racnoden1 dbs]$ /grid_home/app/19c/grid/bin/crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.chad ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.net1.network ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.ons ONLINE ONLINE racnoden1 STABLE ONLINE ONLINE racnoden2 STABLE ora.proxy_advm OFFLINE OFFLINE racnoden1 STABLE OFFLINE OFFLINE racnoden2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 ONLINE OFFLINE STABLE ora.DATA_REP.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.FRA_RECO.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE racnoden1 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE racnoden2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE racnoden2 STABLE ora.OCR.dg(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.asm(ora.asmgroup) 1 ONLINE ONLINE racnoden1 Started,STABLE 2 ONLINE ONLINE racnoden2 Started,STABLE 3 OFFLINE OFFLINE STABLE ora.asmnet1.asmnetwork(ora.asmgroup) 1 ONLINE ONLINE racnoden1 STABLE 2 ONLINE ONLINE racnoden2 STABLE 3 OFFLINE OFFLINE STABLE ora.cvu 1 ONLINE ONLINE racnoden2 STABLE ora.qosmserver 1 ONLINE ONLINE racnoden2 STABLE ora.racnoden1.vip 1 ONLINE ONLINE racnoden1 STABLE ora.racnoden2.vip 1 ONLINE ONLINE racnoden2 STABLE ora.scan1.vip 1 ONLINE ONLINE racnoden1 STABLE ora.scan2.vip 1 ONLINE ONLINE racnoden2 STABLE ora.scan3.vip 1 ONLINE ONLINE racnoden2 STABLE ora.tart.db 1 ONLINE ONLINE racnoden1 Open,HOME=/ora_home/app/19c/db,STABLE 2 ONLINE ONLINE racnoden2 Open,HOME=/ora_home/app/19c/db,STABLE -------------------------------------------------------------------------------- 15=> Change passwordfile location at OCR level as below. -> Get current OCR level password info [oracle@racnoden1 dbs]$ srvctl config database -d TART Database unique name: TART Database name: Oracle home: /ora_home/app/19c/db Oracle user: oracle Spfile: +DATA_REP/TART/PARAMETERFILE/spfiletnt.ora Password file: <<<< NOT AVAILABLE Domain: ace2oracle.lab Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: DATA_REP Mount point paths: Services: Type: RAC Start concurrency: Stop concurrency: OSDBA group: oinstall OSOPER group: oper Database instances: TART1,TART2 Configured nodes: racnoden1,racnoden2 CSS critical: no CPU count: 0 Memory target: 0 Maximum memory: 0 Default network number for database services: Database is administrator managed -> As we copied password file on dbs locations during setup as below. [oracle@racnoden1 dbs]$ ls -lrt orapwTART1 -rw-r-----. 1 oracle oinstall 2048 Jan 26 02:03 orapwTART1 [oracle@racnoden1 dbs]$ ls -lrt orapwTART1 -rw-r-----. 1 oracle oinstall 2048 Jan 26 02:03 orapwTART1 ->Copy to temp location to access for other user [oracle@racnoden1 dbs]$ cp orapwTART1 /tmp/orapwTART1 -> Swith to grid and move to ASM as below [oracle@racnoden1 dbs]$ su - grid [grid@racnoden1 ~]$ asmcmd ASMCMD> ls DATA_REP/ FRA_RECO/ OCR/ ASMCMD> cd DATA_REP ASMCMD> cd TART ASMCMD> ls CONTROLFILE/ DATAFILE/ ONLINELOG/ PARAMETERFILE/ TEMPFILE/ ASMCMD> mkdir PASSWORD ASMCMD> cd PASSWORD ASMCMD> pwd +DATA_REP/TART/PASSWORD ASMCMD> pwcopy --dbuniquename TART /tmp/orapwTART1 +DATA_REP/TART/PASSWORD/pwdtart copying /tmp/orapwTART1 -> +DATA_REP/TART/PASSWORD/pwdtart ASMCMD-9453: failed to register password file as a CRS resource ASMCMD> ls -lrt WARNING:option 'r' is deprecated for 'ls' please use 'reverse' Type Redund Striped Time Sys Name PASSWORD UNPROT COARSE JAN 27 02:00:00 N pwdtart => +DATA_REP/TART/PASSWORD/pwdtart.267.1127182975 PASSWORD UNPROT COARSE JAN 27 02:00:00 Y pwdtart.267.1127182975 ASMCMD> pwd +DATA_REP/TART/PASSWORD ASMCMD> exit [grid@racnoden1 ~]$ logout -> Login to oracle user and register with CRS as below [oracle@racnoden1 dbs]$ [oracle@racnoden1 dbs]$ srvctl modify database -d TART -pwfile +DATA_REP/TART/PASSWORD/pwdtart [oracle@racnoden1 dbs]$ srvctl config database -d TART Database unique name: TART Database name: Oracle home: /ora_home/app/19c/db Oracle user: oracle Spfile: +DATA_REP/TART/PARAMETERFILE/spfiletnt.ora Password file: +DATA_REP/TART/PASSWORD/pwdtart <<<< INFORMATION available now. Domain: ace2oracle.lab Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: DATA_REP Mount point paths: Services: Type: RAC Start concurrency: Stop concurrency: OSDBA group: oinstall OSOPER group: oper Database instances: TART1,TART2 Configured nodes: racnoden1,racnoden2 CSS critical: no CPU count: 0 Memory target: 0 Maximum memory: 0 Default network number for database services: Database is administrator managed -> Move/remove password file from Both nodes from DBS location [oracle@racnoden1 dbs]$ ls -lrt ora* -rw-r-----. 1 oracle oinstall 2048 Jan 26 02:03 orapwTART1 [oracle@racnoden1 dbs]$ mv orapwTART1 orapwTART1_OLD [oracle@racnoden1 dbs]$ ssh racnoden2 Last login: Fri Jan 27 02:06:51 2023 from racnoden1.ace2oracle.lab [oracle@racnoden2 ~]$ cd $ORACLE_HOME/dbs [oracle@racnoden2 dbs]$ mv orapwTART2 orapwTART2_OLD -> Test password as below. [oracle@racnoden2 dbs]$ sqlplus sys/Oracle_4U@TART as sysdba Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 [oracle@racnoden2 dbs]$ 16=> Check database Health as below. [oracle@racnoden2 dbs]$ sqlplus "/as sysdba" SQL> select name,open_mode,database_role,cdb from gv$database; NAME OPEN_MODE DATABASE_ROLE CDB --------- -------------------- ---------------- --- TART READ WRITE PRIMARY NO TART READ WRITE PRIMARY NO SQL> select * from v$logfile; GROUP# STATUS TYPE MEMBER IS_ CON_ID ---------- ---------- ------- -------------------------------------------------- --- ---------- 2 ONLINE +FRA_RECO/TART/ONLINELOG/group_2.259.1127096787 YES 0 1 ONLINE +FRA_RECO/TART/ONLINELOG/group_1.257.1127096787 YES 0 3 ONLINE +DATA_REP/TART/ONLINELOG/group_3.260.1127096749 NO 0 3 ONLINE +FRA_RECO/TART/ONLINELOG/group_3.690.1127096749 NO 0 4 ONLINE +DATA_REP/TART/ONLINELOG/group_4.276.1127096769 NO 0 4 ONLINE +FRA_RECO/TART/ONLINELOG/group_4.654.1127096769 NO 0 6 rows selected. SQL> SELECT FILE_NAME FROM DBA_DATA_FILES; FILE_NAME ---------------------------------------------------------------------------------------- +DATA_REP/TART/DATAFILE/system.278.1127096633 +DATA_REP/TART/DATAFILE/sysaux.261.1127096635 +DATA_REP/TART/DATAFILE/undotbs1.257.1127096637 +DATA_REP/TART/DATAFILE/users.259.1127096639 +DATA_REP/TART/DATAFILE/undotbs2.258.1127096639 +DATA_REP/TART/DATAFILE/rman_tspitr.280.1127096635 6 rows selected. SQL> SELECT FILE_NAME FROM DBA_TEMP_FILES; FILE_NAME ---------------------------------------------------------------------------------------- +DATA_REP/TART/TEMPFILE/temp.277.1127096815 SQL> SHOW PARAMETER CONTROL NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ control_files string +DATA_REP/TART/CONTROLFILE/cont01.ctl, +DATA_REP/TART/CONTROLFILE/cont02.ctl control_management_pack_access string DIAGNOSTIC+TUNING SQL> select DBID,name,DBID,CONTROLFILE_TYPE from gv$database; DBID NAME DBID CONTROL ---------- --------- ---------- ------- 3144695979 TART 3144695979 CURRENT 3144695979 TART 3144695979 CURRENT SQL> exit 17=>Remove SID_LIST_LISTENER as we set in above steps while creation. [grid@racnoden1 ~]$ lsnrctl status LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 27-JAN-2023 02:31:55 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 27-JAN-2023 01:35:53 Uptime 0 days 0 hr. 56 min. 1 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /grid_home/app/19c/grid/network/admin/listener.ora Listener Log File /grid_home/app/oracle/diag/tnslsnr/racnoden1/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.101)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.103)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_DATA_REP" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_FRA_RECO" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_OCR" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "TART.ace2oracle.lab" has 2 instance(s). Instance "TART1", status UNKNOWN, has 1 handler(s) for this service... Instance "TART1", status READY, has 1 handler(s) for this service... The command completed successfully [grid@racnoden1 ~]$ vi /grid_home/app/19c/grid/network/admin/listener.ora [grid@racnoden1 ~]$ cat /grid_home/app/19c/grid/network/admin/listener.ora LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF # line added by Agent - Disabled by Agent because REMOTE_REGISTRATION_ADDRESS is set ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET # line added by Agent #SID_LIST_LISTENER = # (SID_LIST = # (SID_DESC = # (GLOBAL_DBNAME = TART.ace2oracle.lab) # (ORACLE_HOME = /ora_home/app/19c/db) # (SID_NAME = TART1) # ) # ) [grid@racnoden1 ~]$ [grid@racnoden1 ~]$ lsnrctl reload LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 27-JAN-2023 02:32:32 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) The command completed successfully [grid@racnoden1 ~]$ lsnrctl status LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 27-JAN-2023 02:33:52 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 27-JAN-2023 01:35:53 Uptime 0 days 0 hr. 57 min. 58 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /grid_home/app/19c/grid/network/admin/listener.ora Listener Log File /grid_home/app/oracle/diag/tnslsnr/racnoden1/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.101)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.103)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_DATA_REP" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_FRA_RECO" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "+ASM_OCR" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "TART.ace2oracle.lab" has 1 instance(s). Instance "TART1", status READY, has 1 handler(s) for this service... The command completed successfully
© 2021 Ace2Oracle. All Rights Reserved | Developed By IBOX444