机器简介

Red Hat Enterprise Linux Server release 5.11 (Tikanga)  2台，准备组建RAC

128G，2颗cpu*16线程， raid5，多路径，双网卡绑定。

查看版本

[oracle@CQLT-LTE-DB1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 2 10:31:51 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>

 查看安装后的状态

[root@CQLT-LTE-DB1 cdh_install]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....DATA.dg ora....up.type ONLINE    ONLINE    cqlt...-db1 
ora....ER.lsnr ora....er.type ONLINE    ONLINE    cqlt...-db1 
ora....N1.lsnr ora....er.type ONLINE    ONLINE    cqlt...-db2 
ora.asm        ora.asm.type   ONLINE    ONLINE    cqlt...-db1 
ora....SM1.asm application    ONLINE    ONLINE    cqlt...-db1 
ora....B1.lsnr application    ONLINE    ONLINE    cqlt...-db1 
ora....db1.gsd application    OFFLINE   OFFLINE               
ora....db1.ons application    ONLINE    ONLINE    cqlt...-db1 
ora....db1.vip ora....t1.type ONLINE    ONLINE    cqlt...-db1 
ora....SM2.asm application    ONLINE    ONLINE    cqlt...-db2 
ora....B2.lsnr application    ONLINE    ONLINE    cqlt...-db2 
ora....db2.gsd application    OFFLINE   OFFLINE               
ora....db2.ons application    ONLINE    ONLINE    cqlt...-db2 
ora....db2.vip ora....t1.type ONLINE    ONLINE    cqlt...-db2 
ora.cvu        ora.cvu.type   ONLINE    ONLINE    cqlt...-db1 
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    cqlt...-db1 
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    cqlt...-db2 
ora.ons        ora.ons.type   ONLINE    ONLINE    cqlt...-db1 
ora....ry.acfs ora....fs.type ONLINE    ONLINE    cqlt...-db1 
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    cqlt...-db2 
ora.unicom.db  ora....se.type ONLINE    ONLINE    cqlt...-db1 
[root@CQLT-LTE-DB1 cdh_install]# 

安装好后，继续打PSU

p22191577_112040_Linux-x86-64.zip

注：在打补丁之前，看官方文档，补丁要求OPatch是什么版本，就升级为对应的版本，这里要求OPatch为11.2.0.3.6以上。 如果不是，请升级OPatch

在官网下载后解压得到OPatch

[root@CQLT-LTE-DB1 11.2.0.4]# ls
19852360                                p13390677_112040_Linux-x86-64_3of7.zip
22191577                                p13390677_112040_SOLARIS64_1of7.zip
instantclient-basic-nt-11.2.0.3.0.zip   p13390677_112040_SOLARIS64_2of7.zip
OPatch                                  p13390677_112040_SOLARIS64_3of7.zip
p13390677_112040_HPUX-IA64_1of7.zip     p19852360_112040_Generic.zip
p13390677_112040_HPUX-IA64_2of7.zip     p22191577_112040_Linux-x86-64.zip
p13390677_112040_HPUX-IA64_3of7.zip     p6880880_112000_Linux-x86-64.zip
p13390677_112040_Linux-x86-64_1of7.zip  PatchSearch.xml
p13390677_112040_Linux-x86-64_2of7.zip

把OPatch拷贝到对应 ORACLE_HOME。和GRID_HOME 下，把grid和oracle的OPatch都升级，下面是拷贝后赋权操作。

[root@CQLT-LTE-DB1 dbhome_1]# chown -R oracle:oinstall OPatch/
[root@CQLT-LTE-DB1 dbhome_1]# cd OPatch/
[root@CQLT-LTE-DB1 OPatch]# ls -l
total 156
drwxr-x--- 3 oracle oinstall  4096 Jan 27 14:45 crs
drwxr-x--- 2 oracle oinstall  4096 Jan 27 14:45 docs
-rwxr-x--- 1 oracle oinstall 23695 Jan 27 14:45 emdpatch.pl
drwxr-x--- 2 oracle oinstall  4096 Jan 27 14:45 jlib
drwxr-x--- 5 oracle oinstall  4096 Jan 27 14:45 ocm
-rwxr-x--- 1 oracle oinstall 31056 Jan 27 14:45 opatch
drwxr-x--- 4 oracle oinstall  4096 Jan 27 14:45 opatchauto-dir
-rwxr-x--- 1 oracle oinstall 15184 Jan 27 14:45 opatch.bat
-rwxr-x--- 1 oracle oinstall 10056 Jan 27 14:45 opatchdiag
-rwxr-x--- 1 oracle oinstall  9352 Jan 27 14:45 opatchdiag.bat
-rwxr-x--- 1 oracle oinstall  2576 Jan 27 14:45 opatch.pl
drwxr-x--- 4 oracle oinstall  4096 Jan 27 14:45 opatchprereqs
-rwxr-x--- 1 oracle oinstall  3161 Jan 27 14:45 operr
-rwxr-x--- 1 oracle oinstall  4220 Jan 27 14:45 operr.bat
-rw-r----- 1 oracle oinstall  3177 Jan 27 14:45 operr_readme.txt
drwxr-x--- 3 oracle oinstall  4096 Jan 27 14:45 oplan
-rw-r----- 1 oracle oinstall  2915 Jan 27 14:45 README.txt
drwxr-x--- 2 oracle oinstall  4096 Jan 27 14:45 scripts
-rw-r----- 1 oracle oinstall    27 Jan 27 14:45 version.txt
[root@CQLT-LTE-DB1 OPatch]# cd ..
[root@CQLT-LTE-DB1 dbhome_1]# ls
apex         css          EMStage        jdev     nls            oui       scheduler        uix
assistants   ctx          has            jdk      oc4j           owb       slax             usm
bin          cv           hs             jlib     odbc           owm       sqldeveloper     utl
ccr          dbs          ide            ldap     olap           perl      sqlj             wwg
cdata        dc_ocm       install        lib      OPatch         plsql     sqlplus          xdk
cfgtoollogs  deinstall    instantclient  log      opatch_bk.tar  precomp   srvm
clone        demo         inventory      md       opmn           racg      suptools
config       diagnostics  j2ee           mesg     oracore        rdbms     sysman
crs          dv           javavm         mgw      oraInst.loc    relnotes  timingframework
csmig        emcli        jdbc           network  ord            root.sh   ucp
[root@CQLT-LTE-DB1 dbhome_1]# cd OPatch/
[root@CQLT-LTE-DB1 OPatch]# ls
crs          jlib    opatchauto-dir  opatchdiag.bat  operr             oplan       version.txt
docs         ocm     opatch.bat      opatch.pl       operr.bat         README.txt
emdpatch.pl  opatch  opatchdiag      opatchprereqs   operr_readme.txt  scripts
[root@CQLT-LTE-DB1 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.12

OPatch succeeded.

同样的方法，把节点2也升级

[root@CQLT-LTE-DB2 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.12

OPatch succeeded.

生成响应文件

[root@CQLT-LTE-DB1 bin]# ./emocmrsp -no_banner -output /tmp/ocm.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/tmp/ocm.rsp) was successfully created.
[root@CQLT-LTE-DB1 bin]# 

在第一个节点执行(第2个节点不需要执行，在第一个节点会自动执行节点2 的)

下面自动升级（失败了）
Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured.


As root user, execute the following command on each node of the cluster:


# opatch auto <UNZIPPED_PATCH_LOCATION>/22191577 -ocmrf <ocm response file>

[root@CQLT-LTE-DB1 OPatch]# ./opatch auto /it/11.2.0.4/22191577 -ocmrf /tmp/ocm.rsp
Executing /u01/app/grid/product/11.4.0/grid/perl/bin/perl ./crs/patch11203.pl -patchdir /it/11.2.0.4 -patchn 22191577 -ocmrf /tmp/ocm.rsp -paramfile /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatchauto2016-01-27_15-18-12.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatchauto2016-01-27_15-18-12.report.log

2016-01-27 15:18:12: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948347 failed  for /u01/app/grid/product/11.4.0/grid
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948348 failed  for /u01/app/grid/product/11.4.0/grid
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948355 failed  for /u01/app/grid/product/11.4.0/grid
Opatch version check failed for oracle home  /u01/app/grid/product/11.4.0/grid
Opatch version  check failed
ERROR: update the opatch version for the failed homes and retry

opatch auto failed.

这里失败，是因为我只替换了oracle用户的OPatch, 所以这里我们把grid用户的OPatch也替换了

[root@CQLT-LTE-DB2 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.12

OPatch succeeded.
[root@CQLT-LTE-DB2 OPatch]# pwd
/u01/app/grid/product/11.4.0/grid/OPatch


[root@CQLT-LTE-DB1 OPatch]# pwd
/u01/app/grid/product/11.4.0/grid/OPatch
[root@CQLT-LTE-DB1 OPatch]# ./opatch version
OPatch Version: 11.2.0.3.12

OPatch succeeded.
[root@CQLT-LTE-DB1 OPatch]# 

这里节点1和节点2的grid opatch都已经是 11.2.0.3.12

再试试

[root@CQLT-LTE-DB1 OPatch]# ./opatch auto /it/11.2.0.4/22191577 -ocmrf /tmp/ocm.rsp
Executing /u01/app/grid/product/11.4.0/grid/perl/bin/perl ./crs/patch11203.pl -patchdir /it/11.2.0.4 -patchn 22191577 -ocmrf /tmp/ocm.rsp -paramfile /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatchauto2016-01-27_15-18-12.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatchauto2016-01-27_15-18-12.report.log

2016-01-27 15:18:12: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948347 failed  for /u01/app/grid/product/11.4.0/grid
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948348 failed  for /u01/app/grid/product/11.4.0/grid
The opatch minimum version  check for patch /it/11.2.0.4/22191577/21948355 failed  for /u01/app/grid/product/11.4.0/grid
Opatch version check failed for oracle home  /u01/app/grid/product/11.4.0/grid
Opatch version  check failed
ERROR: update the opatch version for the failed homes and retry

opatch auto failed.

还是报错了。。。。  

不要紧，失败乃成功之母。 不失败哪得成长。

发现是权限无法访问，那我继续chmod 755,   grid:oinstall

[root@CQLT-LTE-DB1 11.2.0.4]# ls -l
total 12567480
drwxr-xr-x  4 root root       4096 Oct 20  2014 19852360
drwxr-xr-x  5 root root       4096 Dec 19 13:31 22191577
-rwxr-xr-x  1 root root   51149941 Jan 26 14:51 instantclient-basic-nt-11.2.0.3.0.zip
drwxr-x--- 10 root root       4096 Aug 18 17:03 OPatch
-rwxr-xr-x  1 root root 1864369899 Jan 26 14:49 p13390677_112040_HPUX-IA64_1of7.zip
 
[root@CQLT-LTE-DB1 11.2.0.4]# chown -R grid:oinstall 22191577
[root@CQLT-LTE-DB1 11.2.0.4]# ls -l
total 12567480
drwxr-xr-x  4 root root           4096 Oct 20  2014 19852360
drwxr-xr-x  5 grid oinstall       4096 Dec 19 13:31 22191577
-rwxr-xr-x  1 root root       51149941 Jan 26 14:51 instantclient-basic-nt-11.2.0.3.0.zip
drwxr-x--- 10 root root           4096 Aug 18 17:03 OPatch
-rwxr-xr-x  1 root root     1864369899 Jan 26 14:49 p13390677_112040_HPUX-IA64_1of7.zip
 
[root@CQLT-LTE-DB1 11.2.0.4]# chmod -R 755 22191577/

继续执行 补丁升级（正常了......）

[root@CQLT-LTE-DB1 OPatch]# ./opatch auto /it/11.2.0.4/22191577 -ocmrf /tmp/ocm.rsp
Executing /u01/app/grid/product/11.4.0/grid/perl/bin/perl ./crs/patch11203.pl -patchdir /it/11.2.0.4 -patchn 22191577 -ocmrf /tmp/ocm.rsp -paramfile /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/grid/product/11.4.0/grid/cfgtoollogs/opatchauto2016-01-27_15-51-02.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/grid/product/11.4.0/grid/cfgtoollogs/opatchauto2016-01-27_15-51-02.report.log

2016-01-27 15:51:02: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/grid/product/11.4.0/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.4.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.4.0/dbhome_1 successfully

patch /it/11.2.0.4/22191577/21948347  apply successful for home  /u01/app/oracle/product/11.4.0/dbhome_1 
patch /it/11.2.0.4/22191577/21948348/custom/server/21948348  apply successful for home  /u01/app/oracle/product/11.4.0/dbhome_1 

Stopping CRS...
Stopped CRS successfully

patch /it/11.2.0.4/22191577/21948347  apply successful for home  /u01/app/grid/product/11.4.0/grid 
patch /it/11.2.0.4/22191577/21948348  apply successful for home  /u01/app/grid/product/11.4.0/grid 
patch /it/11.2.0.4/22191577/21948355  apply successful for home  /u01/app/grid/product/11.4.0/grid 

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.4.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.4.0/dbhome_1 successfully

opatch auto succeeded.
[root@CQLT-LTE-DB1 OPatch]# 

成功后，我们还是更新几个脚本

--执行2个脚本（仅仅在第一节点）




Follow these steps only on the node for which the steps in Section 2.5.2, "Loading Modified SQL Files into the Database" were executed during the patch application.:


Start all database instances running from the Oracle home. (For more information, see Oracle Database Administrator's Guide.)


For each database instance running out of the ORACLE_HOME, connect to the database using SQL*Plus as SYSDBA and run the rollback script as follows:


cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle_PSU_<database SID PREFIX>_ROLLBACK.sql
SQL> QUIT
In an Oracle RAC environment, the name of the rollback script will have the format catbundle_PSU_<database SID PREFIX>_ROLLBACK.sql.


If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script in the previous step. If this is the case, run utlrp.sql to re-validate these Java classes.


cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql
Check the log file for any errors. The log file is found in $ORACLE_BASE/cfgtoollogs/catbundle and is named catbundle_PSU_<database SID>_ROLLBACK_<TIMESTAMP>.log where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS. If there are errors, see Section 3, "Known Issues".


Ensure that you verify the Oracle Inventory and compare the output with the one you ran in Section 2.1.3, "Validation of Oracle Inventory" and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command:


$ opatch lsinventory
All other instances can be started and accessed as usual while you are executing the deinstallation steps.

--执行补丁安装后，检查

[oracle@CQLT-LTE-DB1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.4.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.4.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatch/opatch2016-01-27_16-38-41PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.4.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2016-01-27_16-38-41PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: cqlt-lte-db1
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  21948348     : applied on Wed Jan 27 15:54:26 CST 2016
Unique Patch ID:  19576928
Patch description:  "OCW Patch Set Update : 11.2.0.4.160119 (21948348)"
   Created on 13 Dec 2015, 23:42:11 hrs PST8PDT
   Bugs fixed:
     19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
     20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
     18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
     17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
     18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
     16317771, 10052729, 20340620, 16237657, 20317221, 15917869, 18199185
     18399991, 20186278, 17374271, 18024089, 16849642, 20746251, 20246071
     14270845, 20552947, 18882642, 18414137, 17001914, 17927970, 14378120
     16346413, 15986647, 18068871, 21222147, 18143836, 16206997, 21982225
     19168690, 20235511, 18343490, 21875360, 16613232, 19276791, 17722664
     20440643, 12928658, 18952577, 18520351, 16249829, 18226143, 18265482
     18229842, 17172091, 20676340, 17818075, 20091753, 18231837, 14373486
     20136892, 17483479, 20551654, 18120545, 18729166, 13843841, 21225209
     17405302, 18709496, 18330979, 18744838, 17087371, 20531190, 14525998
     18187697, 20598625, 14385860, 18348155, 19479503, 12928592, 17516024
     18370031, 17764053, 19272663, 17551223, 14671408, 18272135, 14207615
     17500165, 18875012, 14769643, 18464784, 19558324, 18848125, 19241857
     14851828, 17955615, 20315294, 14693336, 16284825, 17352230, 20014326
     17238586, 17089344, 17405605, 17531342, 19398098, 17159489, 17640316
     13823394, 16543190, 22024217, 17983675, 20795241, 17598201, 17481314
     16281493, 18346135, 15986311, 19601468, 17208793, 18700935, 18999857
     14076173, 18428146, 18352845, 17435488, 20408163, 17592037, 18352846
     19616601, 17391726, 17387779, 14777968, 15851860, 16206882, 20141091
     21113068, 20175174, 17305100, 15832129, 19885321, 16901346, 17985714
     18536826, 17780903, 18752378, 18946768, 16876500, 16875342, 17769597
     19955755, 16429265, 18336452, 17273003, 17209968, 16988311, 20094984
     19319357, 17059927, 17046460, 18053631, 16867761, 18774591, 21442094
     20235486, 19359787, 15869775, 19642566, 17447588, 16798862, 15920201

Patch  21948347     : applied on Wed Jan 27 15:53:41 CST 2016
Unique Patch ID:  19564435
Patch description:  "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
   Created on 14 Dec 2015, 03:31:48 hrs PST8PDT
Sub-patch  21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch  20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch  20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 21051852, 18607546, 17205719, 17811429, 17816865, 20506699
     17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
     16992075, 17446237, 14015842, 19972569, 17449815, 21538558, 20925795
     17375354, 19463897, 17982555, 17235750, 13866822, 17478514, 18317531
     18235390, 14338435, 20803583, 13944971, 20142975, 17811789, 16929165
     18704244, 20506706, 17546973, 20334344, 14054676, 17088068, 18264060
     17346091, 17343514, 21538567, 19680952, 18471685, 19211724, 13951456
     21847223, 16315398, 18744139, 16850630, 19049453, 18673304, 17883081
     19915271, 18641419, 18262334, 17006183, 16065166, 18277454, 16833527
     10136473, 18051556, 17865671, 17852463, 18554871, 17853498, 18334586
     17588480, 17551709, 19827973, 17842825, 17344412, 18828868, 17025461
     11883252, 13609098, 17239687, 17602269, 19197175, 22195457, 18316692
     17313525, 12611721, 19544839, 18964939, 17600719, 18191164, 19393542
     17571306, 18482502, 20777150, 19466309, 17040527, 17165204, 18098207
     16785708, 17174582, 16180763, 17465741, 16777840, 12982566, 19463893
     22195465, 12816846, 16875449, 17237521, 19358317, 17811438, 17811447
     17945983, 18762750, 17184721, 16912439, 18061914, 17282229, 18331850
     18202441, 17082359, 18723434, 21972320, 19554106, 14034426, 18339044
     19458377, 17752995, 20448824, 17891943, 17258090, 17767676, 16668584
     18384391, 17040764, 17381384, 15913355, 18356166, 14084247, 20506715
     13853126, 18203837, 14245531, 21756699, 16043574, 22195441, 17848897
     17877323, 21453153, 17468141, 20861693, 17786518, 17912217, 17037130
     18155762, 16956380, 17478145, 17394950, 18189036, 18641461, 18619917
     17027426, 21352646, 16268425, 22195492, 19584068, 18436307, 17265217
     17634921, 13498382, 21526048, 20004087, 22195485, 17443671, 18000422
     22321756, 20004021, 17571039, 21067387, 16344544, 18009564, 14354737
     18135678, 18614015, 20441797, 18362222, 17835048, 16472716, 17936109
     17050888, 17325413, 14010183, 18747196, 17761775, 16721594, 17082983
     20067212, 21179898, 17302277, 18084625, 15990359, 18203835, 17297939
     17811456, 16731148, 21168487, 17215560, 13829543, 14133975, 17694209
     18091059, 17385178, 8322815, 17586955, 17201159, 17655634, 18331812
     19730508, 18868646, 17648596, 16220077, 16069901, 17348614, 17393915
     17274537, 17957017, 18096714, 17308789, 18436647, 14285317, 19289642
     14764829, 18328509, 17622427, 22195477, 16943711, 14368995, 17346671
     18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
     17851160, 17546761, 17798953, 18273830, 22092979, 19972566, 16384983
     17726838, 17360606, 22321741, 13645875, 18199537, 16542886, 21787056
     17889549, 14565184, 17071721, 17610798, 20299015, 21343897, 20657441
     17397545, 18230522, 16360112, 19769489, 12905058, 18641451, 12747740
     18430495, 17042658, 17016369, 14602788, 17551063, 19972568, 21517440
     18508861, 19788842, 14657740, 17332800, 13837378, 19972564, 17186905
     18315328, 19699191, 17437634, 19006849, 19013183, 17296856, 18674024
     17232014, 16855292, 21051840, 14692762, 17762296, 17705023, 19121551
     21330264, 19854503, 19309466, 18681862, 18554763, 20558005, 17390160
     18456514, 16306373, 13955826, 18139690, 17501491, 21668627, 17299889
     17752121, 17889583, 18673325, 18293054, 17242746, 17951233, 17649265
     18094246, 19615136, 17011832, 16870214, 17477958, 18522509, 20631274
     16091637, 17323222, 16595641, 16524926, 18228645, 18282562, 17596908
     17156148, 18031668, 16494615, 17545847, 17655240, 17614134, 13558557
     17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
     18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
     17389192, 17612828, 16194160, 17006570, 17721717, 17570240, 17390431
     16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
     21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
     18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
     16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
     15861775, 16399083, 18018515, 21051858, 18260550, 17036973, 16613964
     17080436, 16579084, 18384537, 18280813, 20296213, 16901385, 15979965
     18441944, 16450169, 9756271, 17892268, 11733603, 16285691, 17587063
     21343775, 16538760, 18180390, 18193833, 21051833, 17238511, 17824637
     16571443, 18306996, 14852021, 18674047, 17853456, 12364061, 22195448



Rac system comprising of multiple nodes
  Local node = cqlt-lte-db1
  Remote node = cqlt-lte-db2

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@CQLT-LTE-DB1 OPatch]$ 

补丁升级完成。