Monday, September 9, 2019

SQL Tuning Advisor for a SQL ID


When a SQL process is running for a long time/running more than usual time, we may handle this situation by running sql tuning advisor for that particular query. Based on the report generated by SQL Tuning advisor, DBA's shall advise the technical team to tune the query. 

Below is the step by step approach to run SQL Tuning advisor for a sql process.

select sesion.sid,
       sesion.username,
       optimizer_mode,
       hash_value,
       address,
       cpu_time,
       elapsed_time,
       sql_text
  from v$sqlarea sqlarea, v$session sesion
 where sesion.sql_hash_value = sqlarea.hash_value
   and sesion.sql_address    = sqlarea.address
   and sesion.username is not null
   order by elapsed_time desc;



select SID,SERIAL#,USERNAME,STATUS,SCHEMANAME,SQL_ID from V$SESSION where SID='902';


Here the above SQL id is different, but the sql id used below is different. The above process completed normally, so captured the remaining process for a different sql id.

DECLARE
  l_sql_tune_task_id  VARCHAR2(100);
BEGIN
  l_sql_tune_task_id := DBMS_SQLTUNE.create_tuning_task (
                          sql_id      => '1016uhs5kf8j4',
                          scope       => DBMS_SQLTUNE.scope_comprehensive,
                          time_limit  => 500,
                          task_name   => '1016uhs5kf8j4_tuning_task',
                          description => 'Tuning task1 for statement 1016uhs5kf8j4');
  DBMS_OUTPUT.put_line('l_sql_tune_task_id: ' || l_sql_tune_task_id);
END;
/


EXEC DBMS_SQLTUNE.execute_tuning_task(task_name => '1016uhs5kf8j4_tuning_task');



set long 65536
set longchunksize 65536
set linesize 100
select dbms_sqltune.report_tuning_task('1016uhs5kf8j4_tuning_task') from dual;
spool off;



SELECT TASK_NAME, STATUS FROM DBA_ADVISOR_LOG WHERE TASK_NAME='1016UHS5KF8J4_TUNING_TASK';



execute dbms_sqltune.drop_tuning_task('1016uhs5kf8j4_tuning_task');





Suppose sql_id is not present in cursor but available in AWR Report, then follow the below steps to run tuning advisor.


select a.instance_number inst_id, a.snap_id,a.plan_hash_value, to_char(begin_interval_time,'dd-mon-yy hh24:mi') btime, abs(extract(minute from (end_interval_time-begin_interval_time)) + extract(hour from (end_interval_time-begin_interval_time))*60 + extract(day from (end_interval_time-begin_interval_time))*24*60) minutes,
executions_delta executions, round(ELAPSED_TIME_delta/1000000/greatest(executions_delta,1),4) "avg duration (sec)" from dba_hist_SQLSTAT a, dba_hist_snapshot b
where sql_id='&sql_id' and a.snap_id=b.snap_id
and a.instance_number=b.instance_number
order by snap_id desc, a.instance_number;





DECLARE
  l_sql_tune_task_id  VARCHAR2(100);
BEGIN
  l_sql_tune_task_id := DBMS_SQLTUNE.create_tuning_task (
                          begin_snap  => 1587,
                          end_snap    => 1589,
                          sql_id      => 'bpyxu7n89yqun',
                          scope       => DBMS_SQLTUNE.scope_comprehensive,
                          time_limit  => 60,
                          task_name   => 'bpyxu7n89yqun3_AWR_tuning_task',
                          description => 'Tuning task for statement bpyxu7n89yqun  in AWR');
  DBMS_OUTPUT.put_line('l_sql_tune_task_id: ' || l_sql_tune_task_id);
END;
/




EXEC DBMS_SQLTUNE.execute_tuning_task(task_name => 'bpyxu7n89yqun3_AWR_tuning_task');



spool bpyxu7n89yqun3_AWR_tuning_task.txt
SET LONG 10000000;
SET PAGESIZE 100000000
SET LINESIZE 200
SELECT DBMS_SQLTUNE.report_tuning_task('bpyxu7n89yqun3_AWR_tuning_task') AS recommendations FROM dual;



SELECT TASK_NAME, STATUS FROM DBA_ADVISOR_LOG WHERE TASK_NAME='bpyxu7n89yqun3_AWR_tuning_task';


execute dbms_sqltune.drop_tuning_task('bpyxu7n89yqun3_AWR_tuning_task');



                           ****************************நன்றி****************************

/tmp space not recovered after deleting files

My Environment:
DB - 12.1.0.1
APPS - R12.1.3
OS - IBM AIX 6.1

Normally conc.requests will access the temporary folder during concurrent process. If there is no space available (100% filled) in temporary location then all the requests will fail and the issue will be clearly addressed in the log file like no space available in tmp location. Normally it would be /usr/tmp, sometimes /var/tmp



In general deleting the files under temporary location will clear some space, but sometimes it wont happen. Even after you deleted the files, space will not be recovered and it will be still 100% filled. Reason for such situation is that the files are deleted but the process that access that file is still running and hence the space is logically occupied by this process.  Now killing the process is the only safer way and a quick way to fix the issue.

The best way is to use lsof command, get the pid of the process, do a small check and then kill the process using the pid you identified after you decided it is safe to kill. Now the space occupied by that particular process will be released.

Follow the below steps to achieve this.

Install the lsof rpm using the below command

rpm -Uvh <*.rpm>



Once the lsof rpm package is installed, use lsof command to find the details as follows 

/opt/freeware/sbin/lsof | grep /var/tmp 

Output of the above command may show some output like below: 
httpd 30027 oracle 57u REG 104,7 24803361 40 /var/tmp/filemQMbGZ.TMP (deleted) httpd 30033 oracle 75u REG 104,7 773833 44 /var/tmp/file7s8VAe.TMP (deleted) 
And you need to check which process is using the file. 
ps-ef|grep 30027 and than kill them from OS level 


httpd 30962 oracle 71u REG 104,7 72391 88 /var/tmp/filewzh7U1.TMP (deleted) 

kill -9 30027

Once the process is killed at OS level, the space will be freed, as a result conc.req will run normally.



****************************நன்றி****************************





Masking E-Mail address in Workflow Mailer of Oracle APPS R12.2


Masking mail id in WFM is nothing but null the value for the parameter 'EMAIL_ADDRESS' in two tables.

Below are the step by step process to do the same.

Stop the WFM services and perform the below steps to mask the email address.








Once completed, start the WFM services and proceed with the testing.

If email address needs to be unmasked, restore the table from the backup.


                          ****************************நன்றி****************************






'Could not lock request' during cancelling the concurrent request

Tried to cancel some long running requests in my environment and faced the issue 'Could not lock request'. Below are the steps I followed to fix this issue and to cancel those long running/hanging requests.




SELECT inst_id, DECODE(request,0,'Holder: ','Waiter: ')||sid sess, id1, id2, lmode, request, type FROM GV$LOCK  WHERE (id1, id2, type) IN (SELECT id1, id2, type FROM GV$LOCK WHERE request>0) ORDER BY id1, request;




Set Pages 1000 lines 1000
Set head on
Column Manager   Format A12
Column Request   Format 999999999
Column Program   Format A30
Column User_Name Format A15
Column Started   Format A15
Column FNDLIBR  Format A9
prompt Managers that is running a request and FNDLIBR PROCESS;
select substr(Concurrent_Queue_Name, 1, 12) Manager,
       Request_Id Request,
       User_name,
       Fpro.OS_PROCESS_ID "FNDLIBR",
       substr(Concurrent_Program_Name, 1, 35) Program,
       Status_code,
       To_Char(Actual_Start_Date, 'DD-MON-YY HH24:MI') Started
  from apps.Fnd_Concurrent_Queues    Fcq,
       apps.Fnd_Concurrent_Requests  Fcr,
       apps.Fnd_Concurrent_Programs  Fcp,
       apps.Fnd_User                 Fu,
       apps.Fnd_Concurrent_Processes Fpro
 where Phase_Code = 'R' And Status_Code <> 'W' And
       Fcr.Controlling_Manager = Concurrent_Process_Id and
       (Fcq.Concurrent_Queue_Id = Fpro.Concurrent_Queue_Id and
       Fcq.Application_Id = Fpro.Queue_Application_Id) and
       (Fcr.Concurrent_Program_Id = Fcp.Concurrent_Program_Id and
       Fcr.Program_Application_Id = Fcp.Application_Id) and
       Fcr.Requested_By = User_Id and
       Fcr.request_id =&request_id;





select ses.sid,
              ses.serial# serial#,
              proc.spid,
              ses.sql_id,
              ses.process,
              ses.last_call_et,
              ses.event
         from gv$session ses, gv$process proc
        where ses.paddr = proc.addr and ses.process in ('&process_ID');



                            ****************************நன்றி****************************

AC-50480: Internal error occurred: java.sql.SQLException: ORA-28112: failed to execute policy function during Autoconfig

My Environment

DB - 12.1.0.2.0
Apps - R12.2.5
Type - Single Instance

Both DB and Apps resides in the same box.

Issue:
While running autoconfig on both DB and APPS faced the below issue.

AC-50480: Internal error occurred: java.sql.SQLException: ORA-28112: failed to execute policy function


We checked the autoconfig log of both the tiers and found the similar issue issues recorded in both the log files.

DB Tier Autoconfig Log:

                   Generate Tns Names
##########################################################################
                Classpath                   : :/ebstst/oracle/SMOKETST/12.1.0/jdbc/lib/ojdbc6.jar:/ebstst/oracle/SMOKETST/12.1.0/appsutil/java/xmlparserv2.jar:/ebstst/oracle/SMOKETST/12.1.0/appsutil/java:/ebstst/oracle/SMOKETST/12.1.0/jlib/netcfg.jar:/ebstst/oracle/SMOKETST/12.1.0/jlib/ldapjclnt12.jar

Loading ORACLE_HOME environment from /ebstst/oracle/SMOKETST/12.1.0
Logfile: /ebstst/oracle/SMOKETST/12.1.0/appsutil/log/SMOKETST_r12smokedb/05111507/NetServiceHandler.log
AC-50480: Internal error occurred: java.sql.SQLException: ORA-28112: failed to execute policy function

Error generating tnsnames.ora from the database, temperory tnsnames.ora will be generated using templates
Instantiating templates
tnsnames.ora instantiated

WARNING: [CVM Error Report]
The following report lists errors encountered during CVM Phase
      <filename>  <return code where appropriate>
  /ebstst/oracle/SMOKETST/12.1.0/appsutil/bin/adgentns.pl  2

 No. of scripts failed in CVM phase: 1



WARNING: [AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>

  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /ebstst/oracle/SMOKETST/12.1.0/appsutil/install/SMOKETST_r12smokedb
      afdbprf.sh              INSTE8_PRF         1                                                                                            


Application Log:

Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.sql.SQLException: ORA-28112: failed to execute policy function
ORA-06512: at "APPS.FND_MESSAGE", line 536
ORA-06512: at "APPS.FND_MESSAGE", line 576
ORA-06512: at "APPS.FND_MESSAGE", line 665
ORA-06512: at "APPS.FND_MESSAGE", line 644
ORA-06512: at "APPS.FND_CP_FNDSM", line 218
ORA-06512: at "APPS.FNDSM", line 6
ORA-04088: error during execution of trigger 'APPS.FNDSM'
ORA-06512: at "APPS.FND_APP_SYSTEM", line 1368
ORA-06512: at "APPS.FND_NET_SERVICES", line 1734
ORA-06512: at line 1                                                                                                                               


With this issue we could not find much details on the reported issue and the solutions available also not worked for us. So we tail through the alert log to find if any record of this issue addressed there. While doing so, we noticed a particular trace file from the alert log and checked the trace file which actually contains the error/cause for this issue.

Content of the trace file

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options
ORACLE_HOME = /ebstst/oracle/SMOKETST/12.1.0
System name:    Linux
Node name:      r12smokedb.sify.net
Release:        4.1.12-37.4.1.el6uek.x86_64
Version:        #2 SMP Tue May 17 07:23:38 PDT 2016
Machine:        x86_64
Instance name: SMOKETST
Redo thread mounted by this instance: 1
Oracle process number: 53
Unix process pid: 44187, image: oracle@r12smokedb.sify.net


*** 2019-05-11 22:43:37.967
*** SESSION ID:(47.53264) 2019-05-11 22:43:37.967
*** CLIENT ID:() 2019-05-11 22:43:37.967
*** SERVICE NAME:(SYS$USERS) 2019-05-11 22:43:37.967
*** MODULE NAME:(JDBC Thin Client) 2019-05-11 22:43:37.967
*** CLIENT DRIVER:(jdbcthin) 2019-05-11 22:43:37.967
*** ACTION NAME:() 2019-05-11 22:43:37.967

Incident 14146 created, dump file: /ebstst/oracle/SMOKETST/12.1.0/admin/SMOKETST_r12smokedb/diag/rdbms/smoketst/SMOKETST/incident/incdir_14146/SMOKETST_ora_44187_i14146
.trc
ORA-00600: internal error code, arguments: [pesldl03_MMap: errno 1 errmsg Operation not permitted
], [], [], [], [], [], [], [], [], [], [], []

*********START PLSQL RUNTIME DUMP************
***Got internal error Exception caught in pl/sql run-time while running PLSQL***
***Got ORA-600 while running PLSQL***
ANONYMOUS BLOCK:
library unit=c1922a70 line=1 opcode=191 static link=0 scope=0
FP=0x7fa44003af30 PC=0xc2825288 Page=0 AP=(nil) ST=0x7fa44003c000
DL0=0x7fa43f64ffe0 GF=0x7fa43f650038 DL1=0x7fa43f650000 DPF=0x7fa43f650028
HS=0xc28253f8 AR=0x7fa43f64ff98 DS=0xc28253c8
PB_PC=(nil) SV_PC=(nil)
DS pkg desc :
0C28252D0                   02920314 00000050          [....P...]
0C28252E0 00020004 00000000 00000000 00000001  [................]
0C28252F0 00C80000 00000005 00E80000 00000204  [................]
0C2825300 00000000 00000000 00000000 00000000  [................]                                                     


ORA-00600: internal error code, arguments: [pesldl03_MMap: errno 1 errmsg Operation not permitted. -à This particular issue leads us to the document Doc ID : 1625010.1 which provides the solution.


Cause:
If /dev/shm is mounted with "noexec" such an error occurs.


Solution for the error :


After performing this change, autoconfig completed successfully in both the tiers.


Ref : ORA-600 [pesldl03_MMap: Errno 1 Errmsg Operation Not Permitted] (Doc ID 1625010.1)


                       ****************************நன்றி****************************

Find Keystore password in Oracle Apps R12.2


To identify the existing keystore and key passwords run the following SQL script connected as the APPS user:

SQL> set serveroutput on
declare
spass varchar2(30);
kpass varchar2(30);
begin
ad_jar.get_jripasswords(spass, kpass);
dbms_output.put_line(spass);
dbms_output.put_line(kpass);
end;
/

This will output the passwords in the following order:

store password (spass)
key password (kpass) 




                           ****************************நன்றி****************************

IAS Cache initialization failed

 Today I faced an Issue in R12.2 instance. The solution I followed to overcome the issue is very simple, but they are more than one solution...