Thursday, January 19, 2017

Adpatch Issues - Worker failed, Network failure and Pre-requisite patch missing

Common Adpatch issues:

While we apply patch we may get multiple issues. Here I discussed some of the most common issues 
  1. Worker failed patching
  2. Network failure
  3. Pre-Requisite patch is missing

Whenever adpatch gives error and stops patching because some reason. First check the status of worker with ADCTRL  utility and then go through your adpatch log and get some hint why patch is failed, then try to fix accordingly. Each issue has specific approach for fixing it.

Worker failed patching:
While we check the adpacth log at location PPL_TOP/ADMIN/SID/LOG/adpatch.log and we notice that adpatch failed because of worker has failed patching. We must immediately check the worker log at location APPL_TOP/ADMIN/SID/LOG/adworkxxx.log and check for the issue and fix it.

Once issue is fixed then follow below steps:
  1. Login with APPLMGR user
  2. Source the environment
  3. Start adpatch session again and will ask for continuation of previous session
  4. Say yes
  5. It will continue patching with previous session only where it was left.

.rf9 is the file which helps the adpatch to determine where patch was left patching. Patching information is held into tables AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES.

Network Issue:
While we are applying application patch and network is failed in between. In this case we have to apply the patch again with below steps:
  1. Login with APPLMGR user
  2. Source the environment
  3. adpacth - It will ask for continue to previous session
  4. Say yes

If still patching is not working, Please check the log at location $APPL_TOP/ADMIN/SID/LOG/adpatch.log and sometimes it shows some autopatch error. In this case, we must check the worker status with ADCTRL utility with option 1. It shows as worker is still under running.
This is the actual problem and will stop the worker with ADCTRL utility selecting option 3 as below:

AD Controller Menu
---------------------------------------------------

1. Show worker status

2. Tell worker to restart a failed job

3. Tell worker to quit

4. Tell manager that a worker failed its job

5. Tell manager that a worker acknowledges quit

6. Restart a worker on the current machine

7. Exit

But if you check the status of worker again with ADCTRL utility, It shows as worker is still running. Problem is still not resolved and for resolving the issues to continue with patching worker status should change to failed.
So in this case will get into ADCTRL utility menu and opt for option 4: Tell manager that a worker failed its job. Now if you check the worker status, it will change to failed.
You can start adpatch again and continue with session to patch where patch had stopped working.

Pre-Requisite patch is missing:
if patch is failed because of pre-requisite patch missing. In this case , we need to take the backup of AD_DEFERRED_JOBS, FND_INSTALLED_PROCESSES and restart directory(containing .rf9 file). Location of restart directory is $APPL_TOP/ADMIN/SID/restart.
Drop these tables and start with new session and apply the pre-requisite patch and check the log.
One pre-requisite patch is complete, recover the backed up tables and restart directory start adpatch session and continue with previous session.
Steps to apply pre-requisite patch:
1.      Stop the worker with ADCTRL utility.
2.      Backup FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables.
3.      Backup the .rf9 file located at $APPL_TOP/admin/<SID>/restart directory.
4.      Drop FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables.
5.       Apply the pre-requisite patch.
6.      Restore the backed up .rf9 file or restart directory.
7.      Restore the FND_INSTALL_PROCESSES table.
8.      Restore the AD_DEFERRED_JOBS table.
9.      Start adpatch, it will resume where it stopped previously.

What are the tables created while we start adpatch session?
AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES tables are created.

What does AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES tables contain?
FND_INSTALLED_PROCESSES: adpacth creates FND_INSTALL_PROCESSES table, assigns each worker unique ID with new row for each worker. This table is used to store the information about the job for each worker. Once all jobs are completed, worker shuts down and drops FND_INSTALL_PROCESSES table.

AD_DEFERRED_JOBS: This table holds the information of failed (workers which has failed its job). This table is created on starting the session of adpatch utility and drops once it is complete.

From which table ADCTRL gathers information?
FND_INSTALL_PROCESSES

How to use adctrl utility to restart failed worker in patch application

In this example I will demonstrate you how 'adctrl' utility is used for checking the status of workers or restarting workers while applying patches.
1. Open a new terminal window and setup environment to run scripts on Application Tier 
2. Start the adctrl utility
cd $AD_TOP/bin
adctrl
If you don't know what is the directory for $AD_TOP then use the following:
cd /u01/apps/[hostname]/apps/apps_st/appl/ad/12.0.0/bin
adctrl
          It will ask you to provide couple of details such as APPS user DB password, APPL_TOP directory etc.
3. Once you give above information, adctrl utility will start and following menu will appear:
AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit
4. Use option 6 and then option 2 to restart a failed job
5. Now to to the terminal screen where 'adpatch' utility was running. You will see messages that 'worker                 restarted'. And if you have fixed the error successfully, adctrl will resume the patch application.        

How to see the status of a worker?
At any time you can see the status of all workers run by 'adpatch' utility. To see the status:
1. Follow the steps described above (from step 1 to 3).
2. Choose option 1. All worker statuses (with the jobs they were executing or failed) will be shown on the terminal window.

Assumptions
  • Database is up and running
  • Patch application has started with adpatch utility
  • adpatch has thrown an error in a job with a worker
Environment
  • Oracle E Business Suite R12i
  • RHEL 4.5

IAS Cache initialization failed

 Today I faced an Issue in R12.2 instance. The solution I followed to overcome the issue is very simple, but they are more than one solution...