Concurrent Managers not Starting

Issue 

In one of our PROD environment, concurrent manager stopped working all of a sudden. No recent changes were made. We tried bouncing the CM, no help. We take a bounce on MT services, still the issue is not solved.

Troubleshooting

Checked the CM status from backend. All the managers are down, except a few.

From the above it is evident that Internal Manager is not down. Even when we bounce the CM, it doesn't go down. So decided to drill down on Internal Manager. As a first step, checked the log of this particular manager.

Generally it seems that the manager is still active but from the log it is clear that it has been terminated.

Now the issue is clear, since it cannot go down, it cannot be restarted, hence all the managers are down.

So decided to run the CM Recovery. Below are the step by step process to run CM Recovery in R12.2. The same can be followed for R12.1 as well.

Running CM Recovery Wizard is an approach to clear the errors upon bringing back the Internal manager up.

Note: While running CM Recovery, CM service should be down.

In the above SS, you can see three runaway processes are there causing this trouble. 

What is a runaway process?

A process which is been cancelled, doesn't close its session and release its associated memory is called a runaway process. This runaway process will become the blocking session.

Suppose if you're cancelling a CM request, then check the process is cleared or not by using the below queries.

select request_id,oracle_process_id,os_process_id from fnd_concurrent_requests where request_id=’&Req_Id’;

select p.spid , s.sid , s.serial# from v$session s , v$process p where s.paddr = p.addr and s.process = &os_process_id ;

alter system kill session ‘session-id,session-serial’;

So before update, crosscheck for any CM related process still running at OS level. If so kill them manually using the command 'kill -9 <pid>;' 

After killed the existing process/if no such process exists, check at OS level and then click on update in step 2 and proceed with Next.

Here all the processes are cleared.

To get more info on the process, check the above mentioned log files.

Once the CM Recovery is completed without any issues, start the CM and check. 

Once all the managers are UP and Running, submit a sample request and check the completion status before updating the business.

Ref: Concurrent Manager Recovery Wizard - Oracle Applications Manager Troubleshooting and Diagnostics (Doc ID 2130545.1)

****************************நன்றி****************************