Troubleshooting adop Cutover Hung or Port Conflicts in Oracle EBS R12.2

-

 When applying CPU (Critical Patch Update) patches or standard patch sets in Oracle E-Business Suite (EBS) R12.2, the Online Patching Utility (adop) relies on a strict sequence of execution phases: prepare, apply, finalize, cutover, and cleanup.


Among these, the cutover phase is the most critical operational event. It performs the actual switchover between the RUN and PATCH file systems (fs1 and fs2), bringing down the application services, swapping the dual file system mount points/environment variables, and bringing the new application services back online.


A common real-world issue DBAs face during cutover is an unexpected hang where adop appears stuck while shutting down internal managers or service controllers. In this post, we will walk through a live diagnostic scenario during a July 2026 CPU Patching cycle, analyze why orphaned Mobile Supply Chain Applications / Mobile Web Applications (MSCA/MWA) processes block adop, dive deep into Linux process hierarchies (PPID=1), and resolve the issue safely.


1. The Scenario: adop Cutover Hanging on Internal Managers

During the July 2026 CPU Patching on an EBS R12.2 environment (TESTR122), the cutover phase was initiated:

adop phase=cutover


After running for about 10 to 15 minutes, the process appeared to freeze while waiting for the Internal Concurrent Manager (ICM) and related services to go down. It made minor progress, then stalled completely.


+---------------------------------------------------------------------------+

| Online Patching Utility (adop) - Cutover Phase    |

+---------------------------------------------------------------------------+

Waiting for Internal Manager to stop...

[STUCK] No progression for extended duration...


Initial Triage

1. OS Zombie/Orphaned Process Check: A basic check for standard zombie or defunct OS processes yielded no immediate leads.

2. Log File Inspection: To find the exact blocking step, navigate to the active adop session log directory under the non-editioned file system (fs_ne):


cd /u01/oracle/TESTR122/fs_ne/EBSapps/log/adop/76/20260726_065014/cutover/<Hostname>/TXK_CTRL_forceshutdown

tail -100f txkADOPCutOverPhaseCtrlScript.log



The Root Cause Log Output

The log revealed a continuous retry loop inside the verifyPorts() function:


text

================================

Inside verifyPorts()...

================================

ERROR: MSCA/MWA Telnet Server Port 10562 is not free

ERROR: MSCA/MWA Telnet Server Port 10560 is not free

ERROR: MSCA/MWA Telnet Server Port 10564 is not free

ERROR: Some of the ports which are common across RUN and PATCH fs are not yet free.

Waiting for 1 minute before rechecking.



2. Technical Deep-Dive: Port Verification and File System Duality

In Oracle EBS R12.2, the application tier utilizes a dual file system architecture:


  • RUN File System: Actively serves user traffic.
  • PATCH File System: Updated during patching cycles.

While both file systems contain separate directories and binaries, certain service ports are shared across RUN and PATCH file systems.



+-----------------------------------------------------------------------+
|                         Dual File System Switch                       |
|                                                                       |
|   RUN File System (fs2)                   PATCH File System (fs1)     |
|   +-----------------------+               +-----------------------+   |
|   | MSCA/MWA Telnet       |               | MSCA/MWA Telnet       |   |
|   | Ports: 10560, 10562...|               | Ports: Shared         |   |
|   +-----------+-----------+               +-----------+-----------+   |
|               |                                       |               |
|               +--------------> [ CUTOVER ] <----------+               |
|                                     |                                 |
|                       Requires ports 10560, 10562                     |
|                       to be completely RELEASED                       |
+-----------------------------------------------------------------------+


Before adop can flip fs2 to fs1, it calls adstpall.sh or specific control scripts (like txkADOPCutOverPhaseCtrlScript.log) to shut down every application process listening on these shared ports.

If any process continues to bind to these ports, adop pauses and loops indefinitely—rechecking every 60 seconds.


3. Investigating Process Ownership at the OS Level

To identify which processes were holding ports 10560, 10562, and 10564, we ran ss (Socket Statistics) and ps commands on the host server.


Step 1: Identify Bound Sockets

ss -tlnp | grep -E '10560|10562|10564'


Output:

text

LISTEN 0      50                           *:10564            *:*    users:(("java",pid=2218595,fd=571))

LISTEN 0      50                           *:10562            *:*    users:(("java",pid=2218573,fd=571))

LISTEN 0      50                           *:10560            *:*    users:(("java",pid=2218559,fd=571))



Step 2: Inspect Process Details

ps -ef | grep -i telnet


Output:

text

oracle   2218559       1  0 Jun15 ?        00:23:40 /u01/oracle/TESTR122/fs2/EBSapps/comn/util/jdk64/bin/java ... oracle.apps.mwa.presentation.telnet.Listener 10560

oracle   2218573       1  0 Jun15 ?        00:23:45 /u01/oracle/TESTR122/fs2/EBSapps/comn/util/jdk64/bin/java ... oracle.apps.mwa.presentation.telnet.Listener 10562

oracle   2218595       1  0 Jun15 ?        00:23:47 /u01/oracle/TESTR122/fs2/EBSapps/comn/util/jdk64/bin/java ... oracle.apps.mwa.presentation.telnet.Listener 10564


Key Analytical Findings:

1. Source File System: The path (/u01/oracle/TESTR122/fs2/...) confirms these processes belonged to the active fs2 RUN file system.

2. Start Time: The processes had been running since Jun 15.

3. Parent PID (PPID): All three processes showed PPID = 1.


4. Understanding Process Adoption: What Does PPID = 1 Mean?

Every process running on a Linux system possesses a Process ID (PID) and a Parent Process ID (PPID). PID 1 is assigned to the operating system's primary initialization process (e.g., systemd or init).


NORMAL EXECUTION HIERARCHY:

systemd (PID 1)

   └── adstrtal.sh

          └── mscactl.sh / mwactl.sh

                 └── java oracle.apps.mwa.presentation.telnet.Listener (PPID = mscactl PID)


ORPHANED EXECUTION HIERARCHY:

systemd (PID 1)

   └── java oracle.apps.mwa.presentation.telnet.Listener (PPID = 1) [ORPHANED]


How Processes Become Orphaned

1. When MSCA/MWA services are started normally via mwactl.sh or adstrtal.sh, the service script acts as the parent.

2. If the parent script exits abnormally, is forcibly closed, or if a user executes a service directly from an interactive SSH terminal session that disconnects, the parent process dies.

3. Linux re-parents these "orphaned" child processes directly to PID 1 (systemd/init) so they are not left without a controller.


Why adop / mwactl.sh Failed to Stop Them

When adop executes adstpall.sh during cutover, service shutdown scripts rely on standard process tracking (such as PID file matching or parent-child process tree signals). Because these Java processes were orphaned under PID 1, standard control scripts (mscactl.sh / mwactl.sh) could not signal or locate them. They were essentially invisible to standard shutdown controls while remaining bound to ports 10560, 10562, and 10564.


5. Resolution Step-by-Step

Since standard shutdown scripts could not reach these orphaned daemons, manual intervention was required.


Step 1: Terminate the Orphaned Processes Gracefully

Use kill -15 (SIGTERM) first to give the Java processes a chance to release socket bindings cleanly:

kill -15 2218559 2218573 2218595

Important Operational Note: Avoid jumping straight to kill -9 (SIGKILL) on network listening Java processes unless necessary. Ungraceful termination can occasionally leave file sockets or TCP states stuck in FIN_WAIT_2 or TIME_WAIT, which can keep the port locked until an OS-level timeout or server reboot occurs.


Step 2: Observe adop Auto-Recovery

Because adop runs an internal retry loop every 60 seconds, you do not need to restart adop or kill the cutover process. Within seconds of terminating the processes, txkADOPCutOverPhaseCtrlScript.log automatically resumed:


================================

Inside verifyPorts()...

================================


All the common ports are free.

The cutover phase proceeded automatically to complete the file system switchover without requiring any adop phase=cutover re-execution.


6. Post-Cutover Actions & Best Practices

After the cutover phase completes successfully:

1. Service Verification: When starting services on the new RUN file system (fs1), run adstrtal.sh. Standard EBS startup routines will initialize the new MSCA/MWA Telnet listeners on the new RUN environment automatically.

2. Manual Intervention Check: Only invoke mscactl.sh / mwactl.sh manually if MSCA/MWA services fail to start after the full stack startup sequence completes.

3. Core Takeaway for EBS DBAs: Any EBS application process showing PPID = 1 during a patching window will bypass adstpall.sh routines. Common culprits include:

  • MSCA/MWA Telnet Listeners
  • Workflow Mailer Java processes
  • Standalone OAM/Custom data collection utilities executed from interactive shells

Pre-checking for PPID = 1 application processes prior to triggering adop phase=cutover can save significant downtime during scheduled maintenance windows. 


****************************நன்றி****************************

Comments

Post a Comment

Popular posts from this blog

Common R12.2 adcfgclone Issues

-
ERROR: Failed to configure the target system – Failed to enable EBS_LOGON trigger The clone procedure for EBS R12.2 is quite different when compared with EBS R12.1. This post covers very generic   issues we may encounter while configuring the target server as a part of clone Process. ERROR : 1 Checking the port pool 6 done: Port Pool 6 is free Report file located at /u01/appl_PROD/PROD/fs2/inst/apps/PRODERP_erpnode3/admin/out/portpool.lst The new APPL_TOP context file has been created : /u01/appl_PROD/PROD/fs2/inst/apps/PRODERP_erpnode3/appl/admin/PRODERP_erpnode3.xml Check Clone Context logfile /u01/appl_PROD/PROD/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0811105848.log for details. FMW Pre-requisite check log file location : /u01/appl_PROD/PROD/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log Running: FMW pre-req check... ERROR: Run file system configuration has failed Check the logfile /u01/appl_PROD/PROD/fs1/inst/apps/PRODERP_erpnode3/admin/l...
Read more

REP-3000: Internal error starting Oracle Toolkit

-
Image
REP-3000 Issue is very common in Oracle Apps environments. If this issue persists, concurrent programs will complete with error code(Whose output is in PDF format). Generally this affects the OPP operations. I've faced this issue in both Linux and Unix platforms. The reason for this issue is common on both the platforms. Improper setting of the DISPLAY variable is the reason for this issue. In my environment, I recently faced this issue in EBS 11i Instance which runs on RHEL-5.11. Below are the detailed step by step process I followed to overcome this issue. When submitted 'Active Users' report it completed with error. When checked the log file found the below issue. Here it is understood, the issue is due to improper DISPLAY variable setting. So I start to work on setting the DISPLAY variable properly. In my server, display variable is set to value "localhost:10.0". Now I need to check whether this value is proper and can open xclock and windows using this Displa...
Read more

Error 404 -- Not Found From RFC 2068 hypertext Transfer Protocol -- HTTP/1.1

-
Image
In my environment we are having EBS(12.2.5) running with 12.1.0.2 database. One day in our DEV Instance  Front-End is not opening and faced the below error. So I followed the below steps as a solution and it worked. First, generate the JAR files Next, reload the generated jar files to the respective database  Now compile the APPS schema to reduce the count of Invalid objects. Once everything completed, you can clear your browser cache and try to login to the application., but normally the above steps will be accompanied by clearing the middle tier cache. So I gave the below steps to clear the middle tier cache from backend. To clear the cache, follow the below steps. Stop the application services Navigate to $OA_HTML/cabo/images/cache Take a backup of contents of the folder cache and then clear all the contents inside that cache folder Navigate to $OA_HTML/cabo/styles/cache Take a backup of contents of the folder ca...
Read more