IAS Cache initialization failed

 Today I faced an Issue in R12.2 instance. The solution I followed to overcome the issue is very simple, but they are more than one solution to it based on the case. So I thought to make a post on this issue.

Issue Faced

After I entered creds to login to my R12.2, faced the below screen. On checking the log found the below issue.

From the issue, I understood something is not correct with the port 12345. So I checked metalink and found two documents talks about the same issue. So I followed the test cases described there and the results from my environment are same as the results of oracle document.

 In my case the instance where I'm facing the issue is not a recently cloned one. It was running fine for the past few months and no recent changes were done either at Instance or Network or Firewall level. So I came to a conclusion that this issue is not related to port. 

So I decided to take a bounce of my managed servers (OA core) alone. After the bounce, behavior of my instance became normal and the issue is resolved. In my case the solution is very simple, but for some of few it may not be so. So go through the oracle documents mentioned in the reference section.

Reference

Unable to access login page - Error "The Distributed Caching System failed to initialize on port<port number>" (Doc ID 2370266.1)

E-Business Suite Applications Technology Stack HTTP / IAS Server Exception In Static Block Of jtf.cache.CacheManager. Stack Trace Is: oracle.apps.jtf.base.resources.FrameworkException : IAS Cache Initialization Failed (Doc ID 1484840.1)

****************************நன்றி****************************

REP-3000: Internal error starting Oracle Toolkit

REP-3000 Issue is very common in Oracle Apps environments. If this issue persists, concurrent programs will complete with error code(Whose output is in PDF format). Generally this affects the OPP operations.

I've faced this issue in both Linux and Unix platforms. The reason for this issue is common on both the platforms.

Improper setting of the DISPLAY variable is the reason for this issue.

In my environment, I recently faced this issue in EBS 11i Instance which runs on RHEL-5.11.

Below are the detailed step by step process I followed to overcome this issue.

When submitted 'Active Users' report it completed with error. When checked the log file found the below issue.

Here it is understood, the issue is due to improper DISPLAY variable setting. So I start to work on setting the DISPLAY variable properly.

In my server, display variable is set to value "localhost:10.0". Now I need to check whether this value is proper and can open xclock and windows using this Display variable. I can open all the needed things with this value which means the current DISPLAY variable's value is a proper and usable one.

 Now I need to check what value the variable DISPLAY point to from my EBS Application. Run "Prints environment variable values" for DISPLAY variable and check the value

My Application environment points to a different value for this variable. This is the reason for the issue. Now I need to correct this in-order to overcome this issue.

1. From your EBS Application, open the seeded report "Active Users" - Change the CONCURRENT > PROGRAM > DEFINE > ACTIVE USERS.
2. Change the Output Type to Postscript and save.
3. Now edit the uiprint.txt file. This file will be located under the location $ORACLE_HOME/guicommon6/tk60/admin. 
4. Now add the following entry: <your postscript printername>:PostScript:1:5th FLOOR:default.ppd                          Note: I skipped this 4th step.
5. Now shutdown the APPS services.
6. Change the value of DISPLAY variable in your APPS context file.
7. Post changes, run autoconfig on APPS environment to re-generate the application configuration files.
8. Now the edit the environment file located under $APPL_TOP and add the below lines                 #Begin customizations
   DISPLAY="<host server name>:0.0" --> In my case it is "localhost:10.0"
   export DISPLAY
   # End customizations
9. Now start the APPS services
10. Run "Prints environment variable values" for DISPLAY variable and check the value has been updated or not.
    
    
    
11. Run "CP Postscript Report Regression Test" with parameter BASIC(default). This report should complete successfully. If completed, the issue is resolved. If not you need to find some other way to make it work.
    
    
    
12. From any Java page, Navigate to HELP > DIAGNOSTICS > EXAMINE. Click on LOV for block and choose '$ENVIRONMENT$', for Field 'DISPLAY', then click on Value. You should see the updated display value there.
    
    
    
13. After all this run the "Active Users" report and this time it'll complete successfully.
    

Note: If still the "Active Users" completed with the same error, start a new putty session enable X11 forwarding and run the command xclock. If the clock is displayed, leave the putty session as it is and then re-submit the Active Users program. This time it'll complete successfully. This is from my Real Time experience. This should help you.

References: 

• Troubleshooting Tips For REP-3000 Error when Running E-Business Suite PDF/POSTSCRIPT/HTML Reports (Doc ID 207532.1)
• http://oracle4ryou.blogspot.com/2012/09/rep-3000-oracle-toolkit-error.html#:~:text=In%20the%20main%20environment%20file,and%20available%20X%20display%20server.&text=Typically%2C%20this%20action%20needs%20to,X%20display%20server%20is%20bounced.

****************************நன்றி****************************

Decrypting Weblogic Password

Sometimes we may forget the password we set for weblogic. At such times this method will be very much helpful.  

Below is the step by step process to decrypt the password for weblogic.

1. Take a backup of the boot.properties file
   
   
   
2. Note the value of encrypted password
   
   
   
3. Now invoke the WebLogic scripting tool                                                                                            sh wlst.sh
   
   
   
4. Pass the values for Domain, service, encryption and password                                                  wls:/offline> domain = "<Domain Value>"

   wls:/offline> service = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domain)

   wls:/offline> encryption = weblogic.security.internal.encryption.ClearOrEncryptedService(service)

   wls:/offline> print encryption.decrypt("<Encrypted Password>")

   
   
   
5. Your password will be decrypted
6. Now you can use the decrypted password to start the Admin server
   
   
   

****************************நன்றி****************************

Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database does not match that of the oracle executable - USER ( APPS ) has been detected in FND_WEB_SEC.GET_OP_VALUE.

After completed cloning of EBS-12.2.4, working on the post clone activities. When tried to change the APPS and SYSADMIN passwords using FNDCPASS utility faced the below issue.

This is the first time I'm facing this issue. I can't understand why this issue has occurred as I didn't face any issues during cloning. Below is the step by step process to overcome this issue.

Create two scripts 'rmcorejvm.sql' & 'corejvminst.sql' and execute them on the DB Tier.

Content of rmcorejvm.sql

connect / as sysdba

set echo on

set serveroutput on

select * from v$instance;

select owner, status, count(*) from all_objects

where object_type like '%JAVA%' group by owner, status;

execute rmjvm.run(false);

shutdown immediate

set echo off

spool off

exit

Content of corejvminst.sql

set serveroutput on

set echo on

startup mount

alter system set "_system_trig_enabled" = false scope=memory;

alter database open;

select owner, status, count(*) from all_objects

where object_type like '%JAVA%' group by owner, status;

create or replace java system

/

shutdown immediate

set echo off

spool off

exit

Now run the scripts in the below mentioned order at DB Tier via sqlplus as sysdba user.

Now mount the database, change the parameter _system_trig_enabled to false and take a bounce on the DB.

Once the DB is started, check the below value and proceed to change the apps/sysadmin passowrds.

This time I could change the passwords without any issues.

****************************நன்றி****************************

Enabling EM Express Console in OCI Environment

By default EM Express Console is not enabled in 18.1.0.0, 12.2.0.1 and 12.1.0.2 databases.  Below is the step by step process to enable EM Express Console in Oracle Database.

In my environment, I had enabled the EM Console at PDB level.

Before enabling the console listener status would be

To enable the console and set its port

Now confirm the listener is listening on the port '5500'.

Now to open the port '5500' on DB system, follow the below steps

Once the DB port is opened, update your security list in OCI console. To do this on an existing security list, follow the below steps.

1. Open the navigation menu. Under Oracle Database, click (Bare Metal / VM / Exadata) ---> as per your environment

2. Choose your Compartment.

   A list of DB systems is displayed.

3. Locate the DB system in the list.
4. Note the DB system's Subnet name and click its Virtual Cloud Network.
5. Locate the subnet in the list, and then click its security list under Security Lists.

6. Click Edit All Rules and add an ingress rule with source type = CIDR, source CIDR=<source CIDR>, protocol=TCP, and port=<port number or port range>.

   The source CIDR should be the CIDR block that includes the ports you open for the client connection.

Known Issue:

You might get a "Secure Connection Failed" error message when tried to connect to the EM Express Console from your 1 node DB system because the correct permissions were not applied automatically. To overcome this issue follow the below steps.

Now copy the location of the wallet directory, navigate there and follow the below steps.

Ref: Monitoring a Database (oracle.com)

        Known Issues (oracle.com)

****************************நன்றி****************************

PL/SQL: Statement Ignored PLS-00201: identifier 'DBMS_OUTPUT' must be declared

Recently I migrated a database(11gR1) from Windows Platform to 12C on Linux Platform. Migration was successful. During the sanity check, found lot of Invalid objects even after all the grants and privileges were provided. Ran utlrp.sql couple times, still the invalids count remains the same.

So I choose an Invalid object, compiled it manually and faced the below issue.

This was the issue for almost all the invalid objects. So I understood something is not right with the object DBMS_OUTPUT. So I checked this object and to my surprise it was valid.

Since the status is Valid, before doing any change to the object 'DBMS_OUTPUT', I decided to check the coding of the invalid object, since the DBMS_OUTPUT is an oracle standard one.

If I comment the line "dbms_output.put_line" from the invalid object, it is getting compiled and then it is a valid one but this is not the solution. This can be accepted as an workaround when a minimal no.of objects becomes invalid due to this issue. In my case, the invalids count is 226 and it's a humungous task to comment this line in all the coding of those objects. So something must be fishy with the DBMS_OUTPUT object.

Since I know DBMS_OUTPUT is oracle's standard object, I decided to read some oracle documents to get a solution for this. Luckily I got what I looked for.

I ran the script 'dbmsotpt.sql' as SYS user and bingo!!!

Compiled an invalid object manually, and it got compile without any issues.

I ran utlrp.sql to compile all the objects, post which the count got reduced form 226 to 8.

Finally all my issues are resolved.

Ref: https://docs.oracle.com/database/121/ARPLS/d_output.htm#ARPLS67301

****************************நன்றி****************************

SEND_MAIL - ORA-01775: looping chain of synonyms

 In one of our environment, post migration we tested the SEND_MAIL procedure to initiate the mail via SMTP server from DB level and faced the below issue.

ORA-01775: looping chain of synonyms

Looping chain of synonyms means, the created synonyms points to another object in a circle.

Here for better understanding,

1. SEND_MAIL is the procedure used to trigger mails from DB Server
2. UTL_SMTP and DBMS_OUTPUT are the dependent objects for SEND_MAIL procedure
3. UTL_SMTP synonym is also dependent object for SEND_MAIL procedure

So it is clearly evident that the dependent objects should be valid before fixing the issues with the SEND_MAIL procedure.

So, I started the drilldown on the required dependent objects and found the procedure 'SEND_MAIL' and package body of the object 'UTL_SMTP' is invalid.

So I made an attempt to compile the UTL_SMTP body to make it a valid object and faced the below issue.

This issue indicates the format of the wrapped unit being compiled is not understood by the compiler. This may be because the unit was edited or modified after it was wrapped. In our case this is what exactly I did. I copied this packages from some other database and compiled it in this database which leads to this issue. So now the ideal solution would be create this package from this database itself. Also since this UTL_SMTP is a standard package it should remain under SYS schema's ownership. In my case it was under the ownership of custom user 'TFGADMIN'.

So I dropped the package and created it separately in this database as follows.

After I dropped, checked and found only the synonym is available and this is the required result.

So to create the package, run the below scripts as SYSDBA.

After I ran the scripts, found the packages created under the SYS ownership and the status is Valid.

Now the next step is to compile the SEND_MAIL procedure as it was invalid since the dependent objects were Invalid. Now the dependencies are valid, so I compiled the procedure and it becomes a valid object.

Post this, run the below given ACL to complete the mail sending process. 

grant execute on utl_http to TFGADMIN;

BEGIN DBMS_NETWORK_ACL_ADMIN.CREATE_ACL(acl => 'ACL_TEST3.xml', description => 'ACL FOR TEST PURPOSES', principal => 'TFGADMIN' ,  is_grant => true,

privilege => 'connect', start_date => null,end_date => null );COMMIT; END;

SELECT DECODE( DBMS_NETWORK_ACL_ADMIN.check_privilege('ACL_TEST5.xml', 'TFGADMIN', 'connect'),1, 'GRANTED', 0, 'DENIED', NULL) privilege FROM dual;

BEGIN DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(acl => 'ACL_TEST5.xml', principal => 'TFGADMIN', is_grant => true, privilege => 'connect', position => 1);COMMIT;END;

BEGIN DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL (acl => 'ACL_TEST5.xml', host => '172.17.104.72', lower_port=> null, upper_port => null);COMMIT;END;

Once the ACL is created, I ran the SEND_MAIL procedure and the mail is triggered and the same is received in my mailbox.

****************************நன்றி****************************