Friday, June 18, 2010

Unable to start OACORE OC4J in R12

We were doing JRE upgrade today and had issue in one of the instance not coming up. Basically OACORE OC4J service was unable to get start. Script adstrtal.sh was failing to start this service and erroing out:
==================================================
Executing service control script:/Path/admin/scripts/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance ...
****************************************************
===================================================

After looking at log file :
===================================================
06/18/10-01:32:04 :: adoacorectl.sh: Starting OPMN managed OACORE OC4J instance
opmnctl: starting opmn managed processes...
================================================================================opmn id=tsgsd2107:6204
0 of 1 processes started.
ias-instance id=XXXXXXXXXXXXX.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/oacore/default_group/
Error
--> Process (index=1,uid=237255567,pid=24554)
failed to start a managed process after the maximum retry limit
Log:
/Path/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
06/18/10-01:36:43 :: adoacorectl.sh version 120.13
06/18/10-01:36:43 :: adoacorectl.sh: Stopping OPMN managed OACORE OC4J instance
==============================================
OPMN log says this:

==============================================
10/06/18 01:35:32 [libopmnoc4j] Host and Port information for port type jms not sent by the OC4J process
10/06/18 01:35:32 [libopmnoc4j] WARNING: OC4J did not send the protocol value for port id: jms
10/06/18 01:35:32 [libopmnoc4j] Port information in the ONS notification is incorrect for proc: 237255567.
Some of the reasons for the failure are:
1. ajp, rmi and jms ports could not be bound by the process.
2. If a port value for a certain port ID is handed by OPMN to the OC4J process and this value is not returned back in the ONS notifications.
For example, this can happen for certain bad JGroups related configurations.
10/06/18 01:35:32 [libopmnoc4j] Process Start Error: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [pm-process] Stopping Process: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [libopmnoc4j] Process Stop Error: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [libopmnoc4j] Failed to construct stop command for proc: 237255567
10/06/18 01:35:34 [libopmnoc4j] Forcefully Terminating Process: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:36 [pm-requests] Request 7 Completed. Command: /start?process-type=oacore
=======================================

Now big thing is solution. Here you go:
Go to $INST_TOP/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1 on application tier and look for jms.state and *.lock files. If you find them in this location once all you services are stop. Delete or rename them. Once you are done try again. Now OACORE OC4J service will come up.

Happy Troubleshooting !!!