Creating Self-Signed SSL Certificates for Siebel Innovation Pack 2017

APPLIES TO:
Siebel CRM - Version 17.0 [IP2017] to 17.0 [IP2017] [Release V17]
Information in this document applies to any platform.

PURPOSE
This document describes how to quickly create and configure a Secure Sockets Layer (SSL) certificate and certificate authority for use with Siebel Innovation Pack 17. It is recommended that you use your company’s keystore and truststore during installation. However, if a keystore and truststore are unavailable to you, this document describes how to create a self-signed certificate for a quick installation.

DETAILS
It is recommended that you use this document together with the steps described on the following site: https://sites.google.com/site/ddmwsst/create-your-own-certificate-and-ca

Before you begin.

You must have installed and configured OpenSSL and Java. This allows you to use OpenSSL and Java Keytool directly from the command line.

Creating and configuring a SSL certificate and certificate authority for Siebel Innovation Pack 17 is determined by certain components as follows:
• Web Browser
• Application Interface
• Certificate Authority
• Gateway

Command usage:

keytool -genkey -alias siebel -keystore c:\keys\siebelkeystore.jks -keyalg RSA
-sigalg SHA1withRSA -dname "cn=ip17demohost.demo.cp.com"

Is not working for them since the keystore file just coded for single host name where the authentication will be accepted from this host : cn=ip17demohost.demo.cp.com

To generate the certificate in multi tier installation in the same domain is to use the command :
keytool -genkey -alias siebel -keystore c:\keys\siebelkeystore.jks -keyalg RSA
-sigalg SHA1withRSA -dname "cn=*.demo.cp.com"

By giving "cn=*.demo.cp.com" it will accept the connection for all incoming request from Domain :demo.cp.com.

Oracle E-Business Suite Workflow Mailer Troubleshooting Guide

R12 – Workflow Mailer Troubleshooting Guide & Key Points

Oracle Workflow

• Oracle workflow is used to integrate ERP business processes into Oracle applications.
• Oracle workflow is a solution for integrating complete end-to-end business process in ERP.
• Oracle workflow allows people send/receive emails or notifications for approval.

Workflow Mailer Notification via email

The following are the key points that need to be taken care of if workflow emails are not received for the particular notification:

1.       A valid email address should be set for the recipient, for whom the email notification has to be received.

2.       The notification preference of the recipient must be ‘MAILHTML’ i.e ‘HTML Mail with attachments’, you can set the same from preferences section once the user login.

3.       You can check the notification preference and email address details from wf_local_roles/wf_roles tables.

4.       The workflow notification Mailer should be up and running (Outbound thread count should >=1 and Inbound thread count = 1)

Workflow Background process

• Workflow Background process is the standard concurrent program which should be scheduled for every 10 minutes with the following parameters:
• Y,N,N
• N,Y,N
• N,N,Y

Notification Mailers

• Check if all the agent listeners are up and running as shown below:
• Navigation Path:
• Go to ‘Workflow Administrator Web Applications’ responsibility and click on ‘Workflow Manager’ as shown below.

Notification Mailers, Agent Listeners, Background Engines etc., should be up and running

The cause of the workflow issue can be workflow not running or Notifications not being fired.

• If the workflow process is not getting initiated, get the workflow name and itemkey. Itemkey is a key to identify the workflow instances
• Eg: itemkey examples

1.       PO Approval Workflow

• SELECT wf_item_key FROM po_headers_all
• WHERE segment1 = :po_number
• AND org_id =  :organization_id ;
•  

2.       Requisition Workflow –

• SELECT wf_item_key FROM po_requisition_headers_all
• WHERE segment1 = :requisition_number
• AND org_id = :organization_id
• With the workflow name and the itemkey for the workflow which is failed, follow the below steps:

Go to workflow status monitor

Enter the Workflow type and Itemkey of the workflow

• Status of each workflow can be of Active/Error/Complete/Defferred

• Workflow Notifications not getting triggered:
• All the oracle workflow notifications are stored in the WF_NOTIFICATIONS table.
• Select * from WF_Notifications where notification_id = :notification_id and item_key = :item_key(po_header_id or requisition_header_id);
• Mail_status:
• Sent: – Mails are successfully sent to the recipients.
• Error: – Mails are not delivered to the recipient due to invalid email address.
• Status:
• Open: – Mails are sent to the recipient, but the user not read the email.
•      Closed: – Mail has been viewed by the recipient.
• Error: – Mail server is not able to deliver the message.
• Cancelled :- Workflow got cancelled
• Timeout :- Notification got timed out

Failed Notifications

• If notifications are not received, check if record exists in wf_notifications table.
• select * from wf_notifications where item_key = (Item key from status monitor windlow). Check the status and recipient_role values.
• Mail_status of the notifications should be ‘SENT’ for all open notifications. If notification is closed, Mail_Status will be null.
• Recipient_role should be approver name.
• To update the notification_preference from backend.
• update wf_local_roles set notification_preference = ‘MAILHTML’ where name = ‘ASECO.DAVID’

Tips:

• Clear the cache always if the notifications listed in the notification worklist were not able to open.
• Set the workflow Administrator privilege to “*” from sysadmin login to view all the workflow status diagrams of all the workflows owned by other users.

Debugging steps if Workflow errors out

• Login à User (with Workflow Administrator responsibility)
• Workflow Administrator -> Administrator Workflow -> Status Monitor-> Enter Type Internal Name (,POAPPRV, REQAPPRV, XXEFTPYM )-> Go
• List of workflows will be displayed. Select the workflow with respective to Payment batch id.
• Click status diagram/Activity History button. Errors can be listed out on the particular activity if any.

Check if the Workflow notification has been sent or not

• select mail_status, status from wf_notifications where notification_id= :notification_id
• If mail_status is MAIL, it means the email delivery is pending for workflow mailer to send the email notification
• If mail_status is SENT, its means workflow mailer has sent email
• If mail_status is Null and status is OPEN, then no need to send email as notification preference of user is “Don’t send email”
• Notification preference of user can be set by user by logging in application -> click on preference -> the notification preference
•  

How to Resend OPEN, CANCELLED Workflow Notification mails

• Many notification mailers stuck in the mailer queue (status=’OPEN’ and mail_status=’MAIL’)
• SQL> select count(*) from wf_notifications where status=’OPEN’ and mail_status=’MAIL’;
• select notification_id, status, mail_status, subject, from_user, begin_date from WF_NOTIFICATIONS where status = ‘OPEN’ and mail_status = ‘MAIL’;
• Oracle Workflows notifications cannot be sent due to mail server problems, network problems
• We can resend the notifications by doing the following:
• 1) stop workflow mailer
• 2) cd $FND_TOP/patch/115/sql
• 3) sqlplus APPS/<pwd> @wfntfqup.sql APPS <pwd> APPLSYS
• 4) start workflow mailer and monitor the queue

Vacation Rules:

• Vacation rules handle notifications automatically when the users are not available or on vacation/leave to manage their notifications personally. These rules are defined by the user according to the item type for the particular notification. In Oracle Applications, we can control what item types are available for vacation rules using the WF: Routing Rule Item Types lookup type and the WF: Vacation Rules – Allow All profile option.
• Note – If the reassign button is hidden from the notifications tab, then vacation rule will not function.

FYI Notifications are Autoclosed

• If FYI notifications for Approve/Reject has been autoclosed, then Autoclose FYI Flag in workflow notification mailer should be set to “N”

Metalink References

• Note: 1054215.1 – How to Check if the Workflow Mailer is Running
• Note: 415516.1 – How to Check Whether Notification Mailer is Working or Not
• Note: 831982.1 – 11i/R12 – A guide for troubleshooting Workflow Notification Emails – Inbound and Outbound
• Note: 1012344.7 – Notifications Not Being Sent In Workflow
• Note: 560472.1 – Workflow Mailers Not Sending Notifications

How to Delete a User in Oracle Solaris SPARC

Assume the root role.
$ su -
Password:
#

Note -  This method works whether root is a user account or a role.

Archive the user's home directory.

Delete the user.

# userdel -r username

The –r option removes the account from the system.

Because user home directories are now ZFS datasets, the preferred method for removing a local home directory for a deleted user is to specify the –r option with the userdel command.

If the user's home directory is on a remote server, manually delete it.
# userdel username

For a full list of command options, see the userdel (1M) man page.

Next Steps

Additional cleanup might be required if the user that you deleted had administrative responsibilities, for example creating cron jobs, or if the user had additional accounts in non-global zones.

Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection 

SYMPTOMS

Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Error Code: 17002

CAUSE

 Changed the database host address and the configuration files still  has the old database address not able to contact the database

 

SOLUTION

To implement the solution, please execute the following steps:

1. Updated the corrects database host names in the config files and restart the server. so that server will connect to the database

a) config.xml ( user_projects/<your_domain>/config)

b)jps-config.xml ( user_projects/<your_domain>/config/fmwconfig)

c) jps-config-jse.xml ( user_projects/<your_domain>/config/fmwconfig)

d)jdbc folder ( user_projects/<your_domain>/config/)

save the files and restart the server

Oracle on Solaris Fixing the ORA-27102 out of memory Error

ORA-27102: out of memory Error while starting the database

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an

ORA-27102: out of memory

error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup

ORA-27102: out of memory

SVR4 Error: 22: Invalid argument

Diagnosis

$ oerr ORA 27102

27102, 00000, "out of memory"

// \*Cause: Out of memory

// \*Action: Consult the trace file for details

% prtconf | grep Mem

Memory size: 32760 Megabytes

% id -p

uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3

project: 3: default

NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT

project.max-shm-memory

        privileged      7.84GB      -   deny                                 -

        system          16.0EB    max   deny                                 -

% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3

project: 3: default

NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT

project.max-shm-memory

        privileged      10.0GB      -   deny                                 -

        system          16.0EB    max   deny                                 -

How to Enable and Disable Firewall in Solaris server

To Enable the Firewall:

svcadm enable svc:/network/ipfilter:default

To Disable the Firewall:

svcadm disable svc:/network/ipfilter:default

To check the status of the Firewall:

ipfstat -io

Siebel Server is not starting because of the error "Unable To Initialize The Database Environment -- Unable To Start Common Api"

Unable To Initialize The Database Environment -- Unable To Start Common Api

SYMPTOMS:

Server seems not to be connecting to the database successfully,

Srbroker.log file shows the following messages:

GenericLog GenericError 1 0 2011-08-02 13:26:26 (srbthrd.cpp (3920) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api 

GenericLog GenericError 1 0 2011-08-02 13:26:26 (srbmtsrv.cpp (71) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api 

SrbLayerLog Error 1 0 2011-08-02 13:26:26 Main Init fails 

GenericLog GenericError 1 0 2011-08-02 13:26:26 (smimtsrv.cpp (1061) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api

CAUSE:

The following file was missing in the installed folder.

'C:\Siebel78\siebsrvr\sqltempl\comdb42.sql'.

SOLUTION:

The odbc connection was tested and the server was able to connect to Siebel database.

Since customer backs up the environment regularly, the file comdb42.sql was restored from the backup copy to the appropriate folder 'C:\Siebel78\siebsrvr\sqltempl'.

Srbroker error solved and the Server started up successfully.

Siebel does not start because System component initialization timed out

SBL-SMI-00151: System component initialization timed out. System will shut down

Error in the Log File   :

ServerLog ProcessExit 1 000fd13855f816e4:0 2015-09-16 12:52:45 SRBroker 3188 TERMINATED Process 3188 was terminated

NotifyEvt NotifyErrors 1 000fd13855f816e4:0 2015-09-16 12:52:45 SBL-ADM-09152Could not send a notification message because we could not connect to the notification pipe (ENTSRVPRD_SRVTI339_7), retCode = (9609), nSysErr = (0)

GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2296) err=1376409 sys=6) SBL-SMI-00153: SRBroker NOT READY Component is not ready.

GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2237) err=1376408 sys=0) SBL-SMI-00152: TIMEOUT Component initialization for priority level 2 timed out after 300 s.

GenericLog GenericFatal 0 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2248) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.

GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (708) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.

GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (146) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.

ServerLog ProcessExit 1 000fd13855f816e4:0 2015-09-16 12:57:39 SCBroker 4304 TERMINATED Process 4304 was terminated

GenericLog GenericError 1 000fd12d55f816e4:0 2015-09-16 12:57:39 (listener.cpp (434) err=1376297 sys=126) SBL-SMI-00041: Internal: Could not start the scheduler thread (1376407)

GenericLog GenericError 1 000fd12d55f816e4:0 2015-09-16 12:57:39 (lstnsvc.cpp (181) err=1376297 sys=0) SBL-SMI-00041: Internal: Could not start the scheduler thread (1376407)

ServerLog ServerShutdown 1 000fd12d55f816e4:0 2015-09-16 12:57:39 Scheduler error

SCBroker logs shows below errors     :

2021 2015-09-16 12:52:39 0000-00-00 00:00:00 -0300 00000000 001 ffff 0001 09 SCBroker 3145731 4304 3044 D:\Siebel\8.1.1.11.0\ses\siebsrvr\log\SCBroker_0003_3145731.log 8.1.1.11 SIA [23030] ENU

GenericLog GenericError 1 0000000255f910d0:0 2015-09-16 12:52:39 (scbcomp.cpp (447) err=4653071 sys=0) SBL-SCB-00015: The component is down or not available on this server.

GenericLog GenericError 1 0000000255f910d0:0 2015-09-16 12:52:39 (scbcomp.cpp (266) err=4653071 sys=0) SBL-SCB-00015: The component is down or not available on this server.

Upon Siebel server startup, the SRBroker component does not initialize and the SCBroker logs for similar timestamp shows that the component is down. The system components were not initializing and timing out and this could be the reason why the Siebel server was not starting up.

CAUSE    :  

The cause of the issue was determined to be with the Siebel server missing the SQLTEMPL folder. This was identified after comparing the working and non-working servers and it was identified that the SQLTEMPL folder was missing on the problematic server.

The Siebel installation procedure populates this SQLTEMPL folder with SQL files, containing SQL templates that can be used to create SQL statements that the Siebel Server uses to perform for specific database operations. This was missing and hence the components could not initialize as none of the pre-defined queries could be executed because of the missing directory.

SOLUTION:

The solution action plan for the issue is as below:

1. Stop the Siebel and gateway server.

2. Copy the SQLTEMPL folder from the working server to the non-working server

3. Restart the Siebel gateway and Siebel server and check to see if the system component initializes and does not timeout and bring down the Siebel server.

After copying the SQLTEMPL folder from working server, the Siebel server started up without any issues.

How to change the Siebel DB user Password

Changing the Table Owner (DBO) Password

The Siebel Database Server installation script also creates a database Table Owner (DBO) account used to modify the Siebel database tables. The default user ID and password for this database account are SIEBEL and SIEBEL (case-sensitive). You should change the password for this account.

The Table Owner is used to reference table names in SQL statements that are generated by the Siebel application (for example, SELECT * FROM SIEBEL.S_APP_VER).

A corresponding parameter is configured for the Siebel Enterprise, named Table Owner (alias TableOwner). Siebel application modules such as Application Object Managers (AOMs) use this parameter value to provide the Table Owner name when generating SQL for database operations. You specify the Table Owner name during Siebel Enterprise Server configuration, which provides a value for this parameter.

A related parameter is Table Owner Password (alias TableOwnPass). For most database operations performed for Siebel Business Applications, the Table Owner password does not need to be provided. For this reason, this parameter is not configured during Siebel Enterprise Server configuration.

However, if the Table Owner Password parameter is not defined, then the Table Owner password may sometimes need to be provided manually.

Note the following requirements for changing the Table Owner password:

If you have not defined the Table Owner Password parameter, then the Table Owner password only has to be changed in the Siebel Database. (The changed password may also need to be provided manually for certain operations.)

If you have defined the Table Owner Password parameter, then you must also update the value for this parameter when you change the password in the Siebel Database.

To change the password for the Table Owner account

Change the Table Owner password for the Enterprise, using Server Manager.

Log into a Siebel employee application, such as Siebel Call Center.

From the application-level menu, choose Navigate > Site Map > Administration - Server Configuration > Enterprises.

Click the Parameters tab.

In the Enterprise Parameters list, locate the Table Owner Password parameter (alias TableOwnPass).

In the Value field, type in the new value, then commit the record.

If you have Siebel Developer Web Client users, also change the value for the TableOwnPass parameter in the [ServerDataSrc] section of each application configuration file, such as uagent.cfg for Siebel Call Center. This step must be done for each user's Siebel client installation.

Change the password in the database.

alter user siebel identified by ******;

Restart the Siebel Server.

Siebel - The SCBroker listening port may already be in use by another process

SBL-SCB-00016: Inherited socket (0) is invalid. The SCBroker listening port may already be in use by another process

STEPS
-----------------------

The issue can be reproduced at will with the following steps:
1. Login to siebel server box.
2. Run siebenv.sh
3. Run start_server all
4. Server is not coming up and throwing error.

BUSINESS IMPACT
-----------------------

The issue has the following business impact:
Customer has schedule a downtime to restart the siebel servers.As the siebel server is not come up all the Siebel Marketing users will affect,it will impact their production.

CAUSE
-----------------------

Issue is caused due to SCBroker component port is used by some other process in the System.

SOLUTION
------------------------

1)Open a command prompt in siebel server.
2)Run netstat -a
3)verify and kill if any process using  port 2321 or reboot the OS to release the 2321 port.
4)Bring up the Siebel server and verify the status.

Killing Session using alter session Command

ALTER SYSTEM KILL SESSION

For Single Instance Database:

SELECT S.SID, S.SERIAL#, S.OSUSER, S.PROGRAM FROM V$SESSION S;

ALTER SYSTEM KILL SESSION 'sid,serial#';

For RAC Instances:

SQL> select inst_id,sid,serial# from gv$session where username='SCOTT';

   INST_ID        SID    SERIAL# 

---------- ---------- ---------- 

         1        252 45632

SQL>  alter system kill session '252,45632,1'; 

 alter system kill session '252,45632,1' 

* 

ERROR at line 1: 

ORA-00026: missing or invalid session ID

Now, it works:

SQL>  alter system kill session '252,45632,@1';

System altered.

Query to find Scheduled Concurrent Request in Oracle E - Business Suite

Query to find Oracle E - Business Suite scheduled concurrent requests:

select r.request_id,
       p.user_concurrent_program_name ||
       case
          when p.user_concurrent_program_name = 'Report Set' then
            (select ' - ' || s.user_request_set_name
              from apps.fnd_request_sets_tl s
             where s.application_id = r.argument1
               and s.request_set_id = r.argument2
               and language = 'US'
            )
          when p.user_concurrent_program_name = 'Check Periodic Alert' then
            (select ' - ' || a.alert_name
              from apps.alr_alerts a
             where a.application_id = r.argument1
               and a.alert_id = r.argument2
               and language = 'US'
            )
       end concurrent_program_name,
       decode(c.class_type,
              'P', 'Periodic',
              'S', 'On Specific Days',
              'X', 'Advanced',
              c.class_type
             ) schedule_type,  
       case
          when c.class_type = 'P' then
            'Repeat every ' ||
            substr(c.class_info, 1, instr(c.class_info, ':') - 1) ||
            decode(substr(c.class_info, instr(c.class_info, ':', 1, 1) + 1, 1),
                   'N', ' minutes',
                   'M', ' months',
                   'H', ' hours',
                   'D', ' days') ||
            decode(substr(c.class_info, instr(c.class_info, ':', 1, 2) + 1, 1),
                  'S', ' from the start of the prior run',
                  'C', ' from the completion of the prior run')
          when c.class_type = 'S' then
             nvl2(dates.dates, 'Dates: ' || dates.dates || '. ', null) ||
             decode(substr(c.class_info, 32, 1), '1', 'Last day of month ') ||
             decode(sign(to_number(substr(c.class_info, 33))),
                    '1',  'Days of week: ' ||
                    decode(substr(c.class_info, 33, 1), '1', 'Su ') ||
                    decode(substr(c.class_info, 34, 1), '1', 'Mo ') ||
                    decode(substr(c.class_info, 35, 1), '1', 'Tu ') ||
                    decode(substr(c.class_info, 36, 1), '1', 'We ') ||
                    decode(substr(c.class_info, 37, 1), '1', 'Th ') ||
                    decode(substr(c.class_info, 38, 1), '1', 'Fr ') ||
                    decode(substr(c.class_info, 39, 1), '1', 'Sa '))
       end schedule,
    r.requested_start_date next_run,
       case
          when p.user_concurrent_program_name != 'Report Set' and
               p.user_concurrent_program_name != 'Check Periodic Alert' then
               r.argument_text
       end argument_text,
       r.hold_flag on_hold,
       c.date1 start_date,
       c.date2 end_date,
       c.class_info, user_name
  from apps.fnd_concurrent_requests r,
       applsys.fnd_conc_release_classes c,
       apps.fnd_concurrent_programs_tl p,
       apps.fnd_user                    usr,
       (SELECT release_class_id,
               substr(max(SYS_CONNECT_BY_PATH(s, ' ')), 2) dates  ,a
          FROM (select release_class_id,
                       rank() over(partition by release_class_id order by s) a,
                       s
                  from (select c.class_info,
                               l,
                               c.release_class_id,
                               decode(substr(c.class_info, l, 1), '1', to_char(l)) s
                          from (select level l
                                  from dual
                               connect by level <= 31),
                               apps.fnd_conc_release_classes c
                         where c.class_type = 'S')
                  where s is not null)
         CONNECT BY PRIOR
                     (a || release_class_id) = (a - 1) || release_class_id
        group by release_class_id,a) dates
  where r.phase_code = 'P'
    and c.application_id = r.release_class_app_id
    and c.release_class_id = r.release_class_id
    and nvl(c.date2, sysdate + 1) > sysdate
    and c.class_type is not null
    and p.concurrent_program_id = r.concurrent_program_id
    and p.application_id = r.program_application_id
    and p.language = 'US'
    and dates.release_class_id(+) = r.release_class_id
    and usr.user_id = requested_by
  order by requested_by,on_hold, next_run;

ORA-16024: parameter log_archive_dest_1 cannot be parsed

ORA-16024 Referencing LOG_ARCHIVE_DEST_n Parameter

SYMPTOMS

You are using the log_archive_dest_n parameter in your pfile:

log_archive_dest_1 = 'location = /u01/oracle/arc' 

and are receiving an:

ORA-16024: parameter log_archive_dest_1 cannot be parsed

SOLUTION

Ensure that /u01/oracle/arc is a valid directory and that Oracle has write permissions there.

Remove the spaces on either side of the '=' sign.
You cannot have as space on either side on the '=' sign after 'location'.
Note that the word 'location' can be in either upper or lower case.
log_archive_dest_1 = 'location=/u01/oracle/arc'

Make sure that there is no trailing /.

Make sure that LOG_ARCHIVE_DEST is not set in the init.ora.
These 2 parameters are not compatible with each other.

NOTE

The error ORA-16024 could also be encountered when an SPFILE is used, for instance when upgrading a database.

To prevent that, you can remove the log_archive_dest from spfile by issuing

SQL> alter system reset log_archive_dest scope=spfile;

If still not working set the LOG_ARCHIVE_DEST_n parameter to a null value:

LOG_ARCHIVE_DEST_n=''

------------------------------------------------------------------------------------------------

How to change EBS R12 Database and Application Port Number

Execute the below in Application Tier:

Run adpreclone on Application Tier
   $ perl ./adpreclone.pl appsTier

Execute the following on the Database Tier:

Logon as oracle database user and source the env file for R12 RDBMS ORACLE_HOME

Start the database instance for which you want to change R12 port

Backup contextfile

Change as below in the contextfile

        dbport  : Change database port from 1521 to 1531
        cmanport : 1521 to 1531

Run adautocfg.sh

Execute the following in Application Tier:

Logon as application user and source the env for R12

Backup contextfile

Run adcfgclone and choose new port pool
   $ cd $COMMON_TOP/clone/bin
   $ perl ./adcfgclone.pl appsTier

Check new application URL

SQL> select home_url from icx_parameters;

HOME_URL
------------------------------------------------------------
http://<hostname.domainname>:8010/OA_HTML/AppsLogin

Run adpreclone on both apps and database tier

----------------------------------------------------------------------------------------------------

weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT

The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
        at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
        at weblogic.store.io.file.Heap.open(Heap.java:325)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

A search on support.oracle.com for keywords resulted in this hit:

WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)

WORKAROUNDS:

This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted. 

Go to DOMAIN_HOME/servers/<SERVERNAME>/data/ldap/ldapfiles and remove the EmbeddedLDAP.lok lock file
Delete the *.dat file under  DOMAIN_HOME/servers/<SERVERNAME>/data /store/default and DOMAIN_HOME/servers/<SERVERNAME>/data /store/diagnostics/
Remove the contents under  DOMAIN_HOME/servers/<SERVERNAME>/ tmp and DOMAIN_HOME/servers/<SERVERNAME>/ cache

Now start the server
./startWebLogic.sh

How to modify resource dependencies in CRS

I wanted to change the diskgroup used for flash recovery area. So I created a new diskgroup and updated the parameters

SQL> show parameter db_recovery_file_dest 

db_recovery_file_dest string +RACFRA

SQL> alter system set db_recovery_file_dest=’+FRA’

System altered.

SQL> show parameter db_recovery_file_dest

db_recovery_file_dest string +FRA

Also removed the old diskgroup

SQL> drop diskgroup RACFRA

Diskgroup dropped.

After that shutdown the database for some maintenance. But when I tried starting it back, it started throwing

following error

oragrid@node-20> srvctl start database -d mydb

PRCR-1079 : Failed to start resource ora.mydb.db

CRS-2640: Required resource ‘ora.RACFRA.dg’ is missing.

I was not sure, what exactly happened. As I already dropped this diskgroup, database & ASM diskgroup

dependency should have been removed. But error message was indicating something else

So my first check, was to see if the resource was still present

oragrid@node-20> crsctl stat res -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Local Resources

——————————————————————————–

ora.FRA.dg

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.LISTENER.lsnr

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.DG.dg

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.DG2.dg

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.DG3.dg

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.DG4.dg

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.asm

ONLINE ONLINE node-20 Started

ONLINE ONLINE node-21 Started

ora.eons

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.gsd

OFFLINE OFFLINE node-20

OFFLINE OFFLINE node-21

ora.net1.network

ONLINE ONLINE node-20

ONLINE ONLINE node-21

ora.ons

ONLINE ONLINE node-20

ONLINE ONLINE node-21

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE node-21

ora.oc4j

1 OFFLINE OFFLINE

ora.node-20.vip

1 ONLINE ONLINE node-20

ora.node-21.vip

1 ONLINE ONLINE node-21

ora.scan1.vip

1 ONLINE ONLINE node-21

ora.mydb.db

1 OFFLINE OFFLINE Instance Shutdown

2 OFFLINE OFFLINE Instance Shutdown

crsctl output did not show this diskgroup. So checked the dependency for the database resource

# crsctl status resource ora.mydb.db -f

Where I found, that START_DEPENDENCIES still have reference to old diskgroup. So removed, re-added the

database & associated instances

oragrid@node-21> srvctl remove database -d mydb

oragrid@node-21> srvctl add database -d mydb -o /oracle/11gr2

ora11gr2@node-21> srvctl add instance -d mydb -i mydb1 -n node-20

ora11gr2@node-21> srvctl add instance -d mydb -i mydb2 -n node-21

Again checked the dependency for the database resource

# crsctl status resource ora.mydb.db -f

This time START_DEPENDENCIES was having correct set of ASM diskgroups & the old DG reference was

removed.

Now started the database, all was fine.

oragrid@node-21> srvctl start database -d mydb

So to avoid this problem, we should be using srvctl remove diskgroup, instead of using sqlplus.