Tuesday, 19 September 2017

Oracle E-Business Suite Workflow Mailer Troubleshooting Guide

R12 – Workflow Mailer Troubleshooting Guide & Key Points
Oracle Workflow
  • Oracle workflow is used to integrate ERP business processes into Oracle applications.
  • Oracle workflow is a solution for integrating complete end-to-end business process in ERP.
  • Oracle workflow allows people send/receive emails or notifications for approval.
Workflow Mailer Notification via email
The following are the key points that need to be taken care of if workflow emails are not received for the particular notification:
1.       A valid email address should be set for the recipient, for whom the email notification has to be received.
2.       The notification preference of the recipient must be ‘MAILHTML’ i.e ‘HTML Mail with attachments’, you can set the same from preferences section once the user login.

1
3.       You can check the notification preference and email address details from wf_local_roles/wf_roles tables.
4.       The workflow notification Mailer should be up and running (Outbound thread count should >=1 and Inbound thread count = 1)
Workflow Background process
  • Workflow Background process is the standard concurrent program which should be scheduled for every 10 minutes with the following parameters:
  • Y,N,N
  • N,Y,N
  • N,N,Y
1
1
Notification Mailers
  • Check if all the agent listeners are up and running as shown below:
  • Navigation Path:
    • Go to ‘Workflow Administrator Web Applications’ responsibility and click on ‘Workflow Manager’ as shown below.
Notification Mailers, Agent Listeners, Background Engines etc., should be up and running
1
The cause of the workflow issue can be workflow not running or Notifications not being fired.
  • If the workflow process is not getting initiated, get the workflow name and itemkey. Itemkey is a key to identify the workflow instances
  • Eg: itemkey examples
1.       PO Approval Workflow
  • SELECT wf_item_key FROM po_headers_all
  • WHERE segment1 = :po_number
  • AND org_id =  :organization_id ;
  •  
2.       Requisition Workflow 
  • SELECT wf_item_key FROM po_requisition_headers_all
  • WHERE segment1 = :requisition_number
  • AND org_id = :organization_id
  • With the workflow name and the itemkey for the workflow which is failed, follow the below steps:
Go to workflow status monitor
Enter the Workflow type and Itemkey of the workflow
  • Status of each workflow can be of Active/Error/Complete/Defferred
1
  • Workflow Notifications not getting triggered:
    • All the oracle workflow notifications are stored in the WF_NOTIFICATIONS table.
    • Select * from WF_Notifications where notification_id = :notification_id and item_key = :item_key(po_header_id or requisition_header_id);
    • Mail_status:
      • Sent: – Mails are successfully sent to the recipients.
      • Error: – Mails are not delivered to the recipient due to invalid email address.
    • Status:
      • Open: – Mails are sent to the recipient, but the user not read the email.
      •      Closed: – Mail has been viewed by the recipient.
      • Error: – Mail server is not able to deliver the message.
      • Cancelled :- Workflow got cancelled
      • Timeout :- Notification got timed out
Failed Notifications
    • If notifications are not received, check if record exists in wf_notifications table.
    • select * from wf_notifications where item_key = (Item key from status monitor windlow). Check the status and recipient_role values.
    • Mail_status of the notifications should be ‘SENT’ for all open notifications. If notification is closed, Mail_Status will be null.
    • Recipient_role should be approver name.
    • To update the notification_preference from backend.
    • update wf_local_roles set notification_preference = ‘MAILHTML’ where name = ‘ASECO.DAVID’
Tips:
    • Clear the cache always if the notifications listed in the notification worklist were not able to open.
    • Set the workflow Administrator privilege to “*” from sysadmin login to view all the workflow status diagrams of all the workflows owned by other users.
1
Debugging steps if Workflow errors out
  • Login à User (with Workflow Administrator responsibility)
  • Workflow Administrator -> Administrator Workflow -> Status Monitor-> Enter Type Internal Name (,POAPPRV, REQAPPRV, XXEFTPYM )-> Go
  • List of workflows will be displayed. Select the workflow with respective to Payment batch id.
  • Click status diagram/Activity History button. Errors can be listed out on the particular activity if any.
Check if the Workflow notification has been sent or not
  • select mail_status, status from wf_notifications where notification_id= :notification_id
  • If mail_status is MAIL, it means the email delivery is pending for workflow mailer to send the email notification
  • If mail_status is SENT, its means workflow mailer has sent email
  • If mail_status is Null and status is OPEN, then no need to send email as notification preference of user is “Don’t send email”
  • Notification preference of user can be set by user by logging in application -> click on preference -> the notification preference
  •  
How to Resend OPEN, CANCELLED Workflow Notification mails
  • Many notification mailers stuck in the mailer queue (status=’OPEN’ and mail_status=’MAIL’)
  • SQL> select count(*) from wf_notifications where status=’OPEN’ and mail_status=’MAIL’;
  • select notification_id, status, mail_status, subject, from_user, begin_date from WF_NOTIFICATIONS where status = ‘OPEN’ and mail_status = ‘MAIL’;
  • Oracle Workflows notifications cannot be sent due to mail server problems, network problems
  • We can resend the notifications by doing the following:
  • 1) stop workflow mailer
  • 2) cd $FND_TOP/patch/115/sql
  • 3) sqlplus APPS/<pwd> @wfntfqup.sql APPS <pwd> APPLSYS
  • 4) start workflow mailer and monitor the queue
Vacation Rules:
  • Vacation rules handle notifications automatically when the users are not available or on vacation/leave to manage their notifications personally. These rules are defined by the user according to the item type for the particular notification. In Oracle Applications, we can control what item types are available for vacation rules using the WF: Routing Rule Item Types lookup type and the WF: Vacation Rules – Allow All profile option.
  • Note – If the reassign button is hidden from the notifications tab, then vacation rule will not function.
FYI Notifications are Autoclosed
  • If FYI notifications for Approve/Reject has been autoclosed, then Autoclose FYI Flag in workflow notification mailer should be set to “N”
Metalink References
  • Note: 1054215.1 – How to Check if the Workflow Mailer is Running
  • Note: 415516.1 – How to Check Whether Notification Mailer is Working or Not
  • Note: 831982.1 – 11i/R12 – A guide for troubleshooting Workflow Notification Emails – Inbound and Outbound
  • Note: 1012344.7 – Notifications Not Being Sent In Workflow
  • Note: 560472.1 – Workflow Mailers Not Sending Notifications



Friday, 15 September 2017

How to Delete a User in Oracle Solaris SPARC

Assume the root role.
$ su -
Password:
#

Note -  This method works whether root is a user account or a role.

Archive the user's home directory.

Delete the user.

# userdel -r username

The –r option removes the account from the system.

Because user home directories are now ZFS datasets, the preferred method for removing a local home directory for a deleted user is to specify the –r option with the userdel command.

If the user's home directory is on a remote server, manually delete it.
# userdel username

For a full list of command options, see the userdel (1M) man page.

Next Steps

Additional cleanup might be required if the user that you deleted had administrative responsibilities, for example creating cron jobs, or if the user had additional accounts in non-global zones.

Tuesday, 11 July 2017

Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection 


SYMPTOMS


Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
Error Code: 17002

CAUSE


 Changed the database host address and the configuration files still  has the old database address not able to contact the database
 

SOLUTION


To implement the solution, please execute the following steps:

1. Updated the corrects database host names in the config files and restart the server. so that server will connect to the database
a) config.xml ( user_projects/<your_domain>/config)
b)jps-config.xml ( user_projects/<your_domain>/config/fmwconfig)
c) jps-config-jse.xml ( user_projects/<your_domain>/config/fmwconfig)
d)jdbc folder ( user_projects/<your_domain>/config/)

save the files and restart the server


Oracle on Solaris Fixing the ORA-27102 out of memory Error

ORA-27102: out of memory Error while starting the database


Symptom:


As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an
ORA-27102: out of memory
error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Diagnosis
$ oerr ORA 27102
27102, 00000, "out of memory"
// \*Cause: Out of memory
// \*Action: Consult the trace file for details


% prtconf | grep Mem
Memory size: 32760 Megabytes

% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      7.84GB      -   deny                                 -
        system          16.0EB    max   deny                                 -


% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      10.0GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Monday, 10 July 2017

How to Enable and Disable Firewall in Solaris server

To Enable the Firewall:

svcadm enable svc:/network/ipfilter:default

To Disable the Firewall:

svcadm disable svc:/network/ipfilter:default

To check the status of the Firewall:

ipfstat -io

Friday, 7 July 2017

Siebel Server is not starting because of the error "Unable To Initialize The Database Environment -- Unable To Start Common Api"

Unable To Initialize The Database Environment -- Unable To Start Common Api

SYMPTOMS:

Server seems not to be connecting to the database successfully,

Srbroker.log file shows the following messages:

GenericLog GenericError 1 0 2011-08-02 13:26:26 (srbthrd.cpp (3920) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api 

GenericLog GenericError 1 0 2011-08-02 13:26:26 (srbmtsrv.cpp (71) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api 

SrbLayerLog Error 1 0 2011-08-02 13:26:26 Main Init fails 

GenericLog GenericError 1 0 2011-08-02 13:26:26 (smimtsrv.cpp (1061) err=3200016 sys=0) SBL-SRM-00016: Unable to initialize the Database environment -- Unable to start common api

CAUSE:

The following file was missing in the installed folder.

'C:\Siebel78\siebsrvr\sqltempl\comdb42.sql'.

SOLUTION:

The odbc connection was tested and the server was able to connect to Siebel database.

Since customer backs up the environment regularly, the file comdb42.sql was restored from the backup copy to the appropriate folder 'C:\Siebel78\siebsrvr\sqltempl'.

Srbroker error solved and the Server started up successfully.


Siebel does not start because System component initialization timed out

SBL-SMI-00151: System component initialization timed out. System will shut down



Error in the Log File   :


ServerLog ProcessExit 1 000fd13855f816e4:0 2015-09-16 12:52:45 SRBroker 3188 TERMINATED Process 3188 was terminated
NotifyEvt NotifyErrors 1 000fd13855f816e4:0 2015-09-16 12:52:45 SBL-ADM-09152Could not send a notification message because we could not connect to the notification pipe (ENTSRVPRD_SRVTI339_7), retCode = (9609), nSysErr = (0)
GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2296) err=1376409 sys=6) SBL-SMI-00153: SRBroker NOT READY Component is not ready.
GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2237) err=1376408 sys=0) SBL-SMI-00152: TIMEOUT Component initialization for priority level 2 timed out after 300 s.
GenericLog GenericFatal 0 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (2248) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.
GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (708) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.
GenericLog GenericError 1 000034de55f911c8:0 2015-09-16 12:57:39 (schedule.cpp (146) err=1376407 sys=0) SBL-SMI-00151: System component initialization timed out. System will shut down.
ServerLog ProcessExit 1 000fd13855f816e4:0 2015-09-16 12:57:39 SCBroker 4304 TERMINATED Process 4304 was terminated
GenericLog GenericError 1 000fd12d55f816e4:0 2015-09-16 12:57:39 (listener.cpp (434) err=1376297 sys=126) SBL-SMI-00041: Internal: Could not start the scheduler thread (1376407)
GenericLog GenericError 1 000fd12d55f816e4:0 2015-09-16 12:57:39 (lstnsvc.cpp (181) err=1376297 sys=0) SBL-SMI-00041: Internal: Could not start the scheduler thread (1376407)
ServerLog ServerShutdown 1 000fd12d55f816e4:0 2015-09-16 12:57:39 Scheduler error

SCBroker logs shows below errors     :


2021 2015-09-16 12:52:39 0000-00-00 00:00:00 -0300 00000000 001 ffff 0001 09 SCBroker 3145731 4304 3044 D:\Siebel\8.1.1.11.0\ses\siebsrvr\log\SCBroker_0003_3145731.log 8.1.1.11 SIA [23030] ENU
GenericLog GenericError 1 0000000255f910d0:0 2015-09-16 12:52:39 (scbcomp.cpp (447) err=4653071 sys=0) SBL-SCB-00015: The component is down or not available on this server.
GenericLog GenericError 1 0000000255f910d0:0 2015-09-16 12:52:39 (scbcomp.cpp (266) err=4653071 sys=0) SBL-SCB-00015: The component is down or not available on this server.

Upon Siebel server startup, the SRBroker component does not initialize and the SCBroker logs for similar timestamp shows that the component is down. The system components were not initializing and timing out and this could be the reason why the Siebel server was not starting up.

CAUSE    :  

The cause of the issue was determined to be with the Siebel server missing the SQLTEMPL folder. This was identified after comparing the working and non-working servers and it was identified that the SQLTEMPL folder was missing on the problematic server.

The Siebel installation procedure populates this SQLTEMPL folder with SQL files, containing SQL templates that can be used to create SQL statements that the Siebel Server uses to perform for specific database operations. This was missing and hence the components could not initialize as none of the pre-defined queries could be executed because of the missing directory.

SOLUTION:

The solution action plan for the issue is as below:

1. Stop the Siebel and gateway server.
2. Copy the SQLTEMPL folder from the working server to the non-working server
3. Restart the Siebel gateway and Siebel server and check to see if the system component initializes and does not timeout and bring down the Siebel server.

After copying the SQLTEMPL folder from working server, the Siebel server started up without any issues.