Slow Lombardi Process Designer

During development time the Process Designer makes many queries to LSW_ENV_VAR.
This table is not large, however, one of the queries used by PD has a sub-query against LSW_PO_VERSIONS

This article (http://www-01.ibm.com/support/docview.wss?uid=swg21610258
) describes adding a few indexes to LSW_PO_VERSIONS and it worked a treat.

This other article (http://www-01.ibm.com/support/docview.wss?uid=swg1IC82730) was a little more involved and we didn't try it.. but worth a mention.

Broken Portal in BPM v8

Don't ask me why.. but if you enable debug on your Mashup Config Service (Admin Console > Resources > Resource Environment > Resource Environment Providers > Mashups_ConfigService > Custom properties > isDebug = true) it brakes the BPM Portal (blank screen and half blank header - see below)
 


 You'll also notice a bundle of errors in Firebug, the 2 key ones (I suspect) are shown below






...neither of these errors appear on the working portal (although I should point out that there are no shortage of errors reported in Firebug which I can only assume are "normal" and do not appear to affect the Portal's functionality)

Set this property back to false and it'll magically start to work...
I understand when set to true an uncompressed version of Dojo javascript libs are used. Why this would break things is a mystery to me though.

Lombardi BPM Instrumentation

Great article taken from http://www-01.ibm.com/support/docview.wss?uid=swg21613989

1. Log into the Process Admin Console as a user in the tw_admins group, such as tw_admin or admin. The error SECJ0305 may appear if this is run without the base internal users.
   
   For example:
   http://server_name:port_number/ProcessAdmin
   
   
2. Under Monitoring, click Instrumentation.
   
   
3. Click Start Logging to collect data. Note where it stores the file. For example, in IBM Business Process Manager Version 7.5, the file might be stored in the following directory:
   C:\IBM\BPM\v7.5\profiles\ProcCtr01\logs\inst001.dat
   
   
4. Click Stop Logging.
   
   
5. Retrieve the file for processing. 

 
How do I to decode the instrumentation .dat files? 
Use the following command to extract the data: 
java -Xmx1024M -cp svrcoreclnt.jar  com.lombardisoftware.instrumentation.log.tools.NonXMLDump inst001.dat >  inst001.txt 
 
Notes: 

• The svrcoreclnt.jar file needs to be in a path that Java can execute. You can copy this file to a separate directory to your desktop to run. The location of  the Java archive (JAR) file is: [install_dir]\Lombardi\lib
  
  
• 1024MB heap will be sufficient for almost all cases.
  
  
• inst001.dat is the instrumentation file.
  
  
• inst001.txt is a decoded file in the plain text format. 

Reading the decoded file
The instrumentation log files are important because they tell you exactly how long it took for something to run. The following command finds all threads that took longer than 1000 milliseconds (1 second) to complete, which are the threads that you should examine first.
 
grep -E "(THREAD|period [0-9]{4,}ms)" inst001.txt 

Example Sample output: 
>> THREAD Thread-47 << 
>> THREAD WebContainer : 2 << 
03:17:10.356 period 1156ms 'Resume Workflow Engine' { 
03:17:10.418    period 1031ms 'Do Job'  
Worker=com.lombardisoftware.component.javaconnector.worker.JavaConnectorWorker { 
03:17:10.418       period 1031ms 'Java Execution' { 
>> THREAD WebContainer : 13 << 
>> THREAD WebContainer : 16 << 

The times are nested,  1156ms 'Resume Workflow Engine' is running a coach and took 1.2 seconds. Within here, you see 1.0 seconds for executing Java. After this function, there might be another function that took 400 milliseconds.  

A note about the browser based Process Inspector and SSL trust

It's worth noting that when you use the browser based Lombardi Process Inspector...



.... an outbound  connection is made from an AppTarget cluster member JVM to the endpoint specified in 100Custom.xml (or 99Local.xml depending on how proper you're being).   In many cases I've seen this endpoint left as the relevant WAS node ... but with me being OCD I made this endpoint the Load Balancer so that things were load balanced. Under these circumstances the public certificate issued by the webservers that sit behind the load balancer (unless of course you terminate SSL on the load balancer itself in which case you'd need to trust this instead)  wouldn't be trusted by the WAS Cell, so you'll need to import it into the relevant trust store. I import into the CellDefaultTrustStore because I like the convenience.

If you don't trust the cert you'll see things like javax.net.ssl.SSLPeerUnverifiedException and javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated in the AppTarget logs

BPM off line deployments

Courtesy of my colleague Jeff Tan of IBM

 
Define an offline server
On the Process Center...  Servers > Add a New Offline Server

Create Deployment Package
This effectively deploys the process app snapshot to the offline server defined in the step above. This generates a package in the Process Center database which can be extracted later.

1. Login to the Process Center, navigate to the snapshot you wish to perform an offline deployment for.
2. Click Install, and select the offline server you created earlier.
3. Click Create installation packageAt this point, the Process Center has built the package for deployment. Now we need to extract it via wsadmin.

Extract Deployment Package

1. Login to a node hosting an AppTarget server of the Process Center.
   cd to the <profile path>/bin directory hosting the AppTarget server, e.g. /opt/bpm/profiles/profile1/bin 
2. Run ./wsadmin.sh -lang jython -username admin -password ***** -host <node hostname> -port <node SOAP port>.  (NOTE:  the node port is the SOAP port for the AppTarget server, NOT the deployment manager)
3. AdminTask.BPMExtractOfflinePackage('[-containerAcronym <process app acronym> -containerSnapshotAcronym <snapshot acronym> -containerTrackAcronym <track acronym - default 'Main'> -serverName <offline server name> -outputFile <path to output offline deployment package zip>]')

where:

• process app acronym  is the acronym of the process application, e.g. BCSBX
• snapshot acronym  is the snapshot acronym, e.g 1.1.0.6 (you can find this in the LSW_SNAPSHOT table)
• track acronym  is the name of the track within the snapshot (The default is Main but if it's different you can find the acronym in the LSW_BRANCH table)
• offline server name  is the name of the offline server created earlier. (This really is the name, there is no acronym so it you're contains whitespace you'll need to use quotes)
• path to output offline deployment package zip  is the full path to a zip file which will be the generated offline deployment package.(NOTE, if you don't put a dot in the filename you'll get a "Array out of bounds error" ... how rubbish is that?)

For example
clarkeb@jupiter > ./wsadmin.sh -lang jython -username admin -password ***** -host jupiter -port 13205

wsadmin>AdminTask.BPMExtractOfflinePackage('[-containerAcronym BCSBX -containerSnapshotAcronym 1.1.0.6 -containerTrackAcronym Main -serverName "Offline DEV" -outputFile /opt/bpm/dbp1.1.0.6.zip]')


Install Deployment Package

1. Login to a node hosting an AppTarget server of the Process Server you want to perform the offline deployment.
2. Ensure that the previously extracted offline deployment package has been copied over to this node via scp or other means.
3. cd to the <profile path>/bin directory hosting the AppTarget server, e.g. /opt/bpm/bpm751/profiles/psuat001/bin
4. Run ./wsadmin.sh -lang jython -username admin -password ***** -host <node hostname> -port <node SOAP port>.  (NOTE  the node port is the SOAP port for the AppTarget server, NOT the deployment manager)
5. AdminTask.BPMInstallOfflinePackage('-inputFile <path to input offline deployment package zip>')

Lombardi BPD not starting task

I had a Simulate User Task activity which I tried to assign to a List of Users - i.e. under Routing, the drop down was set to List of Users which contained 2 LDAP users and 1 user from the local repository (in this case tw_admin). When I ran this BPD, the user task simply didn't run. No errors in the SystemOut or SystemErr logs, nothing.  However, if I took the local repository user our of the List of Users it all worked fine. The key difference between the two users was their OU. Needs further investigation

Running BPD instances not visible in Process Designer but OK in browser based Process Inspector - also unable to terminate.

When using the REST API (/bpmrest-ui) to start a BPD instance, it does not appear in PD but can be seen in the browser based Process Inspector.
In addition, instances cannot be terminated via the browser based Process Inspector and when an attempt is made to do so, the following message appears in SystemOut.log



E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet BPMInspectorRestFacade in application IBM_BPM_ProcessAdmin_PSSIT001.AppTarget. Exception created : javax.servlet.ServletException: javax.servlet.ServletException: com.ibm.processinspector.rest.ProcessAdminRestException: com.ibm.processinspector.rest.ProcessAdminRestException: {"status":"error","Data":{"status":"error","exceptionType":"com.ibm.bpm.wle.api.NotAuthorizedActionException","errorNumber":"CWTBG0549E","errorMessage":"CWTBG0549E: You are not authorized to perform the 'abort' action.","errorMessageParameters":["abort"],"responses":null}}

This problem occurs because the started instance has it's TIP flag set as T (true) in the LSW_BPD_INSTANCE table.
As an experiment you can change this flag manually and you'll notice the instance now appears as expected in PD and can also be terminated as expected in the browser based PI.

This is a know bug in 7.5.* and 8.0.0.0. The APAR details are here.
V8 - http://www-01.ibm.com/support/docview.wss?uid=swg1JR44597
V7 - http://www-01.ibm.com/support/docview.wss?uid=swg1JR43362

I applied 7.5.1.0-WS-BPM-IFJR43362 to my V7.5.1 environment and it appeared to work well except (oddly) for one particular snapshot. I haven't had a chance to investigate further.

Nudging stuck task in BPM 7.5 Event Manager

If you have stuck tasks in the EM async queue, setting the EM instance to null and the task status to 1 followed by a pause and resume of all EM instances will cause the other EM instance/instances to pick up the tasks. It still means the EM instance in question won't pick up further jobs as at this stage it's broken. Maybe i'll be able to work out why it breaks and how to fix it at some point, but for now this workaround will get you out of trouble.

Here's the database query you need to do this

UPDATE lsw_em_task SET task_status = 1, task_owner = null WHERE task_status = 3 AND task_owner = <the id of your EM instance>


Note1: In SQL Developer the date columns in the Lombardi tables only display the date. To display the time as well you need to run something like select to_char(scheduled_time, 'hh24:mi:ss') from lsw_em_task. (or you could always change the NLS settings in SQL Developer to ensure DATE columns display timestamps Tools > preferences > database > NLS - in the Date format field specify DD-MON-RR  HH24.MI.SS
Note2: The ID of your EM instance(s) can be found in the LSW_EM_INSTANCE table

Install a BPD template from the command line

I had a situation where a normal deployment carried out from the Process Centre screen failed. The ZIP archive was reaching the target Process Server (/tmp/import.zip672953119225633327) but the subsequent unpack and import was failing. The error was pretty vague so I decided to attempt a deploy on the target Process Server using the ZIP file directly. The install still failed but it's worth documenting the command for future reference:

/opt/bpm/bpm751/BPM/Lombardi/tools/process-installer/installProcessAppPackage.sh -host localhost -port <WC or webserver port> -profileName node1 -nodeName node1 -serverName server1 /tmp/import.zip672953119225633327 <user> <password>

How to trick Process Designer into connecting to a Process Server that you have not registered

There might on occasion, be the need to connect your Process Designer to a Process Server that is not registered with the Process Centre that your PD's eclipse.ini is pointing at. Let's say for example that you need to give your developers access to a Process Server for debugging purposes only but you do not want them to be able to deploy to it.

The answer is fairly straight forward, you simply insert a row in the Process Centre's LSW_SERVER and LSW_SERVER_CAPABILITY tables that relates to the said Process Server making sure that you set its IS AVAILABLE column to false. Once this is done you'll notice that the Process Centre will have a new (albeit unavailable) Process Server in it's Servers list. 

Now, when and Process Designer instances are connected to the said Process Centre, they will be able to connect to the new Process Server via the drop down list in the Process Inspector screen.

From Evernote:

Authorisation for deploying from a Process Centre to a Process Server

Users with read access to the process app can deploy to dev Process Servers

Users with write access to the process app can deploy to test Process Servers

Users with admin access to the process app can deploy to prod Process Servers

This can be overridden by adding a group (ldap based or local) to 100Config.xml on the Process Server

(note, this even overrides tw_admin's ability to deploy) 

<server merge="mergeChildren">

        <process-center-install-group merge="replace">uk_bpm_foo_users</process-center-install-group>

</server>  

Note that in the example above uk_bpm_foo_users is an LDAP group but that the full DN is not required

How an IBM BPM 75 Process Centre keeps track of the Process Servers it manages

If you have a Process Centre managing several Process Server environments, you might wonder where the Process Centre stores data about the hostnames and port numbers for these environments.

The answer is that it stores this in it's BPMDB database in a table called LSW_SERVER.
This table contains a single line per Process Server containing, amongst other things, the TCP port number that it should be contacted on for deployments etc. (errr.... it also contains the plain text password of the tw_author account so make sure you dont let too many people look at it)

Where does this information originate from I asked myself. Well, it turns out the Proces Server reads it from it's 99Local.xml (or 100Custom.xml if you've chosen to overide things in the proper fashion)  and sends it with every heartbeat to the Process Centre  (good old Wireshark told me this) - you can confirm this if you manaully update the port number in the database table and watch up magically change back a few minutes later :)

 Unfortunately 99Local.xml is not commented awfully well. The section in quesion is....

  <server-name>ProcessServer01</server-name>

  <server-description>A running process server</server-description>

  <server-host>my_proc_server_hostname</server-host>

  <server-port>9443</server-port>

When I first saw this in the Process Servers's own 99Local.xml it didn't make sense, after all, why would you tell the Process Server about itself I wondered. Now I realise this is the data that get's sent in the heartbeat and subsequently ends up in the LSW_SERVER table.

NOTE: if you do decide to follow best practice and define this in 100Custom.xml.. remember to add the required attributes (highlighted in red below)

  <server-name merge="replace">ProcessServer01</server-name>

  <server-description merge="replace">A running process server</server-description>

  <server-host merge="replace">my_proc_server_hostname</server-host>

  <server-port merge="replace">9443</server-port>

Process Inspector and SSL trust

I had the whole SSL trust model sorted when I was running BPM 7.5.0, I even had Client auth set up and for this to work properly the Application Server had to trust the IHS plug-in.

Imagine my suprise then after I upgraded to BPM 7.5.1,  navigated to the ProcessAdmin screen, pressed the new "Process Inspector" button and realised (after checking the application server logs) that the Application server was complaining that it didn't trust the webserver cert!!!!... that's right, not the plug-in cert but the webserver cert!  .

The only explanation was that the App Server must be making a call (i.e as an SSL client) to the webserver. A quick check using Wireshark confirmed this.

So.. you need to make sure your WAS Cell trusts the webserver certificate (or it's signer chain)

My assumption at this moment is that a Dojo server side component is responsible for the call, and it's getting the webserver hostname / ipaddress from the clientside Dojo running "on the glass"

BPM Version 8 Standard Edition Databases

Version 8 Standard Edition uses 4 Databases

1. BPMDB - The Process Centre / Process Server databases
2. PDWDB - The Performance Data Warehouse Database
3. BSPDB - The Business Space Database (New in version 8, Business Space used to only be part of Advanced in v 7.x)
4. MEDB - The Messaging Engine Database (In v7.x the ME tables were schema's under PDWDB and BPMDB

For some reason the documentation states that the Common Database is needed, but as far as I can tell it's not. There are no datasources configured for it during any part of the installation.

Example DB2 commands for creating the databases are as follows:

db2 create database <Database Name> automatic storage yes  using codeset UTF-8 territory GB pagesize 32768;

db2 connect to <Database Name>;

db2 grant dbadm on database to user <user name>;

db2 UPDATE DB CFG FOR <Database Name> USING LOGFILSIZ 4096 DEFERRED;

db2 UPDATE DB CFG FOR <Database Name> USING LOGSECOND 64 DEFERRED;

db2 connect reset;

Once the databases have been created, you can create the tables and stored procedures within by using the scripts generated when you imported and generated the deployment environment:
(Note - create database scripts are provided for all but MEDB, however in most cases your DBA's will want to create the databases, apply any table space settings specific their standards (as above) and set you up with a user with rights for you to create the tables etc. (Not sure why none were provided for MEDB by the way. I guess somebody just forgot ;) )

BPMDB <dmgr profile>/dbScripts/PerformanceDW/DB2/BPMDB 
 

• db2 connect to BPMDB 
• db2 -tvf createTable_ProcessServer.sql
• db2 -tvf createProcedure_ProcessServer.sql
• db2 terminate  

PDWDB <dmgr profile>/dbScripts/PerformanceDW/DB2/PDWDB

• db2 connect to PDWDB 
• db2 -tvf createTable_PerformanceDW.sql
• db2 terminate  

BSPDB <dmgr profile>/dbScripts/PerformanceDW/DB2/BSPDB

A wrapper script is provided, however I've forgotten it's name...
Will follow


MEDB

The MEDB DDL needs to be generated by the SIBDDLGenerator.sh script as follows

sibDDLGenerator [ -systemDBMS name ][ -versionDBMS version ][ -platformDBMS platform ][ -schemaschema name ][ -useruser name ][{ -create -drop }][ -databasedatabase ][ -storagegroupstorage group ][ -cataloghigh level qualifier ][ -bufferpoolbuffer pool ][ -statementendstatement end ][ -nolinebreaks ][ -noblanklines ][ -firstlinefirst line ][ -lastlinelast line ][ -permanentnumber ][ -temporarynumber ]

You'll need 2 sets of DDL, one for the Process Server ME's and one for the PDW ME's. 
I usually create a directory alongside the ones generated for the other databases (e.g. <dmgr profile>/dbScripts/PerformanceDW/DB2/MEDB) and then run the above command twice substituting the schema names each time to match the ones you specified in your Deployment Environment, and redirecting to a suitably named file each time