Application Engine Parallel Processing (batch)

Parallel processing comes in picture when there is a requirement to process large amount of rows without compromising the performance which might otherwise get impacted greatly with non-parallel based processing. Sharing this concept with an actual example where this concept has been used.

I had written an application engine program which read data via a SQL object, performed some computations and finally wrote the result to an extract file which was sFTP'ed to the vendor. Pretty much run of the mill stuff. The AE was reading from PS_JOB, PS_PERSONAL_DATA, PS_EMAIL_ADDRESSES tables etc via a single single SQL object in a peoplecode step, performing the data manipulation as required and writing to a file. It was sequentially selecting an employee at a time, performing the data manipulations and writing to the file. To process around 3K employees it was taking a little over an hour, so not a lot of data but too much time to process this data-set.

So to solve this issue of performance, this is what I did. 

1. I created a single TAO or temporary table. The fields in this table are essentially the unique list of fields or values that the single SQL object was returning. Did not build the TAO table yet, will do it in a later step.
2. The TAO table has two keys, PROCESS_INSTANCE and EMPLID.
3. Then before the peoplecode step was called added a new step to select from the various HR tables and write to this newly created TAO table. 
4. SQL step looks something like this.

         INSERT INTO %Table(N_MY_TAO)          SELECT %Bind(PROCESS_INSTANCE)         , B.EMPLID         .....        .....

        .....

        FROM PS_JOB A, PS_PERSONAL_DATA B .....

        WHERE ....

    5. After this added a new step to update statistics on the newly populated TAO table.
        %UpdateStats(N_MY_TAO)
    6. In the Peoplecode step replaced the SQL fetch as follows.
        &EESQL = CreateSQL(FetchSQL(SQL.N_MY_SQL), N_MY_AET.PROCESS_INSTANCE);
    7. I had already defined a state record in my AE which has process_instance field in it, so did not have to do anything different with the state record. In the N_MY_SQL SQL object I am selecting all fields FROM %Table(N_MY_TAO) WHERE PROCESS_INSTANCE = :1 

ORDER BY EMPLID. I have some SQL case statements and formatting rules defined in the SQL itself.

8. Added the newly created TAO table under temp tables program properties and provided an instance count of 1. Instance count has to be 1 or more, if its set to 0 then you will see the following message in your AE log and there won't be much of performance improvement. 

WARNING: NO DEDICATED INSTANCES AVAILABLE FOR N_MY_TAO - USING BASE TABLE. (108,544)

9. Under PeopleTools > Utilities > Administration > PeopleTools Options, in my case the Temp Table Instances (Total) and Temp Table Instances (Online) is set to 3. So when I add 1 as the instance count in my AE and then build the TAO table, it will create PS_N_MY_TAO, PS_N_MY_TAO1, 2, 3 and 4 and when the process runs it will use PS_N_MY_TAO4 as the first 3 are used for online processing. 

10. You can add a step at the beginning or end to purge the TAO table like %TruncateTable (%Table(N_MY_TAO)) or better yet use the check-box under program properties "Use Delete for Truncate Table". With this option the TAO table is purged at the beginning of each run.

Addition of a single TAO table enabled the process to complete in 7 mins. Massive improvement.

Native sFTP transmission in PeopleSoft using RSA Keys

I have been using PeopleSoft's URL definition to setup sftp based transmission based on password authentication successfully. Recently was challenged to use sFTP but with RSA keys. Seemed simple at first but then as I dug deeper found out that there are a lot of nuances that one needs to be aware off. So here is what I learnt so far. I am using PT 8.53.26 for this exercise.

At a high level the steps are
1. Create a RSA public/private key pair. I am using PuTTY generator but there are a lot of other tools out there which do the same.
2. Create a URL definition in PeopleSoft and associate the RSA keys with the same.

So lets look at creating the RSA key pair first.

Launch the PuTTY Key Generator and select SSH-2 RSA (default) and then click on the Generate button. Once complete the public key, key fingerprint and key comment values are displayed. Key comment can be updated to some other value or can be left as-is. Put in a key passphrase.
Save the keys by clicking on the "save public key" and "save private key" button respectively.
Use .pub extension for the  public key and .ppk for the private key. Filename can be anything for your choice. So for this discussion we will call this putty-gen.pub and putty-gen.ppk.

The highlighted text in blue in the above picture is the public key in OpenSSH format so copy that and then paste it to a text file. We will name this file sftpuser. No extension just sftpuser. I will explain shortly the reason behind this naming convention.

Provide putty-gen.pub or sftpuser file to the vendor so that they can add it to their sftp server.
Once vendor sets an account for you they will provide the sftp address, port is generally 22 and an userid. This userid should match the filename, so we will consider that the userid created by the external vendor is called sftpuser. Hence the naming convention. When you send the public key to the vendor you wouldn't know the userid so at that point the file can be called anything, the filename is more critical to the PeopleSoft setup than the vendor's setup.

Via the PuTTY gen tool navigate to Conversions > export OpenSSH Key and export the private key to a text file. Name this file sftpuser.ppk

You can always go back in time load the privatekey "putty-gen.ppk" file in PuTTY Generator and update the private key password.

While the vendor is doing their setup and configuration we have following two options in PeopleSoft.
The first one and the preferred one is to import the keys in the Digital Certificates component as this options stores them in the database. The second option is to store the keys on the file server. We will discuss both the options here.

Digital Certificate Option (preferred option)

Navigate to PeopleTools > Security > Security Objects > Digital Certificates and add a new row. Select Type as SSH, provide some value for Alias and then a "Copy", hyperlink should show up. Click on that link and then paste the contents of sftpuser.ppk and sftpuser in the appropriate boxes and click on Save (which is to the bottom-right of the page).

Next navigate to PeopleTools > Utilities > Administration > URLs and create a new URL.

Add in the sftp URL provided by the vendor. It could be something like

sftp://vendorname.com

or  sftp://vendorname.com/~/drop-directory

or sftp://vendorname.com/drop-directory

The ~ indicates that a successful logon takes you to the home directory and from there you can navigate to lets say a sub-directory called as "drop-directory" where you have to drop the files.

Then click on the URL Properties link

Provide the following properties

AUTHTYPE = 1 - PUBLICKEY 

SSHKEYALIAS = alias set on the digital certificates component ( you can do a lookup here and pick the correct value)

USER = sftpuser (userid provided by the vendor)

PASSWORDKEY = private key passphrase (in plain text not encrypted) 

Option 2 – Keys stored on file system

We have to create some directories under the PS_SERVDIR location. So if the call to transmit the files is going to be made from the application server then in case of de-coupled PS_HOME it would be PS_CFG_HOME\appserv\{Domain} and process scheduler it would be PS_CFG_HOME\appserv\prcs\{Domain} and in case of a non de-coupled PS_HOME like the older way (traditional) it would be PS_HOME\appserv\{Domain} and PS_HOME\prcs\appserv\{Domain}

If you do not want to create sub-directories under here then there is some more information in Peoplebooks to create a new environment variable to point to a new location. 

So under PS_SERVDIR create a directory called sshkeys and 2 sub-directories under that called as private and public. The names of these 3 directories cannot change. Copy the sftpuser.ppk file to the private directory and sftpuser file to public directory. The filename naming convention is primarily enforced for this option. 

Now back to within PIA create a new URL definition. The URL properties is what is slightly different from the previous option. 

Provide the following properties

AUTHTYPE = 1 - PUBLICKEY 

PRIVATEKEY = sftpuser.ppk 

PUBLICKEY = sftpuser

USER = sftpuser (userid provided by the vendor)

PASSWORDKEY = private key passphrase (in plain text not encrypted) 

So that's essentially it. 

Following is the basic peoplecode call to transmit the file. 

&returncode = PutAttachment(URL.N_VENDOR, &filename, &fullpathtofile);

where N_VENDOR is the URL definition created above
&filename is the filename as it test.txt

and &fullpathtofile is the complete path as in C:\temp\test.txt