Collect, visualize and analyze external data in PeopleSoft

January 5, 2022 | 6 minute read
Rajesh Shankare Gowda
Principal Software Engineer
Text Size 100%:

PeopleTools 8.59 introduced an External Data Integration feature as part of the Search Framework. Powered by Logstash, we can now collect, visualize and analyze external data within PeopleSoft. The following post will demonstrate how we can get started!
 

What is external data?

External data can be defined as the data available outside of a PeopleSoft application environment. This is secondary supplemental data that is not part of PeopleSoft transactions. This data may be available in various systems outside PeopleSoft and would be residing in data stores of different sizes and shapes.
 

What is the use of external data?

This data would be useful for various purposes for PeopleSoft users like:

  • Business decision making
  • Visualization and analytics
  • Supplementing data available in PeopleSoft

Some of the examples of external data include

  • Information about a candidate from social media sources like Linkedin, Facebook after obtaining consent from the candidate.
  • Information about co-pay policies which will be available on external sites.
  • Payroll customers would like to get details of company products.
  • Customers would like to do Cash Discounting / Dynamic discounting comparative to the Federal funds rate.
  • FSCM customers using Maximize collections would like to get data from external sources like Moody’s, S& P, or D&B.
  • COVID information (active/recovered numbers) by province.


External Data Integration

The External Data Integration feature in PeopleTools 8.59 allows customers to fetch external data following a few simple steps of configuration. Peoplesoft uses Logstash-7.10 delivered as part of ELK DPK for this feature. The DPK gives you the option to install Elasticsearch 7.10, Kibana 7.10, and Logstash 7.10 together or separately.


Would there be an additional load on PeopleSoft?

No, fetching of external data bypasses the PeopleSoft system and external data will directly be indexed into the Elasticsearch server using Logstash.


External Data Configuration

The “External Data Source Setup” activity guide under “Peopletools > Search Framework > Deploy External Source Defns” collates all the configuration steps needed for external data setup.

External data configuration is broadly divided into External Definition and External Data Source. The External Definition represents the external data in PeopleSoft and includes attributes, PeopleSoft data mapping, and security. The External Data Source contains the information about the external source data collection (input file or URL) and where we write the collected/processed data (Elasticsearch server or output file).

Following are two steps to configure the External Data Definition

  1. Create External Data Definition
  2. Deploy External Data Definition

Following are two steps to configure the External Data Source

  1. Create External Data Source
  2. Manage the External Data index


Create External Data Definition

External Data definition is configured using the “External Data Search Defns” page. External Data definitions define the following:

  • All the required data parameters to fetch, their types, and appropriate description.
  • Mapping an existing search definition/attribute.
  • Specify the security at the source level or component level.

The following page shows a sample External Data definition needed to collect COVID information from an external website.
 


Deploy External Data Definition

The External Data definition can be deployed using the “Deploy External Search Defns” page. The External Data definition created can be deployed/undeployed to the Elasticsearch server with an internally generated index name using the “Create/Delete index” button. The following page shows the sample index name.
 


Create External Data Source

The external data source is configured using the “Define External Data Sources” page. We can create a Data Source by giving an appropriate name and associating it with the desired External Data definition.

Following are the parameters to be configured on the External Data Source page.

  • External URL or a file where data will be fetched
  • Header parameters
  • Proxy information
  • Additional options like certificate (if any)
  • Filters and Transformations can be either PeopleSoft or external
  • Split Fields
  • Data Source Attribute Mappings contains the mapping of external data source attribute to the target attribute in the External Data definition
  • Output can be either file or Elasticsearch server
     


Manage the External Data Index

The “Manage External Search Index” page contains information about scheduling the external data index. The external data is fetched in this step using the Start/Stop button. When we select Start, Logstash initiates the indexing process as per the selected schedule. When we select Stop, Logstash aborts the indexing process. We can use the calendar icon to access the Schedule Definition page to set up the time and frequency of the indexing process.

The following image shows the Logstash pipeline configuration detail once the Start button is pressed.
 


The following page shows different scheduling options.
 


The resulting external data fetched can be used to create visualizations in Kibana. The visualizations created in Kibana can be configured as a Tile or as Related Information on a component using the “Configure Kibana Dashboards” page.

External Data displayed as Related Information on a component:

 


External data displayed in modal when opened from the component related information frame:
 


 

External data displayed as a Tile visualization:
 


 

External data displayed in the Dashboard view when we click on the Tile:
 


In summary, the external data integration feature provides the ability to extend and supplement PeopleSoft applications with external data.

Rajesh Shankare Gowda

Principal Software Engineer