Future of Search and the use of ELK

June 23, 2022 | 6 minute read
David Bain
Sr. Director PeopleTools Product Management
Sasank Vemana
Director, PeopleTools Product Management
Text Size 100%:

During the recent conferences – Blueprint 4D 2022 and HIUG Interact 2022, we announced a change to search in PeopleSoft and our use of ELK (Elasticsearch, Logstash, Kibana) as the search and analytics engine behind the Search Framework. The announcement and this blog post are primarily intended as a heads-up to everyone. After all, we want to make this as smooth and seamless as possible for customers. Just know that there is time and more information will be forthcoming as we move along.

In case you missed the announcement, PeopleSoft is transitioning from Elasticsearch 7.10.0 and Kibana 7.10.0 to OpenSearch and OpenSearch Dashboards. Since OpenSearch is an open source version of the Elasticsearch source code, we do not expect any impact. Our recommendation is to continue to implement the delivered search, Kibana dashboards, and real-time indices as planned, and expect them to be compatible when the cutover happens.

Let’s unpack and dive into this! To put this in context, let us first understand the history and the current state of Search in PeopleSoft.

History of search in PeopleSoft


Over the years, we have seen search in PeopleTools shift with the technologies available in the industry. We initially started with Verity. Then, being good corporate citizens, we moved to Oracle SES (Secure Enterprise Search). During this period, search was largely restricted to navigation and certain specialized application data scenarios such as Job Search, etc.

Next, we prepared for the transition to Elasticsearch. During this move, we put a lot more effort and focus on the Search Framework to make the switch of the backend ‘search engine’ as seamless as possible. If we recall, the move from Oracle SES to Elasticsearch was surprisingly smooth - relative to past experiences with search transitions. In fact, we also built a live cutover option in the Search Framework which allowed customers to turn on the new search engine (Elasticsearch) using a simple configuration.

Why did we move to Elasticsearch? Because it was a well-established open source search engine in the industry that not only provides powerful search capability but also a high level of scalability. More on Elasticsearch can be found here.

Since we moved to Elasticsearch, the use cases have exploded, and the uptake of search has also increased significantly. The beauty about this is that the Search Framework and more importantly Elasticsearch provides the scalability for this type of growth.

In PeopleTools 8.58, we incorporated Kibana as part of the Search Framework and provided the UI integration with PeopleSoft. As you can imagine, this further establishes our reliance on the Search Framework as a critical component of the PeopleSoft user experience today. More on the power of Search Framework later!

How does search work right now?


Let’s level set here.

A simplified architecture diagram that illustrates how PeopleSoft applications interact with Elasticsearch (search engine) and Kibana (analytics engine) through the Search Framework. The Search Framework facilitates all interactions related to search and analytics.

As we can see in the simplified architecture diagram above, the interactions of the PeopleSoft application with the backend search (Elasticsearch) and analytics (Kibana) engine, are managed and facilitated by the Search Framework in PeopleTools. Therein lies the power of the Search Framework.

The other important point to note here is that, even though we use the open source version of the ELK stack, customers do not rely on Elastic from a delivery standpoint - for download, installation, or support. In fact, customers must install the version of ELK that we deliver. Why do we mandate this? Because we add our plugins to ELK to facilitate the integration with PeopleSoft which includes (amongst other things) security for search data access control and the ability to embed analytics in the user interface.

It is very important to reinforce the fact that we take a version of ELK, add our plugins, package, and ship that as part of PeopleTools. Customers currently install Elasticsearch and Kibana using the ELK DPK (Deployment Package). Since we package and ship ELK, we support it through the lifetime of that PeopleTools release. This also means we provide any updates and patching necessary for ELK as part of PeopleTools.

This is the current state!

OpenSearch and OpenSearch Dashboards


Before we move on to the planned changes to Search in PeopleSoft, let us review some resources on OpenSearch and OpenSearch Dashboards.

https://opensearch.org/
https://opensearch.org/faq

What is OpenSearch and what is OpenSearch Dashboards? The FAQ resource above provides a great overview. However, they are effectively a fork of Elasticsearch and Kibana as of ELK 7.10.0. This fork is a new open source project driven by a community effort to continue to keep the software open and free for developers to use, modify, extend, embed, and offer as part of their products and services.

Announcement


Moving on to our announcement made during the conferences in 2022 — Blueprint 4D, HIUG Interact, and RECONNECT. Here are the main points:

  • Elasticsearch 7.10.0 and Kibana 7.10.0 will be the last version delivered by Oracle PeopleSoft
  • In the future, we will move to OpenSearch (OS) and OpenSearch Dashboards (OSD)
  • Our commitment is to make the migration as seamless as possible
  • Recommendation is to continue to move forward with the adoption of Search and Kibana Analytics

To be honest, this is the extent of details we can provide at this point as we are working hard behind the scenes to make this as smooth and seamless as possible for us and more importantly for our customers.

That said, why are we making this change? The licensing model adopted by Elastic will make it difficult for us to continue to use ELK in PeopleTools beyond version 7.10. If we did, it would inevitably mean that we would bear licensing costs which will consequently impact our customers. This is not the path we prefer to take!

Also, OpenSearch and OpenSearch Dashboards project is maintained and advanced by an open source community, including a network of partners, and is open to contribution.

These are some of the main reasons why we are moving from ELK to OpenSearch and OpenSearch Dashboards.

OS/OSD Delivery Update (January 16, 2023)

  • OpenSearch/OpenSearch Dashboards (OS/OSD) will be delivered in patches for supported PeopleTools releases
    • Targeted for PeopleTools 8.60 and 8.59 in 2023
  • After the first release of OS/OSD is available:
    • We will continue to support the PeopleSoft technology that utilizes Elasticsearch (ES) and Kibana for a year or as long as the PeopleTools release is supported, whichever is shorter
    • Support will cover PeopleSoft’s integration with and use of ES and Kibana, including indexing, UI integration, PeopleTools plugin, etc.
    • Since we cannot patch ES and Kibana beyond 7.10.0, any fixes directly related to ES and Kibana will not be available
      • This may include fixes to bugs, SVs, etc., related to Elasticsearch 7.10.0 and Kibana 7.10.0
  • The migration to OS/OSD is straightforward with very low impact
    • Currently planned that customers will not even require re-indexing of search definitions or changes to dashboards
  • We encourage customers to move as quickly as possible when OS/OSD is available

Keep Calm and Carry On


As described earlier, customers never really relied on Elastic to install or support ELK in PeopleSoft. It was packaged and managed entirely within PeopleTools. Similarly, OpenSearch (for search) and OpenSearch Dashboards (for analytics) are planned to be packaged (DPK) and managed as part of PeopleTools. We plan to incorporate similar plugins which will include security for search data access control and the ability to embed analytics in the user interface.

Another important point to reiterate is that OpenSearch and OpenSearch Dashboards are effectively a fork of ELK 7.10.0. In that regard, any end-user facing functional changes is expected to be minimal (if any).

Finally, we continue to invest in search and analytics as a major user experience component in PeopleSoft. The power and integration capabilities offered by the Search Framework are illustrated below.

Diagram that illustrates the Power of the Search Framework through capabilities including Global Search, Keyword Search, Data Visualizations, AI/ML Pipeline, and External Data Integration

Until we have more information to share, stay tuned, keep calm, and carry on with your search and analytics implementations!

David Bain

Sr. Director PeopleTools Product Management

Oracle/PeopleSoft employee since 1996.  Most of the time spent working on PeopleTools in some capacity.

Sasank Vemana

Director, PeopleTools Product Management

Oracle/PeopleSoft Enthusiast, Developer, Architect, Strategist, Advocate


Previous Post

Oracle Announces the 2022 Class of PeopleSoft Innovators – PeopleSoft NOW! Episode 2.5

Vaseem Khan | 3 min read

Next Post


PeopleSoft Match Exception Collaboration Centre Provides You With Tools to Clear Roadblocks to Timely Supplier Payments Part II

Sriram Sadagopan | 4 min read