OCI APM and Logging Analytics improve stack visibility with continuous workflows

Application monitoring involves data sources like traces and logs available in dedicated services like OCI’s Application Performance Monitoring (APM) and Logging Analytics. Ultimately, these sources bring visibility to the same stack. Therefore, correlation and integration between sources are critical for holistic monitoring and continuous workflows from one service to the other – like combining shards of glass into one window. OCI APM has released new features to strengthen its correlation and collaboration with other services:

Log injection: APM Java agents can now inject trace and span IDs to Java application server logs. Explore traces in APM and correlate them with logs sharing the same IDs.
Custom drilldowns: Create dynamic links for spans to drill down to any relevant data in any service outside APM. Use this with, for example, external monitoring or ticketing services and build efficient workflows from APM to other tools.

This blog demonstrates how to take advantage of both features, use the APM Java agent to inject span/trace IDs in logs and combine it with a custom drilldown as a shortcut to those very same logs in OCI Logging Analytics. Familiarity with OCI APM and Logging Analytics is assumed.

APM Java Agent Log Injection expedites troubleshooting

Based on distributed tracing, APM collects traces as end-to-end chains of operations known as spans – from a page click, update, or load in a user’s browser to related calls and method executions in the back end. APM’s Java agent captures the spans in Java application servers. All spans and traces have IDs. The APM Java agent can inject these IDs into application server logs. This reveals which traces and spans are related to which logs for quicker troubleshooting.

Log injection is supported for:

Apache Log4j, Log4j 2, and Logback
Java.util.logging
WebLogic Server Access Logs
Mapped Diagnostic Context (MDC)

Enabling log injection is straightforward:

After provisioning and deploying the APM Java agent (more about that here), go to oracle-apm-agent/config/<VERSION>/ProbeConfig.acml
Set the LOG_LIB probe to “true” on the list of enabled_probes:
# If you need to disable a particular probe, set the value to false.
enabled_probes: SERVLET: true ASYNC_SERVLET: true JDBC: true HTTP_CLIENT: true SPRING: true CUSTOM: true JAXRS_SERVER: true JAXRS_CLIENT: true SOA: true INJECTION: true OSB: true LOG_LIB: false ➡️ true

Enable the relevant log injection in the LOG_LIB section of the same file – e.g. to enable WLS access log injection:
LOG_LIB:
[...] # Configuration for injection into the WLS access log - NAME: "WLS_ACCESS_LOG" ENABLED: FALSE ➡️ TRUE FORMAT_STRING: "{ORACLE.APM.SPANID} {ORACLE.APM.TRACEID}" APPEND: TRUE [...]

NOTE: Log injection comes with APM Java agent version 1.7. If the log injection probe and section are not visible in ProbeConfig.all or the version number in the file path is below 1.7, make sure the agent is upgraded (more about that here).

APM Custom Drilldown to Logging Analytics

While log injections correlate traces and logs, the new custom drilldown feature takes you straight to those logs using dynamic links. OCI Logging Analytics can create a query URL for desired log results (more about searching logs here).

Below is a WebLogic access log with trace and span IDs injected into the Supplemental Detail field in Logging Analytics. Click on the IDs and add extended fields to the log source. This way, the IDs are parsed as separate attribute values:

Figure 1: Adding extended field definition based on Supplemental Detail value in OCI Logging Analytics

Insert the following extract expression for Trace ID and Span ID:

{Span ID:(\S*)} {Trace ID:(\S*)}

Figure 2: Add extended field definition for Span ID and Trace ID in OCI Logging Analytics

NOTE: If other information is added to Supplemental Detail, the suggested extract expression might capture it as Span ID or Trace ID fields. To avoid this, add static text as suggested by Logging Analytics to the format_string in the LOG_LIB configuration of ProbeConfig.acml:
FORMAT_STRING: "spanId={oracle.apm.spanId} traceId={oracle.apm.traceId}"

Now a more specific extract expression can be used for the extended field definition:
spanId={Span ID:(\S*)} traceId={Trace ID:(\S*)}

Create a query that looks up logs with a specific trace ID and copy the query URL from the Actions-drop down menu. I have replaced the trace ID value with “0000” and will get back to why later:

Figure 3: Copy URL for Trace ID query in OCI Logging Analytics

The query URL is perfect for a custom drilldown in APM. Under Administration in APM, select your APM domain. Then go to Drilldown configurations under Resources. From here you can manage all drilldowns. Several are already provided out-of-the-box by Oracle:

Appserver Dashboard: Drill down to the span’s related app server metrics
Autonomous Database: Drill down to the span’s related SQL ID in the ADB Performance Hub
Perfhub: Drill down to the span’s related SQL ID in Database Management’s Performance Hub
SQL Warehouse: Drill down to the span’s related SQL ID in Operation Insights’s SQL Warehouse
Traces in Session: Drill down to all traces belonging to the span’s session in APM’s Trace Explorer

Figure 5: Drilldown configurations section under domain administration in OCI APM — Figure 4: Drill down configurations section under domain administration in OCI APM

NOTE: Remember to follow the instructions in the description of each drill down created by Oracle to enable them properly.

Click Create drill down to get started. Enter a name for the drill down and paste the copied query link from Logging Analytics into the URL field. Replace the trace ID value from the query with <TraceId>, so the drill down dynamically links to different logs for different trace IDs. Using a string like “0000” makes it easier to find the relevant part of the URL. Click Save to complete your drill down:

Figure 6: Create Drilldown for Trace ID query in OCI APM — Figure 5: Create Drilldown for Trace ID query in OCI APM

NOTE: If the drilldown takes place within the OCI tenancy (e.g. from APM to Logging Analytics), only the path and query components of the URL are required as shown in Figure 5. It is of course also possible to use the full URL.

You will see all enabled drilldowns in Span details when investigating data in Trace Explorer. You can use any attribute listed here or under the details of other spans including TraceId, SpanId, DbOracleSqlId, OkeClusterId, and many others.

Figure 7: Span with dynamic log drilldown for Trace ID in OCI APM — Figure 6: Span with dynamic log drill down for Trace ID in OCI APM

Another low-hanging drilldown could be for ECIDs. These are both collected by the APM Java Agent and reported in logs:

Figure 8: Span with dynamic log drilldowns for Trace ID and ECID in OCI APM — Figure 7: Span with dynamic log drilldowns for Trace ID and ECID in OCI APM

Here is another example for drilling down to an error message in Jira:

https://jira.oci.oraclecorp.com/issues/?jql=text ~ "<ErrorMessage>"

Conclusion

Enable OCI APM’s application server log injection to create meaningful relations with OCI Logging Analytics or similar services. Enhance application monitoring and management further with APM’s custom drilldowns to build continuous workflows from APM trace data to other critical tools.