Re-imagining edge data analysis with LLMs and open-source technologies

May 20, 2024 | 9 minute read
Text Size 100%:

During season two of the Accenture Oracle Mastermind Hackathon, our diverse team of Solution Architects and Engineers set out to explore the potential of large language models (LLMs) and open source, cloud native technologies. Our goal was to revolutionize how industries approach edge data analysis, inference, transmission, and transformation.

With OCI, we offer a strategic and innovative approach, enabling our clients to leverage the power of cloud computing and artificial intelligence to drive transformation and stay ahead of the curve. Our focus on edge computing showcases our commitment to bringing cutting-edge technology to our clients, helping them unlock new possibilities and solve complex problems with this proposed solution: Multi-modal raw signals intelligence capture and secure transmission. This assistant is designed to work with various data formats, from .tiff images to .mp4 videos and syslogs, demonstrating its versatility and adaptability.

Our mission was to re-imagine the end-to-end IT infrastructure and application deployment challenges emergency responders face, and to enable them to maximize the GenAI value they can derive at the edge, in a distributed architecture.

For a quick brief on the power of Oracle Cloud Infrastructure (OCI) Roving Edge, watch OCI Roving Edge Infrastructure.

Primary business case

Particularly in regions like the western US, energy and utility companies face significant challenges in maintaining compliance with vegetation management regulations while mitigating disaster risks. With the increasing threat of wildfires and the potential for contamination, they have a critical need for innovative solutions. This business case presents an integrated approach combining OCI Roving Edge, Cohere’s AI capabilities, and next-generation drones with advanced image capture technologies, including high-resolution cameras and LiDAR, to revolutionize vegetation management and enhance disaster relief efforts.

Challenges and opportunities

The use case presents the following challenges and opportunities:

  • Vegetation management and Compliance: Regulations require utilities to maintain clearance distances between power lines and vegetation to prevent encroachment, helping ensure safety and reducing fire risks. Non-compliance can lead to fines and legal issues.
  • Wildfire and contamination risks: Improper vegetation management contributes to wildfire hazards, and contamination events can have severe environmental and health impacts. Early detection and mitigation are crucial.
  • Manual processes and limited resources: Traditional manual inspections are time-consuming, costly, and prone to errors. Utility companies often struggle with limited resources and coverage, especially in remote areas, making it challenging to keep up with vegetation growth.
  • Need for real-time data and insights: Dynamic vegetation changes require real-time data and alerts to enable proactive maintenance and targeted interventions, reducing disaster risks and helping ensure compliance.

Benefits and value proposition

  • Enhanced disaster mitigation: The solution reduces wildfire risks and facilitates early contamination detection, minimizing potential impacts and enhancing grid resiliency.
  • Improved compliance and cost savings: AI-powered vegetation management supports compliance, helping to avoid fines. Optimized maintenance reduces operational costs and improves resource allocation.
  • Increased safety and sustainability: Proactive vegetation management protects communities and the environment, reducing wildfire hazards and potential contamination impacts.
  • Scalability and adaptability: The integrated solution is scalable and adaptable, catering to diverse terrain and regional challenges, which enables efficient vegetation management across service areas.

Solution Architecture:

Our solution centers around an agnostic approach to raw source data, utilizing the Cohere API for Enterprise AI. By employing OCI Roving Edge, we enable edge data to be analyzed, inferred, and transformed at the source. When the data reaches an OCI or multi-domain cloud region, we can perform intensive batch transformations and streams or create event workflows for further data manipulation and enrichment. This processed data can then be consumed by enterprise, low-code, and third-party applications through familiar interfaces.

 

Architecture diagram showing edge AI intelligence synopsis capability with downstream data integration to various target systems

 

Results

We demonstrated how customers can use generative AI for raw data input and assistant queries through an LLM (ithis project, Cohere) for edge device administrators. You can quickly synchornize images from a vegetation management operation into OCI Object Storage and subsequently query them using retrieval-augmented generation (RAG) to aid the operations team.

 

Synopsis Subscriber on OCI, showing the satellite connectivity feed.

 

Bucket details of edge image data in OCI object storage within a compartment in Oracle Cloud

Retrieval augmentation generation AI search for eucalyptus trees in vegetation management in OCI.

Components:

Our solution uses the following components:

  • OCI Roving Edge: OCI Roving Edge is a powerful edge computing platform that enables data processing and analysis directly at the source. It receives data from various sources, including drones, satellites, and CCTV cameras, and performs initial processing and analysis using Cohere's AI models. OCI Roving Edge devices are deployed in proximity to data sources, providing low latency and real-time insights.
  • Cohere AI: With its cutting-edge AI capabilities, Cohere powers the vegetation management system. Cohere's natural language processing and computer vision models analyze images, LiDAR data, and textual information. It identifies vegetation encroachments, classifies plant species, detects changes in vegetation health, and predicts potential risks. Cohere's AI models are continuously trained and improved, enabling accurate and reliable insights.
  • OCI Object Storage: OCI Object Storage serves as a scalable and secure data repository for the solution. It stores raw and processed data, including images, LiDAR scans, and vegetation analysis results. OCI Object Storage provides durable and cost-effective option, enabling efficient data retrieval and management.
  • Autonomous Database: Oracle's Autonomous Database is a self-driving, self-securing, and self-repairing database platform. It handles the storage and processing of structured data within the solution architecture. The database manages information related to power lines, vegetation clearance regulations, maintenance records, and compliance documentation. Autonomous Database provides data integrity, high availability, and fast query performance.
  • GPU Acceleration: GPU acceleration technology boosts the performance of AI workloads and image processing tasks. It accelerates compute-intensive operations, enabling faster and more efficient data processing, especially for computer vision and deep learning models utilized in vegetation analysis.
  • Golang and Python Development: The solution uses Golang for building high-performance, scalable microservices and handling edge computing tasks. Golang’s efficiency makes it ideal for real-time data processing. Constratively, we use Python, for AI model development and integration, taking advantage of its extensive data science and machine learning libraries.
  • Image capture and LiDAR scanning: Advanced image capture technologies, including drones, satellites, cameras, and CCTV systems, provide visual data for vegetation analysis. Drones equipped with high-resolution cameras and LiDAR scanners capture detailed images and create precise 3D models of power lines and surrounding vegetation. LiDAR technology provides accurate distance measurements and terrain data.
  • Client interface and integrations: The solution offers a flexible client interface, integrating with commonly used enterprise systems such as Fusion enterprise resource planning (ERP), SCM, and Palantir. It also supports seamless integration with third-party applications through APIs and data exchange formats, which helps ensure that utility companies can access and utilize the vegetation management insights within their existing workflows and systems.
  • OCI tenancy and edge messaging: The solution operates within a secure OCI tenancy, utilizing OCI’s robust infrastructure and services. An edge messaging and communications service, nats.io facilitates real-time data exchange between OCI Roving Edge devices, allowing reliable and efficient messaging between distributed components.
  • Hybrid cloud and additional other LLM vendors (optional but recommended): You can extend the architecture to a hybrid cloud or multiple clouds. This model offers flexibility and enables data exchange with other LLM vendors, such as Mistral, Llama 3, Claude, OpenAI, and Grok, for specialized AI capabilities and language models.

Edges services:

For our edge software fleet, we selected a powerful combination of tools, including Synopsis and YOLOv9, to ensure efficient and accurate data processing on video and image capture. We also developed custom extract load transform (ELT) services to seamlessly push data into preferred data stores, such as OCI Object Storage and Autonomous Database.

Synopsis is a lightweight (<50MB) Golang-based tool that provides a command-line interface (CLI). Is designed to support emergency responders with raw inquiry capabilities, enhanced by Cohere's R+ and Grounding features. Synopsis can also generate near-instant executive synopses and intelligence reports from raw edge data, such as syslogs and errors. You can securely transmit these reports downstream to critical operators and SOC systems, allowing timely and encrypted communication.

YOLOv9 is an industry-leading Python-based library for real-time object detection. It excels at performing edge inference on video streams and image capture, making it ideal for detecting objects or anomalies in visual data.

Custom integration services were built using Python and Golang, although any preferred programming language would suffice. These services offer flexibility in data pipelines and workflows. You can directly upload data to object storage, supporting multi-tenant and hybrid cloud environments, or ingest it into databases like Autonomous, Postgres, and others. Advanced concepts such as transform-extract-transform, data hydration, and enrichments are also supported, enabling data to be prepared and optimized for analysis and reporting.

Lessons learned and potential improvements:

As we delved into this project, we quickly recognized the importance of adopting GitOps tooling to streamline the process of creating trained LLM binaries for developer ecosystems. By reducing the barriers to entry, we can empower developers to innovate and drive further breakthroughs. Looking ahead, we envision a future where multi-modal LLM APIs are seamlessly embedded into service fleets, revolutionizing edge AI telemetry, intelligence extraction, audit, and compliance with order-of-magnitude performance enhancements.

To continue pushing the boundaries, we identified the following key areas of focus:

  • Data provenance and security: We plan to leverage use metagraph technology to guard the integrity of our edge data, preventing data tampering and spoofing attempts.
  • Optimized data transmission: Further integration of Benthos and NATs can enable direct and efficient data transmission into Oracle systems - with enhanced mapping, hydration, and multiplexing capabilities.
  • Network security enhancements: Implementing private path networking through FastConnect can reduce the attack surface and enhance data security, minimizing potential vulnerabilities.
  • Cloud Guard and SIEM integration: Enabling Cloud Guard and enhancing SIEM flow with comprehensive audit and service logs can bolster security and compliance measures. We also want to define zero-trust access control for the full OSI Model, visualizing identity and access management (IAM) and access governance.
  • Low-code app development: By using Palantir AIP, we can accelerate enterprise app development, expanding data collection into the OCI Database and Storage services.
  • Standardization: We plan to standardize the landing zone architecture across OCI regions, facilitating seamless application adoption and workload migration.

Conclusion

By combining cutting-edge LLMs with open-source cloud-native technologies, our team has demonstrated a transformative and agile approach to edge data analysis. This project showcases how Oracle is innovating and pushing the boundaries of what's possible with cloud-native edge AI topologies, tailored for industry-specific use cases.

We’re proud to showcase the art of the possible with our agnostic cloud-native edge AI architectures, fine-tuned for diverse industry verticals support mission-critical operators that protect, and to manage the most important aspects of humanity’s relationship with the environment.

Oracle emergency response vehicle with an Roving Edge device connected to starlink and OCI Chicago Region

 

To explore more Oracle Cloud Infrastrcuture Roving Edge use cases and get inspired, check out this blog on Oracle role in maintaining Global Security – Oracle successfully demonstrates tactical solutions in NATO exercise.

 

Special shoutout to our talented team and partners

Oracle:

  • Tobalo Torres-Valderas, chief architect
  • Saipriya Thrivakaldu, senior solution architect, Cloud Native
  • Matt Leonard, VP OCI Roving Edge and product owner
  • Edgar Vasquez, director of cloud architecture
  • Aby Joy, principal architect, Manufacturing
  • Tiffany Johns-Abram, senior solution architect, Innovation Lab
  • Marek Kratky, prinicial architect, RAG and AI
  • Surya Komareddy, director of Industry Solutions
  • Carine Cordahi, cloud platform rep, Innovation
  • Michael Reed, edge principal architect, Oracle Solutions Center

 

Accenture:

  • Janusz Jeske, idea owner & chief Accenture architect
  • Ken Avery, senior architect
  • Adam Alpesch, cloud engineer
  • James Gress, director of genaritve AI

 

Synadia:

  • Steve Dischinger, vice president
  • Delaney Gillian, architect

Tobalo Torres-Valderas

Architect


Previous Post

Streamlining identity management: Oracle Identity Manager Connector for SAP Ariba

Kamal Narayan | 3 min read

Next Post


Innovate with Oracle Autonomous Database on Oracle Database@Azure

Marty Gubar | 2 min read