Wallaroo.AI, a leading player in the AI inference space, has partnered with Oracle Coud Marketplace to provide a seamless experience for organizations leveraging Oracle Cloud Infrastructure (OCI). As an Oracle Partner, Wallaroo.AI’s integration with the OCI stack represents a significant step forward for enterprises looking to enhance their AI capabilities effectively and efficiently.

The Wallaroo universal AI inference platform enables realtime and batch predictions on any hardware type (CPU, GPU), and across various AI applications (Time Series, Computer Vision, Classification, Regression and LLMs) in any cloud and at the edge.

The Wallaroo AI Inference Platform model operations center enables enterprise AI teams  to quickly and efficiently operationalize AI at scale through an end to end ML and LLM lifecycle from deployment, and ongoing model management with full governance and observability while realizing optimal performance scaling across CPU & GPU architectures in the cloud, on-prem and at the edge.

The Power of Partnership: Wallaroo.AI and Oracle

The partnership between Wallaroo.AI and Oracle underscores a mutual commitment to empowering enterprises with cutting-edge technology to realize the strategic business potential of AI.

Oracle, with its expansive cloud infrastructure and enterprise solutions, provides the ideal platform for organizations to rapidly build and deploy AI models at scale. Wallaroo.AI, known for its expertise in AI model deployment, inference and operations, complements Oracle’s offerings by streamlining the process of getting AI apps from development to production.

This partnership is more than just a technological integration; it’s a strategic alliance that enables businesses to accelerate their AI initiatives and time to value. By leveraging Wallaroo.AI’s inference platform capabilities within the OCI ecosystem, organizations can deploy AI models with greater speed, efficiency, and effectiveness reducing the time it takes to derive actionable insights, and value from data.

Wallaroo and Oracle Cloud Infrastructure: Better Together

GenAI adoption has been a bumpy ride for a great deal of organizations due to underestimating the time, effort and cost it typically takes to get effective, reliable, and robust LLMs into production. With Wallaroo and OCI, enterprises can overcome this and easily operationalize LLM models and put in place measures using RAG (Retrieval-Augmented Generation) and Wallaroo’s LLM Listeners, to help ensure model integrity, and the staples of security, privacy, compliance for avoiding outputs with toxicity, hallucinations etc.

For LLM output validation with RAG, Wallaroo natively enables the deployment and management of embedding models available in the HuggingFace board, directly on Ampere, without any hardware or infrastructure manipulation. Additionally, Wallaroo’s inference workload automation capabilities and integrations toolkit provide support to most vector databases today, enriching vector databases with more context and embeddings, on demand and on a schedule, as new data becomes available. As a result, AI teams building and deploying LLMs in production can ensure a tight feedback loop and continuously improve the quality of LLMs running in production with low operational burden.

For LLM input and output monitoring for toxicity, Wallaroo enables the deployment and monitoring of pre-built and custom LLM listeners to score LLMs against specific metrics such as toxicity, harmful language etc. Wallaroo’s LLM listeners are compatible with ARM architecture and can be deployed on Ampere-based machines in OCI.

For LLM deployment and inference on Ampere, Wallaroo takes advantage of our full llama-cpp integration on Ampere to ensure Wallaroo’s compatibility with the data and AI ecosystem in OCI and highlight the operational efficiency we can deliver with the integration.

LLM Operations for deployment, validation, monitoring and management in Wallaroo can be achieved in OCI via two methods:

  • Through implementing RAG LLM as an authoritative source for our model to enhance the reliability of the generated text and help ensure that the generated text is up-to-date, and free from potential issues such as hallucinations and toxicity helping to avoid potential risks and safeguard accurate and relevant outputs.

 

 

  • Wallaroo’s LLM Listeners can also be orchestrated to generate real-time monitoring reports and metrics to understand how the LLM is behaving and ensure that it’s effective in production which helps drive the time to value for the business.

These seamless and flexible AI model operations in Wallaroo and OCI enable AI teams to ensure a tight feedback loop and continuously improve the quality of LLMs running in production with low operational burden.

Explore Wallaroo.AI on the OCI Marketplace

For organizations looking to enhance their AI capabilities, Wallaroo.AI on the OCI Marketplace is a powerful solution. The integration not only simplifies the deployment of AI models but also provides the tools needed to manage and optimize these models effectively. By combining the strengths of Wallaroo.AI and Oracle Cloud Infrastructure, businesses can unlock new possibilities in AI, driving innovation and growth.

To learn more about Wallaroo.AI’s offerings and explore their solutions on the OCI Marketplace, visit Wallaroo.AI on OCI Marketplace to get started or contact us to learn more. 

This blog is co-written by Younes Amar, VP of Product, Wallaroo.AI