Friday Jun 28, 2013

New A-Team Web Site Launched

The A-Team has launched a new web site – the A-Team Chronicles which aggregates and organizes content produced by The A-Team members (including your humble blogger).

The A-Team is a central, outbound, highly technical team comprised of Enterprise Architects, Solution Specialists and Software Engineers within the Fusion Middleware Product Development Organization that works with customers and partners, world wide, providing guidance on implementation best practices, architecture, troubleshooting and how best to use Oracle products to solve customer business needs.

This content captures best practices, tips and tricks and guidance that the A-Team members gain from real-world experiences, working with customers and partners on implementation projects, through Architecture reviews, issue resolution and more.

A-Team Chronicles makes this content available, through short and to the point articles to all our customers and partners in a consistent, easy to find and organized way.

If you like the articles we post here, you might find even more interesting articles at the new A-Team Chronicles site, covering a wider range of Fusion Middleware topics. We will be decommissioning this site shortly in favor of A-Team Chronicles site and all new contents will be posted there.

Wednesday Jun 19, 2013

WLS Cluster Messaging Protocols on A-Team Chronicles

Oracle A-Team has launched a new Oracle A-Team Chronicles web site for all A-Team generated content that cover architecture, product features, technical tips, performance tuning, troubleshooting and best practices on Oracle Middleware products.

There is a new blog post on WLS Cluster Messaging protocols in the A-team chronicles. For more details on it and related A-Team recommendations, kindly refer to Weblogic Server Cluster Messaging Protocols.

Tuesday Feb 26, 2013

Solaris on Exalogic - Effect of VNIC over eoib0 & eoib1

There are lots of reason for customer to create VNIC over eoib0 & eoib1 on a compute node running Solaris, two typical examples are

1) compute node needs to connect to a VLAN over the EoIB network

2) there are containers running on the compute node that require 10GbE connectivity

We talked about why Transitive Probe-based Failure Detection is required in previous blog entry, the focus was on the link between IB gateway and customer's 10GbE infrastructure.

In fact, if there are VNIC created over eoib0 and eoib1, there is a chance that bond1 will not fail over even if the link between compute node and IB gateway goes down!

Here is a simple test to illustrate this scenario:

First of all, let's create a VNIC over eoib0 using the following command:

root@el01cn01:~#dladm create-vnic -l eoib0 vnic0

That's what the IPMP groups look like:

root@el01cn01:~# ipmpstat -i
eoib0       yes     bond1       --mb---   up        disabled  ok
eoib1       no      bond1       is-----   up        disabled  ok
bond0_0     yes     bond0       --mb---   up        disabled  ok
bond0_1     no      bond0       is-----   up        disabled  ok

Then we take the link down between compute node and the IB gateway where eoib0 is located, following is what we get:

root@el01cn01:~# ipmpstat -i
eoib0       yes     bond1       --mb---   up        disabled  ok
eoib1       no      bond1       is-----   up        disabled  ok
bond0_0     no      bond0       -------   down      disabled  failed
bond0_1     yes     bond0       -smb---   up        disabled  ok

Notice that bond0 has failover but not bond1. Even the LINK status is still up for eoib0, it has actually lost connectivity to the 10GbE network.

Obviously, the reason behind this behavior is related to the vnic0 that we created over eoib0, from the operating system point of view, the link between eoib0 and vnic0 is still up, therefore no failover of bond1 occurred.

This is another good reason why probe-based failure detection is required.

Friday Sep 28, 2012

OSB, Service Callouts and OQL

Oracle Fusion Middleware customers use Oracle Service Bus (OSB) for virtualizing Service endpoints and implementing stateless service orchestrations. Behind the performance and speed of OSB, there are a couple of key design implementations that can affect application performance and behavior under heavy load. One of the heavily used feature in OSB is the Service Callout pipeline action for message enrichment and invoking multiple services as part of one single orchestration. Overuse of this feature, without understanding its internal implementation, can lead to serious problems.

This series will delve into OSB internals, the problem associated with usage of Service Callout under high loads, diagnosing it via thread dump and heap dump analysis using tools like ThreadLogic and OQL (Object Query Language) and resolving it.

The first section in the series will mainly cover the threading model used internally by OSB for implementing Route Vs. Service Callouts.

The second section of the "OSB, Service Callouts and OQL" blog posting will delve into thread dump analysis of OSB server and detecting threading issues relating to Service Callout and using Heap Dump and OQL to identify the related Proxies and Business services involved.

The final section of the series will focus on the corrective action to avoid Service Callout related OSB serer hangs. Before we dive into the solution, we need to briefly discus about Work Managers in WLS.

Please refer to the blog posting for more details.

Monday Mar 19, 2012

InfiniBand Enabled Diskless PXE Boot

If you ever need to bring up a computer with InfiniBand networking capabilities and diagnostic tools, without even going through any installation on its hard disk, then please read on. In this article, I am going to talk about how to boot a computer over the network using PXE and have IPoIB enabled. Of course, the computer must have a compatible InfiniBand Host Channel Adapter (HCA) installed and connected to your IB network already.

[ Read More ]

Sunday Mar 11, 2012

Configure IPoIB on Solaris 10 branded zone

Compute nodes in Exalogic communicate with each other and mount their shares from the ZFS storage appliance over the bond0 interface. Bond0 is a highly available network interface over the InfiniBand fabric using a portability layer called IPoIB that allows compute nodes and the storage appliance to communicate using TCP/IP protocol over InfiniBand.

In previous entry, we have created a Solaris 10 branded zone, obviously we would also like the branded zone to be able to mount shares from the ZFS storage appliance over IPoIB.

In this entry, I’ll show you the steps.

[Read More]

Create Solaris 10 Branded Zone on Exalogic

One of the reasons that customers choose to run Solaris 11 Express on Exalogic is the capability to create container. Container is a form of Operating System Virtualization that allows multiple operating system environments to coexist on a single system. Container not only allows user to run the same version of Solaris but it is also possible to create a container, also known as branded zone, to run previous Solaris version, such as Solaris 10. It is a very useful feature that enable customer to continue running applications that only certified on older OS version.

In this entry, I will show you the steps to create a Solaris 10 branded zone on Exalogic.

[Read More]

Friday Mar 09, 2012

Advance Routing for Multi-Homed Hosts

Scenario based discussion on enabling access to a host across multiple routers and network. Use of Linux Advance Routing through iproute2 tool set. Comparison on when to use static vs. multiple default routes.

[Read More]

Monday Mar 05, 2012

ThreadLogic version 0.95 released

ThreadLogic version 0.95 is now available for public download.

Key addition is support for externalizing the Advisories and group definitions. Users can use the pre-defined AdvisoryMap.xml to come up with custom advisories with their own definition of what constitutes an advisory - name, pattern/keyword, health, description, advice and let ThreadLogic tag the matching threads with your custom advisory when it finds a match. Similarly the group definitions can be modified or enhanced to include new groupings. This is in addition to other smaller bug fixes and code cleanup.

For more details on how add customized advisories and groups and more features of version 0.95, please check my blog posting.

Feedback and suggestions welcome.

Thursday Mar 01, 2012

Deployments over InfiniBand Infrastructure

What actually drives network requirements ? Use of InfiniBand Partition Keys to create isolation between functional groups of servers. Sample use case illustrating how to design internal InfiniBand networks. Application consolidations and their association with different isolated networks on InfiniBand.

[Read More]


The primary contributors to this blog are comprised of the Exalogic and Cloud Application Foundation contingent of Oracle's Fusion Middleware Architecture Team, fondly known as the A-Team. As part of the Oracle development organization, The A-Team supports some of Oracle's largest and most strategic customers worldwide. Our mission is to provide deep technical expertise to support various Oracle field organizations and customers deploying Oracle Fusion Middleware related products. And to collect real world feedback to continuously improve the products we support. In this blog, our experts and guest experts will focus on Exalogic, WebLogic, Coherence, Tuxedo/mainframe migration, Enterprise Manager and JDK/JRockIT performance tuning. It is our way to share some of our experiences with Oracle community. We hope our followers took away something of value from our experiences. Thank you for visiting and please come back soon.


« February 2017