• WLS
    December 31, 2015

ZDT Rollouts and Singletons

Yamini Kalyandurga
Consulting Member of Technical Staff

Server offers messaging, transaction and other system services to
facilitate building enterprise grade applications. Typically,
services can be either clustered or singleton. Clustered services are
deployed identically to each server in a cluster to provide increased
scalability and reliability. The session state of one clustered
server is replicated on another server in the cluster. In contrast,
singleton services run on only one server in a cluster at any given
point of time so as to offer specific quality of service (QOS) but
most importantly to preserve data consistency. Singleton services can
be JMS-related, JTA-related or user-defined. In highly available (HA)
environments, it is important for all services to be up and running
even during patch upgrades.

new WebLogic Zero
Downtime Patching
(a.k.a ZDT patching) feature introduces a fully
automated rolling upgrade solution to perform upgrades such that
deployed applications continue to function and are available for end
users even during the upgrade process. ZDT patching supports rolling
out Oracle Home, Java Home and also updating applications. Check out
these blogs
or view the documentation
for more information on ZDT.

ZDT rollouts, servers are restarted in a rolling manner. Taking down
a server would bring down the singleton service(s) thus causing
service disruptions, services will not be available until the server
starts back up. The actual down time would vary depending on server
start up time and number or type of applications deployed. Hence, to
ensure that singleton services do not introduce a single point of
failure for dependent applications in the cluster, ZDT rollout
process automatically performs migrations.

highlights of how ZDT rollout handles singletons are:

  • It

    can be applied to all types of rollout (rolloutOracleHome,

    rolloutJavaHome, rollingRestart, rolloutUpdate etc.)

  • Uses

    JSON file based migration options for fine grained control of

    service migrations during a rollout. Can be specified in WLST or


  • Supports

    service migrations (JMS or JTA) as well as server migration (WSM)

  • Automatic

    fail back if needed

Acronyms and Abbreviations




Services that are hosted only on one server in a cluster.

Migratable Target (MT)

A special target that provides a way to group services that

should move together. It contains a list of candidate servers with

only one active server at a given time.

Source Server

Server instance where services are migrated “from”

Destination Server

Server instance where services are migrated “to”

Automatic Service Migration (ASM)

Process of moving affected subsystem service from one server

instance to another running server instance

Whole Server Migration (WSM)

Process of moving entrire server instance from one physical

machine to another

Fail back

Fail back refers to relocating services back to original

hosting or “home” server.


ZDT rollout, servers are shutdown gracefully and then started back
up. To start with, administrators should be well aware of the
implications of restarting managed servers. An arbitrary application
may or may not be tolerant of a restart regardless of whether service
migration is setup or not.

  • It

    may have non-persistent state

  • It

    may or may not be tolerant of runtime client exceptions

  • It

    may be impacted by duration of restart

a server is shutdown gracefully, client connections are closed,
clients consequently get exceptions, and the JMS server is removed
from candidate lists for load balancing and JMS message routing
decisions. Most of the time, such client exceptions are transient - a
retry will be redirected to a different JMS server, or even to the
original JMS Server after it was migrated. But some exceptions will
not be transient and they will instead continue being thrown on each
client retry until a particular JMS server instance comes back up.
Though server does some level of quiescing during shutdown, it
doesn't prevent all errors in JMS client or else where.

respect to JTA, when a server is shutting down gracefully, an

generate any new transaction requests for that particular server. For
EJB/RMI path, the cluster aware stubs would detect server connection
failures and redirect request to secondary server. It is assumed that
applications are designed to handle exceptions during a transaction.

server migration (WSM) is configured in environment, one should be
aware that it usually takes a longer time (when compared to service
migrations) to make services available since entire server instance
needs to boot on a new hardware.

general, the whole server migration is preferred for basic use due to
its relative simplicity, but automatic service migration becomes
attractive when faster fail-over times and advanced control over the
service migration are desirable.


JMS sub system is robust and high-performance and is often used in
conjunction with other APIs to build an enterprise application.
Smooth functioning of the applications largely depends on how the
application is designed (being resilient to failures, using certain
patterns or features) and also depends on how the JMS sub system is
tuned in the admin server.

WebLogic JMS, a message is only available if its host JMS server for
the destination is running. If a message is in a central persistent
store, the only JMS server that can access the message is the server
that originally stored the message. HA is normally accomplished using
either one or all of the following:

  • Distributed

    destinations: The queue and topic members of a distributed

    destination are usually distributed across multiple servers within a

    cluster, with each member belonging to a separate JMS server.

    Applications that use distributed destinations are more highly

    available than applications that use simple destinations because

    WebLogic JMS provides load balancing and failover for member

    destinations of a distributed destination within a cluster.

  • Store-and-Forward:

    JMS modules utilize the SAF service to enable local JMS message

    producers to reliably send messages to remote queues or topics. If

    the destination is not available at the moment the messages are

    sent, either because of network problems or system failures, then

    the messages are saved on a local server instance, and are forwarded

    to the remote destination once it becomes available.

  • HA

    Servers/Services: JMS Servers can be automatically restarted and/or

    migrated using either Whole Server Migration or Automatic Service



production environment designed for high availability would most
likely ensure that JTA service (as well as other services) wouldn't
act as a single point of failure. The WebLogic transaction manager is
designed to recover from system crashes with minimal user
intervention. The transaction manager makes every effort to resolve
transaction branches that are prepared by resource managers with a
commit or roll back, even after multiple crashes or crashes during
recovery. It also attempts to recover transactions on system startup
by parsing all transaction log records for incomplete transactions
and completing them. However, in preparation for maintenance type of
operations like ZDT rollouts, JTA services may be configured for
migrations. JTA migration is needed since in-flight transactions can
hold locks on the underlying resources. If the transaction manager is
not available to recover these transactions, resources may hold on to
these locks as long as pending transactions are not resolved with a
commit/rollback (for long periods of time), causing errors on new
transactions and making it difficult for applications to function

on Service Migrations

migration in WebLogic Server is the process of moving the pinned
services from one server instance to a different server instance that
is available within the cluster.

migration is controlled by logical migratable target, which serves as
a grouping of services that is hosted on only one physical server in
a cluster. You can select a migratable target in place of a server or
cluster when targeting certain pinned services. The migration
framework provides tools and infrastructure for configuring and
migrating targets, and, in the case of automatic service migration,
it leverages WebLogic Server's health monitoring subsystem to monitor
the health of services hosted by a migratable target.

table summarizes various migration options

Policy Type


Manual Only (default)

Automatic service migration is disabled for this


Failure Recovery

Pinned services

deployed to this target will:

  • Initially start only on the preferred server

  • Migrate

    only if the cluster master determines that the


    server has failed

Exactly Once

Pinned services

deployed to this target will:

  • Initially start on a candidate server if the preferred one



  • Migrate

    if the host server fails or is gracefully shut


Migration Strategy and Options

ZDT rollouts, “exactly-once” type of services are not of concern
since the migration sub system automatically handles these services.
It is the failure-recovery type of services that are of main concern.
These services will not migrate if server is gracefully shutdown.
Since the duration of restart may vary, these services need to be
migrated so that end users are not affected.

if user has configured services to be migrated manually, such
services are automatically migrated on administrator's behalf during
a rollout. ZDT rollouts can handle both JMS and JTA services


1. Transaction manager is not

assigned to a migratable target like other pinned services,

instead JTA ASM is a per-server setting. This is because the

transaction manager has no direct dependencies on other pinned

resources when contrasted with services such as JMS.

2. For user-defined singletons, ZDT rollout doesn't need to

take any specific action since they are automatically configured

as “exactly-once”.

can specify the exact migration action on a per-server basis via the
migration properties file passed as an option to any of the rollout
commands. The migration options specified in the migration properties
file is validated against what is configured in the system, Required
migrations are initiated accordingly to mitigate down time. As an
optimization, the workflow generates the order in which rollouts
happens across the servers so as to prevent unnecessary migrations
between patched and unpatched servers.

rollouts also support whole server migration (WSM) if server(s) are
configured for it.

is the list of all the migration options:





All JMS related services running on current

hosting server are migrated to the destination server


JTA service is migrated from current hosting

server to the destination server


Both JMS and JTA services on current hosting

server are migrated to the destination server


Whole server instance will be migrated to

destination machine


No migrations will happen for singleton services

running on current server

will observe that these migration options are very similar to WLST
migrate command options.

Migration Sequence

following picture demonstrates a typical rollout sequence involving
service migrations. Here, the JMS and JTA singleton services are
represented by 2 types of migratable targets configured for each
server. Persistent stores and the TLOG should be accessible from all
servers in the cluster. Administrator has the control for specifying
how the migrations should happen across the servers in the cluster.
The next section describes the control knobs for fine grained control
of migrations during a rollout.

Migration Properties

the migrations takes place for any of the rollouts is specified in a
migration properties file that is passed an option to the rollout
command. The migration properties file is nothing but a JSON file
consisting of 4 main properties:





Denotes the source server (name) which is nothing

but the current hosting server for the singleton(s)


Denotes the destination server (name) where the

singleton service will be migrated to. This can also be a machine

name in case of server migration


Acceptable types are "jms", "jta",

"all", "server", "none" as

described in previous section


Indicates if automatic failback of service to

original hosting server should happen or not

is an example migration properties file:

# Migrate all JMS migratable targets on server1 to server2. Perform a fail back
# if the operation fails.
# Migrate only JTA services from server1 to server3. Note that JTA migration
# does not support the failback option, as it is not needed.
# Disable all migrations from server2
# Migrate all services (for example, JTA and JMS) from server 3 to server1 with
# no failback

# Use Whole Server Migration to migrate server4 to the node named machine 5 with
# no failback

  • If

    migrationType is "None", then it implies services running

    on this server will not be migrated, it also means no failback is


  • If

    there are singleton services detected and administrator hasn't

    passed in migration properties file, rollout command will fail. If

    no migrations are needed, administrator should explicitly state that

    via the migration properties (i.e migrationType=”None”) for each

    of the servers.

  • If

    migrationType is "server", destination should point to a

    node manager machine name and WSM will be triggered for that server


  • The

    default value of failback is false (no failback if option is not


  • For

    a specific server, either ASM or WSM can be applied but not both.

  • Since

    JTA sub system supports automatic failback of JTA service, failback

    is not a valid option for JTA service.


    of the above mentioned validation checks will happen as part of

    pre-requisites check before any rollout.

Rollout Examples

following examples illustrate the usage of migration properties

sample migrationProperties.json file:

{"migrations":[ {
} ]}

migration options to rolloutOracleHome


migration options to rolloutApplications


migration options to rolloutJavaHome

rolloutJavaHome('myDomain', javaHome='/pathto/JavaHome1.8.0_60',

migration options to rolloutUpdate

rolloutUpdate('myDomain', '/pathto/patchedOracleHome.jar',
'/pathto/unpatchedOracleHomeBackup/', false,

migration options to rollingRestart



Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.