Server offers messaging, transaction and other system services to
facilitate building enterprise grade applications. Typically,
services can be either clustered or singleton. Clustered services are
deployed identically to each server in a cluster to provide increased
scalability and reliability. The session state of one clustered
server is replicated on another server in the cluster. In contrast,
singleton services run on only one server in a cluster at any given
point of time so as to offer specific quality of service (QOS) but
most importantly to preserve data consistency. Singleton services can
be JMS-related, JTA-related or user-defined. In highly available (HA)
environments, it is important for all services to be up and running
even during patch upgrades.
new WebLogic Zero
Downtime Patching (a.k.a ZDT patching) feature introduces a fully
automated rolling upgrade solution to perform upgrades such that
deployed applications continue to function and are available for end
users even during the upgrade process. ZDT patching supports rolling
out Oracle Home, Java Home and also updating applications. Check out
or view the documentation
for more information on ZDT.
ZDT rollouts, servers are restarted in a rolling manner. Taking down
a server would bring down the singleton service(s) thus causing
service disruptions, services will not be available until the server
starts back up. The actual down time would vary depending on server
start up time and number or type of applications deployed. Hence, to
ensure that singleton services do not introduce a single point of
failure for dependent applications in the cluster, ZDT rollout
process automatically performs migrations.
highlights of how ZDT rollout handles singletons are:
can be applied to all types of rollout (rolloutOracleHome,
rolloutJavaHome, rollingRestart, rolloutUpdate etc.)
JSON file based migration options for fine grained control of
service migrations during a rollout. Can be specified in WLST or
service migrations (JMS or JTA) as well as server migration (WSM)
fail back if needed
Acronyms and Abbreviations
Services that are hosted only on one server in a cluster.
Migratable Target (MT)
A special target that provides a way to group services that
only one active server at a given time.
Server instance where services are migrated “from”
Server instance where services are migrated “to”
Automatic Service Migration (ASM)
Process of moving affected subsystem service from one server
Whole Server Migration (WSM)
Process of moving entrire server instance from one physical
Fail back refers to relocating services back to original
ZDT rollout, servers are shutdown gracefully and then started back
up. To start with, administrators should be well aware of the
implications of restarting managed servers. An arbitrary application
may or may not be tolerant of a restart regardless of whether service
migration is setup or not.
may have non-persistent state
may or may not be tolerant of runtime client exceptions
may be impacted by duration of restart
a server is shutdown gracefully, client connections are closed,
clients consequently get exceptions, and the JMS server is removed
from candidate lists for load balancing and JMS message routing
decisions. Most of the time, such client exceptions are transient - a
retry will be redirected to a different JMS server, or even to the
original JMS Server after it was migrated. But some exceptions will
not be transient and they will instead continue being thrown on each
client retry until a particular JMS server instance comes back up.
Though server does some level of quiescing during shutdown, it
doesn't prevent all errors in JMS client or else where.
respect to JTA, when a server is shutting down gracefully, an
generate any new transaction requests for that particular server. For
EJB/RMI path, the cluster aware stubs would detect server connection
failures and redirect request to secondary server. It is assumed that
applications are designed to handle exceptions during a transaction.
server migration (WSM) is configured in environment, one should be
aware that it usually takes a longer time (when compared to service
migrations) to make services available since entire server instance
needs to boot on a new hardware.
general, the whole server migration is preferred for basic use due to
its relative simplicity, but automatic service migration becomes
attractive when faster fail-over times and advanced control over the
service migration are desirable.
JMS sub system is robust and high-performance and is often used in
conjunction with other APIs to build an enterprise application.
Smooth functioning of the applications largely depends on how the
application is designed (being resilient to failures, using certain
patterns or features) and also depends on how the JMS sub system is
tuned in the admin server.
WebLogic JMS, a message is only available if its host JMS server for
the destination is running. If a message is in a central persistent
store, the only JMS server that can access the message is the server
that originally stored the message. HA is normally accomplished using
either one or all of the following:
destinations: The queue and topic members of a distributed
destination are usually distributed across multiple servers within a
cluster, with each member belonging to a separate JMS server.
Applications that use distributed destinations are more highly
available than applications that use simple destinations because
WebLogic JMS provides load balancing and failover for member
destinations of a distributed destination within a cluster.
JMS modules utilize the SAF service to enable local JMS message
producers to reliably send messages to remote queues or topics. If
the destination is not available at the moment the messages are
sent, either because of network problems or system failures, then
the messages are saved on a local server instance, and are forwarded
to the remote destination once it becomes available.
Servers/Services: JMS Servers can be automatically restarted and/or
migrated using either Whole Server Migration or Automatic Service
production environment designed for high availability would most
likely ensure that JTA service (as well as other services) wouldn't
act as a single point of failure. The WebLogic transaction manager is
designed to recover from system crashes with minimal user
intervention. The transaction manager makes every effort to resolve
transaction branches that are prepared by resource managers with a
commit or roll back, even after multiple crashes or crashes during
recovery. It also attempts to recover transactions on system startup
by parsing all transaction log records for incomplete transactions
and completing them. However, in preparation for maintenance type of
operations like ZDT rollouts, JTA services may be configured for
migrations. JTA migration is needed since in-flight transactions can
hold locks on the underlying resources. If the transaction manager is
not available to recover these transactions, resources may hold on to
these locks as long as pending transactions are not resolved with a
commit/rollback (for long periods of time), causing errors on new
transactions and making it difficult for applications to function
on Service Migrations
migration in WebLogic Server is the process of moving the pinned
services from one server instance to a different server instance that
is available within the cluster.
migration is controlled by logical migratable target, which serves as
a grouping of services that is hosted on only one physical server in
a cluster. You can select a migratable target in place of a server or
cluster when targeting certain pinned services. The migration
framework provides tools and infrastructure for configuring and
migrating targets, and, in the case of automatic service migration,
it leverages WebLogic Server's health monitoring subsystem to monitor
the health of services hosted by a migratable target.
table summarizes various migration options
Manual Only (default)
Automatic service migration is disabled for this
Migration Strategy and Options
ZDT rollouts, “exactly-once” type of services are not of concern
since the migration sub system automatically handles these services.
It is the failure-recovery type of services that are of main concern.
These services will not migrate if server is gracefully shutdown.
Since the duration of restart may vary, these services need to be
migrated so that end users are not affected.
if user has configured services to be migrated manually, such
services are automatically migrated on administrator's behalf during
a rollout. ZDT rollouts can handle both JMS and JTA services
1. Transaction manager is not
2. For user-defined singletons, ZDT rollout doesn't need to
can specify the exact migration action on a per-server basis via the
migration properties file passed as an option to any of the rollout
commands. The migration options specified in the migration properties
file is validated against what is configured in the system, Required
migrations are initiated accordingly to mitigate down time. As an
optimization, the workflow generates the order in which rollouts
happens across the servers so as to prevent unnecessary migrations
between patched and unpatched servers.
rollouts also support whole server migration (WSM) if server(s) are
configured for it.
is the list of all the migration options:
All JMS related services running on current
JTA service is migrated from current hosting
Both JMS and JTA services on current hosting
Whole server instance will be migrated to
No migrations will happen for singleton services
will observe that these migration options are very similar to WLST
migrate command options.
following picture demonstrates a typical rollout sequence involving
service migrations. Here, the JMS and JTA singleton services are
represented by 2 types of migratable targets configured for each
server. Persistent stores and the TLOG should be accessible from all
servers in the cluster. Administrator has the control for specifying
how the migrations should happen across the servers in the cluster.
The next section describes the control knobs for fine grained control
of migrations during a rollout.
the migrations takes place for any of the rollouts is specified in a
migration properties file that is passed an option to the rollout
command. The migration properties file is nothing but a JSON file
consisting of 4 main properties:
Denotes the source server (name) which is nothing
Denotes the destination server (name) where the
name in case of server migration
Acceptable types are "jms", "jta",
described in previous section
Indicates if automatic failback of service to
is an example migration properties file:
# Migrate all JMS migratable targets on server1 to server2. Perform a fail back
# if the operation fails.
# Migrate only JTA services from server1 to server3. Note that JTA migration
# does not support the failback option, as it is not needed.
# Disable all migrations from server2
# Migrate all services (for example, JTA and JMS) from server 3 to server1 with
# no failback
# Use Whole Server Migration to migrate server4 to the node named machine 5 with
# no failback
migrationType is "None", then it implies services running
on this server will not be migrated, it also means no failback is
there are singleton services detected and administrator hasn't
passed in migration properties file, rollout command will fail. If
no migrations are needed, administrator should explicitly state that
via the migration properties (i.e migrationType=”None”) for each
of the servers.
migrationType is "server", destination should point to a
node manager machine name and WSM will be triggered for that server
default value of failback is false (no failback if option is not
a specific server, either ASM or WSM can be applied but not both.
JTA sub system supports automatic failback of JTA service, failback
is not a valid option for JTA service.
of the above mentioned validation checks will happen as part of
pre-requisites check before any rollout.
following examples illustrate the usage of migration properties
sample migrationProperties.json file:
migration options to rolloutOracleHome
migration options to rolloutApplications
migration options to rolloutJavaHome
migration options to rolloutUpdate
migration options to rollingRestart