Friday May 15, 2015

Deleting stale VNICs

When there are stale VNICs in an Exalogic rack, a "Stale VNICs are present in the switches" warning is thrown in Exachk report. When looking deeper into that warning, the following risk can be seen:
VNICs in states other than "UP" can cause network outages. In a virtual rack, excessive number of unused vNICs can cause performance issues.

Sometimes the steps to delete these stale VNICs can be confusing to some guys, so I am posting here simple steps to achieve it.

The syntax of the deletevnic command is:
# deletevnic connector vnic_id

Ok, but how can I know the connector and vnic_id of each stale VNIC?
Well, they are listed in Exachk.
But also, you can check what VNICs are stale by running the following command:
# showvnics|grep -i WAIT-IOA
And then you will see the vnic_id in the first column and the conector in the last column.

The output would be something like:
74 WAIT-IOA N 27ABA53803F429048 rackcn02 0000 00:14:4F:FB:70:D7 13 0x8007 0A-ETH-2
30 WAIT-IOA N 5E6C31804046A4361 rackcn04 0000 00:14:4F:FA:50:D4 13 0x8007 0A-ETH-3
51 WAIT-IOA N BDC7F7A200761E5E0 rackcn01 0000 00:14:4F:FA:91:3F 13 0x8007 0A-ETH-3
16 WAIT-IOA N A584D51705DA41538 rackcn01 0000 00:14:4F:F8:8D:84 13 0x8007 0A-ETH-4

But, as several times the VNICs are associated to compute nodes, some people may think that deleting the VNIC would affect the networking performance of the compute node.
The answer for that perfectly valid concern is that, as the vnic_id is unique, then you can safely use the deletevnic command for each one of the stale VNICs.

Wednesday Jun 11, 2014

Setting MTU on Exalogic

For many reasons, a system administrator may want to change the MTU settings of a server. But in a system like Exalogic which contains lots of interconnected nodes and other various components, it's important to understand how this applies to the different networks.

For example, when bringing up bonding of InfiniBand an error like the following may be thrown:
Bringing up interface bond1: SIOCSIFMTU: Invalid argument
Both scripts ifcfg-ib0 and ifcfg-ib1 (from the /etc/sysconfig/network-scripts/ direectory) have MTU set to 65500, which is a valid MTU value only if all IPoIB slaves operate in connected mode and are configured with the same value, so the line below must be added to both network scripts and then restart the network:

By the way, an error of the form “SIOCSIFMTU: Invalid argument” indicates that the requested MTU was rejected by the kernel. Typically this would be due to it exceeding the maximum value supported by the interface hardware. In that case you must either reduce the MTU to a value that is supported or obtain more capable hardware. This problem has been seen when trying to modify the MTU using the ifconfig command, like the output of the example below:
[root@elxxcnxx ~]# ifconfig ib1 mtu 65520
SIOCSIFMTU: Invalid argument

It's important to insist that in most cases the nodes must be rebooted after the MTU size has been changed. Although in some circumstances it may work without a reboot, it is not how it is typically documented.

Now, in order to achieve a reduced memory consumption and improve performance for network traffic received on IPoIB related interfaces, it is recommend to reduce the MTU value in interface configuration files for IPoIB related bonds from 65520 to 64000. The change needs to be made to interface configuration files under the /etc/sysconfig/network-scripts directory and applies to the interface configuration files for bonds over IPoIB related slave devices, for example /etc/sysconfig/network-scripts/ifcfg-bond1. However, keep in mind that the numeric portion of the interface filenames that corresponding to IPoIB interfaces is expected to vary across compute nodes and vServers and so cannot be relied upon to identify which interface files are for bonds are over IPoIB rather than EoIB related slave interfaces.
To fix these MTU values to the recommended settings, there are very useful instructions and a script on the MOS Note 1624434.1, and it's applicable physical and virtual configurations of Exalogic.

Regarding the recommended MTU value for EoIB related interfaces, its maximum appropriate value is 1500. If for some reason a vServer has been created with a higher value (set on the /etc/sysconfig/network-scripts/ifcfg-bond0 file), then it must be fixed. An error like the following could be thrown under this circumstance:
[root@vServer ~]# service network restart
Bringing up interface bond0:  SIOCSIFMTU: Invalid argument

Also an error like the one below can be seen on the /var/log/messages file of the vServer:
kernel: T5074835532 [mlx4_vnic] eth1:vnic_change_mtu:360: failed: new_mtu 64000 > 2026
The MOS Note 1611657.1 is very useful for this purpose.

Wednesday Dec 18, 2013

"Cannot allocate memory" message when accessing a Compute Node through SSH, despite ILOM shows available memory

We recently worked an issue where it happened that when accessing the server through ssh it threw a "Cannot allocate memory" message, however ILOM showed available memory.

This happened due to a known bug with a package, related with the ypserv utility:

The problem is that even though there is enough free memory, it is too fragmented to allocate two contiguous pages.

The command below can be used:
# echo 3 > /proc/sys/vm/drop_caches
This may allow for memory to become defragmented enough for further fork() system calls to succeed otherwise it may be necessary to reboot the system.

Wednesday Nov 13, 2013

Unable to change NIS password

On the last few weeks I worked a Service Request where Linux user was unable to change user passwords NIS. As a matter of fact, NIS is used by several people on their Exalogic environments.

[root@computenode ~]# passwd user01
Changing password for user user01.
passwd: Authentication token manipulation error
[root@computenode ~]#
In this case, "user01" is an example username.

This issue may occur on all NIS nodes and even on master server as well.
This error typically corresponds to typos or missing keywords in configuration files from the /etc/pam.d directory.
On the Service Request that I worked, the file system-auth-ac had no nis keyword:

# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required
auth sufficient nullok try_first_pass
auth requisite uid >= 500 quiet
auth required

account required
account sufficient uid < 500 quiet
account required

password requisite try_first_pass retry=3
password sufficient md5 shadow remember=5 nullok try_first_pass use_authtok
password required

session optional revoke
session required
session [success=1 default=ignore] service in crond quiet use_uid
session required

This kind of issue has also been described in a case where the "" line had a typo ("sufficent" instead of correct "sufficient") on the su file.

To solve this kind of issues, first the typos must be (obviously) fixed.
For this NIS case, it is necessary to make sure that keyword is added:
password sufficient md5 shadow nis remember=5 nullok try_first_pass use_authtok

Note that these settings should be consistent across all the NIS nodes (master server and clients).

Tuesday Oct 22, 2013

Minimum percentage of free physical memory that Linux requires for optimal performance

Recently, we have been getting questions about this percentage of free physical memory that OS require for optimal performance, mainly applicable to physical compute nodes.

Under normal conditions you may see that at the nodes without any application running the OS take (for example) between 24 and 25 GB of memory.
The Linux system reports the free memory in a different way, and most of those 25gbs (of the example) are available for user processes.
IE: Mem: 99191652k total, 23785732k used, 75405920k free, 173320k buffers

The MOS Doc Id. 233753.1 - "Analyzing Data Provided by '/proc/meminfo'" - explains it (section 4 - "Final Remarks"):
Free Memory and Used Memory
Estimating the resource usage, especially the memory consumption of processes is by far more complicated than it looks like at a first glance. The philosophy is an unused resource is a wasted resource.The kernel therefore will use as much RAM as it can to cache information from your local and remote filesystems/disks. This builds up over time as reads and writes are done on the system trying to keep the data stored in RAM as relevant as possible to the processes that have been running on your system. If there is free RAM available, more caching will be performed and thus more memory 'consumed'. However this doesn't really count as resource usage, since this cached memory is available in case some other process needs it. The cache is reclaimed, not at the time of process exit (you might start up another process soon that needs the same data), but upon demand.

That said, focusing more specifically on the percentage question, apart from this memory that OS takes, how much should be the minimum free memory that must be available every node so that they operate normally?
The answer is: As a rule of thumb 80% memory utilization is a good threshold, anything bigger than that should be investigated and remedied.

Wednesday Sep 25, 2013

Adding NTP Servers to Exalogic Switches

Sometimes there are some misconfigurations on the Exalogic machines as for some reasons compute nodes may have more specified addresses for NTP than the switches. So here are a few steps to add those in the switches.

To add another NTP server to the Infiniband/Gateway switches:
- Login on the switch ILOM BUI (preferably MSIE)
- Select the Configuration tab
- Select the Clock subtab
- Enter the NTP IP servers addresses appropriately

To configure two NTP servers on the the Cisco Ethernet Switch, please take a look at - "5.4.1 Configuring the Cisco Ethernet Switch".

Note that, besides adding servers, you can also modify these configurations at your convenience.

Wednesday Aug 21, 2013

Resource allocation on vServers

A few facts to keep in mind.

The number of vCPUs to existent vServer cannot be modified. Currently vDC resource management (changing vCPU, memory, etc.) after a vServer has been created is not supported in Exalogic Virtual environments. However, a customer vServer type with required memory can be created and used instead of the default VM types.

Note that the Server pool management is handled by Exalogic, manual management of it is not available. Also, keep in mind that there is no way to control which physical host is assigned a given vServer. The scheduler algorithms are sophisticated but it is fine-grained and there is no reason to assume that the scheduler will not be maximally efficient. EMOC will look at the resources allocated to the vServer that is being planned on creating (CPUs/memory), then look at what is available on each of the compute nodes and make a decision on where to place the vServer. System administrators can try to use Distribution groups and separate the vServers they have to run across different Oracle VM Servers.

If need to increase Root File System and Swap Space of an Exalogic Guest Virtual Machine, then the MOS Note 1575790.1 is useful for that purpose.

Monday Jul 29, 2013

Coherence on Exalogic: dealing with the multiple network interfaces

Recently, we worked an incident where error messages like the following were being thrown when starting the Coherence servers after an upgrade of EECS:
Oracle Coherence GE (thread=Thread-3, member=n/a): Loaded Reporter configuration from "jar:file:/u01/app/fmw_product/wlserver_103/coherence_3.7/lib/coherence.jar!/reports/report-group.xml"
Exception in thread "Thread-3" java.lang.IllegalArgumentException: unresolvable localhost at
Caused by: java.rmi.server.ExportException: Listen failed on port: 8877; nested exception is: Address already in use ...
Caused by: is not a local address

It is a very known fact that Exalgic has several network interfaces (bond/eth 0,1,2, etc). The logic that Coherence uses when deciding what interface to connect to, specifically to support machines with multiple network interfaces as well as enhancements to allow the localaddress to be specified as a netmask to make configuration across larger clusters easier, makes important (even more than on previuous releases of Coherence) to make sure that the tangosol.coherence.localhost parameter is specified appropriately. From that IP address (or properly mapped host address) the desired network interface to be used can easily be found and then the Coherence cluster would work fine on it.

Friday Jan 18, 2013

Two New Exalogic ZFS Storage Appliance MOS Notes

This week I have closed 2 Service Requests related to the ZFS Storage Appliance and I created My Oracle Support (MOS) notes from both of them as, despite they were not complicated issues and the SRs were both closed in less than one week, these procedures were still not formally documented on MOS. Below can be seen the information about these created documents.

MOS Doc Id. 1519858.1 - Will The Restart Of The NIS Service On The ZFS Storage Appliance Affect The Mounted Filesystems?

On this case, for a particular reason it was necessary to restart the NIS service. So, if for any reason, the NIS service needs to be restarted on the ZFS Storage Appliance, will the mounted filesystems be affected during the restart?

The default cluster configuration type of the ZFS storage appliance is active-passive and the storage nodes are supposed to be mirrored, so the restart of NIS should not be causing any issues; it can be done.

Note that restart of NIS should be done on the active storage head. Restarting the NIS itself will not cause any ZFS failover from Active to Passive.

In general terms, even in the event of a storage node failure, the appliance will automatically fail over to the other storage node. Under that condition, an initial degradation in performance can be expected because all of the cached data on the failed node is gone, but this effect decreases as the new active storage node begins caching data in its own SSDs.

MOS Doc Id. 1520223.1 - Exalogic Storage Nodes Hostnames Are Displayed Incorrectly

This was not the first time I saw something like this, so decided to create a note because clearly is a problem that may affect to more than one Exalogic user.

The Exalogic storage node hostnames displayed on the BUI were different than the ones displayed when accessing the node through SSH or ILOM.

This happens because for any reason the hostname is misconfigured on the ZFS Storage Appliance.

To solve this problem, it is necessary to set the system name and location accordingly on the Storage Appliance nodes BUI:
1. Login on the ZFS Storage Appliance BUI
2. Go to the "Configuration" tab, and select the "Services" subtab
3. Under the "Systems Settings" section, click on "System Identity"
4. Set the system name and location accordingly

Friday May 18, 2012

CONNECTION_REFUSED messages on load balancing in Weblogic with OHS or Apache

In the last months I have had to work on some issues related to load balancing. It is very important to understand how the layers interact between them and where specific settings must be done.

Some people gets upset with the fact that OHS/Apache do load balancing even to servers that are shutdown and may be losing transactions.

This document provides very good tips about how many Production critical issues can be resolved just by setting the appropriate values for some parameters.

Personally, I think that the DynamicServerList parameter (which is in fact the first one mentioned on the document linked above) is particularly important to understand. As can be seen at this documentation from Oracle:
In a clustered environment, a plug-in may dispatch requests to an unavailable WebLogic Server instance because the DynamicServerList is not current in all plug-in processes.
DynamicServerList=ON works with a single Apache server (httpd daemon process), but for more than one server, such as StartServers=5, the dynamic server list will not be updated across all httpd instances until they have all tried to contact a WebLogic Server instance. This is because they are separate processes. This delay in updating the dynamic server list could allow an Apache httpd process to contact a server that another httpd process has marked as dead. Only after such an attempt will the server list will be updated within the proxy. One possible solution if this is undesirable is to set the DynamicServerList to OFF.
In a non-clustered environment, a plug-in may lose the stickiness of a session created after restarting WebLogic Server instances, because some plug-in processes do not have the new JVMID of those restarted servers, and treat them as unknown JVMIDs.
To avoid these issues, upgrade to Apache 2.0.x and configure Apache to use the multi-threaded and single-process model, mpm_worker_module.

Also, this Oracle documentation provides inportant information about "Failover, Cookies, and HTTP Sessions", and "Tuning to Reduce Connection_Refused Errors".

As can be seen at this Apache document, the MaxRequestsPerChild directive sets the limit on the number of requests that an individual child server will handle during its life.

Note that mod_proxy and related modules implement a proxy/gateway for Apache HTTP Server, supporting a number of popular protocols as well as several different load balancing algorithms. Third-party modules can add support for additional protocols and load balancing algorithms.

On Oracle Forums I also found a very interesting thread:
The error which you are getting is a common which can be fixed by increasing the "AcceptBackLog" value by 25% until error disappears from weblogic console (Path: Servers => => Configuration tab=> Tuning sub-tab.) and setting the value to ON for "KeepAlive" in the httpd.conf which should take care of your issue.
Topic: Tuning Connection Backlog Buffering
Search for "KeepAliveEnabled":
Also here is a link which would be helpful to understand some common issue which occurs when using a plug-in and there are solutions:

May transactions be affected because of this?
Certainly yes, but it depends on how your application is developed. A good practice would be to create a bunch of transactions and track them to check if some are missed or not. This Transaction and redelivery in JMS article may be helpful.

Wednesday Oct 12, 2011

Focus on Oracle Exalogic Elastic Cloud at Oracle OpenWorld 2011

In the upcoming Oracle OpenWorld there will be several sessions and events that would be of interest to anyone who wants to learn more about the Oracle Exalogic Elastic Cloud. There are many excellent presenters, a ton of new data, fantastic new demos and numerous opportunities to hear about Exalogic directly from Exalogic customers.

For further information, take a look at

Thursday Sep 01, 2011

The MANIFEST.MF filename... Oh My!

In the last month I worked a Service Request along with my colleague Ernani Mattos, where on WebLogic 10.3.4, when attempting to deploy Coherance*Web, the following error occurs on RedHat, but not on Windows or Debian:
weblogic.application.ModuleException: Missing Coherence jar or WebLogic Coherence Integration jar

The customer was following the Caching Sessions with Coherence and WebLogic Server documentation.

Taking a look at the ModuleException, the coherence.jar is part of the counter.war application, and the Weblogic.coherence.integration.jar is declared with the active-cache-1.0.jar lib which is referenced by the MANIFEST.MF of the counter application.

The entire issue was that MANIFEST.MF on the counter.war was named as ""; Therefore RedHat Linux could not recognize it simply due to the naming issues. So make sure that manifest file name is "MANIFEST.MF" and declares to active-cache-1.0.jar which should be deployed as a lib on the servers.

This was solved by executing the following steps:
1. Explode the counter.war application
2. Goto counter/META-INF/ directory
3. Rename the to MANIFEST.MF
4. Deploy as open directory or compact back to war file and deploy

Thursday Mar 31, 2011

Filebased Storage: LH or BDB?

[Read More]

Principal Technical Support Engineer in the Engineered (Systems) Enterprise Support Team - EEST.
Former member of the Coherence and Java Technologies Support Teams.


« July 2016