Saturday Jun 23, 2012

Calculating the Size (in Bytes and MB) of a Oracle Coherence Cache

The concept and usage of data grids are becoming very popular in this days since this type of technology are evolving very fast with some cool lead products like Oracle Coherence. Once for a while, developers need an programmatic way to calculate the total size of a specific cache that are residing in the data grid. In this post, I will show how to accomplish this using Oracle Coherence API. This example has been tested with 3.6, 3.7 and 3.7.1 versions of Oracle Coherence.

To start the development of this example, you need to create a POJO ("Plain Old Java Object") that represents a data structure that will hold user data. This data structure will also create an internal fat so I call that should increase considerably the size of each instance in the heap memory. Create a Java class named "Person" as shown in the listing below.

package com.oracle.coherence.domain;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Random;

@SuppressWarnings("serial")
public class Person implements Serializable {
	
	private String firstName;
	private String lastName;
	private List<Object> fat;
	private String email;
	
	public Person() {
		generateFat();
	}
	
	public Person(String firstName, String lastName,
			String email) {
		setFirstName(firstName);
		setLastName(lastName);
		setEmail(email);
		generateFat();
	}
	
	private void generateFat() {
		fat = new ArrayList<Object>();
		Random random = new Random();
		for (int i = 0; i < random.nextInt(18000); i++) {
			HashMap<Long, Double> internalFat = new HashMap<Long, Double>();
			for (int j = 0; j < random.nextInt(10000); j++) {
				internalFat.put(random.nextLong(), random.nextDouble());
			}
			fat.add(internalFat);
		}
	}
	
	public String getFirstName() {
		return firstName;
	}

	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}

	public String getLastName() {
		return lastName;
	}

	public void setLastName(String lastName) {
		this.lastName = lastName;
	}

	public String getEmail() {
		return email;
	}

	public void setEmail(String email) {
		this.email = email;
	}

}

Now let's create a Java program that will start a data grid into Coherence and will create a cache named "People", that will hold people instances with sequential integer keys. Each person created in this program will trigger the execution of a custom constructor created in the People class that instantiates an internal fat (the random amount of data generated to increase the size of the object) for each person. Create a Java class named "CreatePeopleCacheAndPopulateWithData" as shown in the listing below.

package com.oracle.coherence.demo;

import com.oracle.coherence.domain.Person;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;

public class CreatePeopleCacheAndPopulateWithData {

	public static void main(String[] args) {
		
		// Asks Coherence for a new cache named "People"...
		NamedCache people = CacheFactory.getCache("People");
		
		// Creates three people that will be putted into the data grid. Each person
		// generates an internal fat that should increase its size in terms of bytes...
		Person pessoa1 = new Person("Ricardo", "Ferreira", "ricardo.ferreira@example.com");
		Person pessoa2 = new Person("Vitor", "Ferreira", "vitor.ferreira@example.com");
		Person pessoa3 = new Person("Vivian", "Ferreira", "vivian.ferreira@example.com");
		
		// Insert three people at the data grid...
		people.put(1, pessoa1);
		people.put(2, pessoa2);
		people.put(3, pessoa3);
		
		// Waits for 5 minutes until the user runs the Java program
		// that calculates the total size of the people cache...
		try {
			System.out.println("---> Waiting for 5 minutes for the cache size calculation...");
			Thread.sleep(300000);
		} catch (InterruptedException ie) {
			ie.printStackTrace();
		}
		
	}

}

Finally, let's create a Java program that, using the Coherence API and JMX, will calculate the total size of each cache that the data grid is currently managing. The approach used in this example was retrieve every cache that the data grid are currently managing, but if you are interested on an specific cache, the same approach can be used, you should only filter witch cache will be looked for. Create a Java class named "CalculateTheSizeOfPeopleCache" as shown in the listing below.

package com.oracle.coherence.demo;

import java.text.DecimalFormat;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;

import com.tangosol.net.CacheFactory;

public class CalculateTheSizeOfPeopleCache {
	
	@SuppressWarnings({ "unchecked", "rawtypes" })
	private void run() throws Exception {
		
        // Enable JMX support in this Coherence data grid session...
	System.setProperty("tangosol.coherence.management", "all");
		
        // Create a sample cache just to access the data grid...
	CacheFactory.getCache(MBeanServerFactory.class.getName());
		
	// Gets the JMX server from Coherence data grid...
	MBeanServer jmxServer = getJMXServer();
        
        // Creates a internal data structure that would maintain
	// the statistics from each cache in the data grid...
	Map cacheList = new TreeMap();
        Set jmxObjectList = jmxServer.queryNames(new ObjectName("Coherence:type=Cache,*"), null);
        for (Object jmxObject : jmxObjectList) {
            ObjectName jmxObjectName = (ObjectName) jmxObject;
            String cacheName = jmxObjectName.getKeyProperty("name");
            if (cacheName.equals(MBeanServerFactory.class.getName())) {
            	continue;
            } else {
            	cacheList.put(cacheName, new Statistics(cacheName));
            }
        }
        
        // Updates the internal data structure with statistic data
        // retrieved from caches inside the in-memory data grid...
        Set<String> cacheNames = cacheList.keySet();
        for (String cacheName : cacheNames) {
            Set resultSet = jmxServer.queryNames(
            	new ObjectName("Coherence:type=Cache,name=" + cacheName + ",*"), null);
            for (Object resultSetRef : resultSet) {
                ObjectName objectName = (ObjectName) resultSetRef;
                if (objectName.getKeyProperty("tier").equals("back")) {
                    int unit = (Integer) jmxServer.getAttribute(objectName, "Units");
                    int size = (Integer) jmxServer.getAttribute(objectName, "Size");
                    Statistics statistics = (Statistics) cacheList.get(cacheName);
                    statistics.incrementUnit(unit);
                    statistics.incrementSize(size);
                    cacheList.put(cacheName, statistics);
                }
            }
        }
        
        // Finally... print the objects from the internal data
        // structure that represents the statistics from caches...
        cacheNames = cacheList.keySet();
        for (String cacheName : cacheNames) {
            Statistics estatisticas = (Statistics) cacheList.get(cacheName);
            System.out.println(estatisticas);
        }
        
    }

    public MBeanServer getJMXServer() {
        MBeanServer jmxServer = null;
        for (Object jmxServerRef : MBeanServerFactory.findMBeanServer(null)) {
            jmxServer = (MBeanServer) jmxServerRef;
            if (jmxServer.getDefaultDomain().equals(DEFAULT_DOMAIN) || DEFAULT_DOMAIN.length() == 0) {
                break;
            }
            jmxServer = null;
        }
        if (jmxServer == null) {
            jmxServer = MBeanServerFactory.createMBeanServer(DEFAULT_DOMAIN);
        }
        return jmxServer;
    }
	
    private class Statistics {
		
        private long unit;
        private long size;
        private String cacheName;
		
	public Statistics(String cacheName) {
            this.cacheName = cacheName;
        }

        public void incrementUnit(long unit) {
            this.unit += unit;
        }

        public void incrementSize(long size) {
            this.size += size;
        }

        public long getUnit() {
            return unit;
        }

        public long getSize() {
            return size;
        }

        public double getUnitInMB() {
            return unit / (1024.0 * 1024.0);
        }

        public double getAverageSize() {
            return size == 0 ? 0 : unit / size;
        }

        public String toString() {
            StringBuffer sb = new StringBuffer();
            sb.append("\nCache Statistics of '").append(cacheName).append("':\n");
            sb.append("   - Total Entries of Cache -----> " + getSize()).append("\n");
            sb.append("   - Used Memory (Bytes) --------> " + getUnit()).append("\n");
            sb.append("   - Used Memory (MB) -----------> " + FORMAT.format(getUnitInMB())).append("\n");
            sb.append("   - Object Average Size --------> " + FORMAT.format(getAverageSize())).append("\n");
            return sb.toString();
        }

    }
	
    public static void main(String[] args) throws Exception {
	new CalculateTheSizeOfPeopleCache().run();
    }
	
    public static final DecimalFormat FORMAT = new DecimalFormat("###.###");
    public static final String DEFAULT_DOMAIN = "";
    public static final String DOMAIN_NAME = "Coherence";

}

I've commented the overall example so, I don't think that you should get into trouble to understand it. Basically we are dealing with JMX. The first thing to do is enable JMX support for the Coherence client (ie, an JVM that will only retrieve values from the data grid and will not integrate the cluster) application. This can be done very easily using the runtime "tangosol.coherence.management" system property. Consult the Coherence documentation for JMX to understand the possible values that could be applied. The program creates an in memory data structure that holds a custom class created called "Statistics".

This class represents the information that we are interested to see, which in this case are the size in bytes and in MB of the caches. An instance of this class is created for each cache that are currently managed by the data grid. Using JMX specific methods, we retrieve the information that are relevant for calculate the total size of the caches. To test this example, you should execute first the CreatePeopleCacheAndPopulateWithData.java program and after the CreatePeopleCacheAndPopulateWithData.java program. The results in the console should be something like this:

2012-06-23 13:29:31.188/4.970 Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence.xml"
2012-06-23 13:29:31.219/5.001 Oracle Coherence 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-06-23 13:29:31.219/5.001 Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2012-06-23 13:29:31.266/5.048 Oracle Coherence 3.6.0.4 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.6.0.4 Build 19111
 Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

2012-06-23 13:29:33.156/6.938 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded Reporter configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/reports/report-group.xml"
2012-06-23 13:29:33.500/7.282 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/E:/Oracle/Middleware/oepe_11gR1PS4/workspace/calcular-tamanho-cache-coherence/lib/coherence.jar!/coherence-cache-config.xml"
2012-06-23 13:29:35.391/9.173 Oracle Coherence GE 3.6.0.4 <D4> (thread=Main Thread, member=n/a): TCMP bound to /192.168.177.133:8090 using SystemSocketProvider
2012-06-23 13:29:37.062/10.844 Oracle Coherence GE 3.6.0.4 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "cluster:0xC4DB" with senior Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-06-23 13:29:37.172/10.954 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedCache with senior member 1
2012-06-23 13:29:37.188/10.970 Oracle Coherence GE 3.6.0.4 <Info> (thread=Main Thread, member=n/a): Started cluster Name=cluster:0xC4DB

Group{Address=224.3.6.0, Port=36000, TTL=4}

MasterMemberSet
  (
  ThisMember=Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
  OldestMember=Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
  ActualMemberSet=MemberSet(Size=2, BitSetCount=2
    Member(Id=1, Timestamp=2012-06-23 13:29:14.031, Address=192.168.177.133:8088, MachineId=55685, Location=process:1128, Role=CreatePeopleCacheAndPopulateWith)
    Member(Id=2, Timestamp=2012-06-23 13:29:36.899, Address=192.168.177.133:8090, MachineId=55685, Location=process:244, Role=Oracle)
    )
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0, BitSetCount=0
    )
  )

TcpRing{Connections=[1]}
IpMonitor{AddressListSize=0}

2012-06-23 13:29:37.891/11.673 Oracle Coherence GE 3.6.0.4 <D5> (thread=Invocation:Management, member=2): Service Management joined the cluster with senior service member 1
2012-06-23 13:29:39.203/12.985 Oracle Coherence GE 3.6.0.4 <D5> (thread=DistributedCache, member=2): Service DistributedCache joined the cluster with senior service member 1
2012-06-23 13:29:39.297/13.079 Oracle Coherence GE 3.6.0.4 <D4> (thread=DistributedCache, member=2): Asking member 1 for 128 primary partitions

Cache Statistics of 'People':
   - Total Entries of Cache -----> 3
   - Used Memory (Bytes) --------> 883920
   - Used Memory (MB) -----------> 0.843
   - Object Average Size --------> 294640

I hope that this post could save you some time when calculate the total size of Coherence cache became a requirement for your high scalable system using data grids. See you!

Wednesday Feb 22, 2012

Oracle Coherence: First Steps Using Clusters and Basic API Usage

When we talk about distributed data grids, elastic caching platforms and in-memory caching technologies, Oracle Coherence is the first option that came in our minds. This happens because Oracle Coherence is the oldest and most mature implementation of data grids, creating successful histories across the world. It is Oracle Coherence the implementation with the bigger number of use cases in the world. Since it's aquisition by Oracle in 2007, the product has been enhanced with powerful enterprise features to remain it's position of the "better of the world" against it's competitors.


This article will help you given your first steps with Oracle Coherence. I have prepared a sequence of three videos that will guide you in the process of creating a data grid cluster, managing data using both Java API and CohQL ("Coherence Query Language") to finally test the reliability and fail over features of the product.

Oracle allows you to download and use any of your products for free, if you are interested in learning or testing the technology. Different  of other vendors that put you first in contact with a sales representative or simply not put their software available for download, Oracle encourages you to use the technology so you gain confidence with it. You can download Oracle Coherence at this link. If you don't possess a credential in the OTN ("Oracle Technology Network"), you will be asked to create one.

If you have a powerful computer and a fastest internet bandwidth, change the video quality settings to 1080p HD ("High Definition"). It will improve considerably the quality of your viewing.

About

Ricardo Ferreira is just a regular person that lives in Brazil and is passionate for technology, movies and his whole family. Currently is working at Oracle Corporation.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today