Monitoring Data -- How To Open The Firehose

Are you interested in getting raw Monitoring Data without using JavaScript or Dtrace?  That's why we include the special command:

asadmin get -m

Looks can be deceiving.  It appears that the -m is just some command line option.  Not so.  That turns on a huge pile of code that plows through the run-time tree of monitoring data.  This data can be accessed from instances, clusters of instances and DAS -- in fact ALL the data from all of them can be combined and accessed all at the same time!

This command was available in V3.0  Unfortunately it was very slow if you wanted to collect everything (what I call Firehose Mode).  So what was done was a lot of regular expression parsing to narrow down the matching data in order to get things going faster.

I reworked this command for GlassFish 3.1 to support the clustering feature.  I started DAS (Domain Administration Server) and one instance.  I ran

asadmin get -m "\*"  // Firehose output for DAS and the instance

Then I waited.  And waited.  And waited and waited.  About 90 seconds later the server got an out-of-memory error.  I never saw any output at all.  Oops!

I discovered the performance bottle-neck.  It was unbelievable what the problem is.  If you are interested I've blogged about it here.

We spent a lot of time trying to get the get -m command to handle regular expressions well.  Below are some examples. 

My environment: I have a cluster, c1.  c1 has 2 instances, i1 and i2.  Both instances are running with monitoring-levels all set to HIGH.  DAS is also running.  All servers are running on the same machine so I don't muddy the water with network latency etc.

  • asadmin get -m c1 # this retrieves ALL monitoring data for every running instance in c1 (~14,000 lines of output)
  • asadmin get -m c1.\* # does exactly the same thing as above
  • asadmin get -m i2.web.servlet.activeservletsloadedcount-current  # Example drill-down.
    i2.web.servlet.activeservletsloadedcount-current = 6
  • asadmin get -m "\*" | wc -l # returned 20,000+ lines.  It took around 1 second on my laptop.  
  • asadmin get -m \*.web.servlet.activeservletsloadedcount-current
    server.web.servlet.activeservletsloadedcount-current = 2  #DAS
    i1.web.servlet.activeservletsloadedcount-current = 6  
    i2.web.servlet.activeservletsloadedcount-current = 6

The command is so incredibly fast you can simply collect everything in a file and then run text processing commands on that file and avoid calling the command multiple times:

asadmin get -m "\*" > output

grep hello output # just an example



Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

ByronNevins

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today