X

Oracle Big Data Spatial and Graph - technical tips, best practices, and news from the product team

How Big is My In-Memory Graph?

Alan Wu
Architect

This blog details the steps I used to achieve the following:

  • Start a remote PGX instance with customized server settings
  • Load, from Groovy shell, a pair of property graph flat files (.opv/.ope) into a remote PGX instance
  • Measure the memory consumption

 The test environment used was Oracle BigDataLite VM (v4.8) and Oracle Big Data Spatial and Graph (BDSG v2.2). Updated configuration files and scripts are listed below.

[oracle@bigdatalite ~]$ cd /opt/oracle/oracle-spatial-graph/property_graph/pgx/bin/
[oracle@bigdatalite bin]$ cat ../conf/server.conf
{
  "port": 7007,
  "enable_tls": false,
  "enable_client_authentication": false
}

[oracle@bigdatalite bin]$ cat ../conf/pgx.conf
{
  "allow_idle_timeout_overwrite": true,
  "allow_task_timeout_overwrite": true,
  "allow_local_filesystem": true,
  "enable_gm_compiler": true,
  "graphs": [],
  "max_active_sessions": 1024,
  "max_queue_size_per_session": -1,
  "max_snapshot_count": 0,
  "memory_cleanup_interval": 600,
  "num_workers_analysis": "<no-of-CPUs>",
  "num_workers_fast_track_analysis": 1,
  "num_workers_io": "<no-of-CPUs>",
  "path_to_gm_compiler": null,
  "release_memory_threshold": 0.85,
  "session_idle_timeout_secs": 0,
  "session_task_timeout_secs": 0,
  "strict_mode": true,
  "tmp_dir": "<system-tmp-dir>"
}


[oracle@bigdatalite bin]$ export _JAVA_OPTIONS="-Dpgx.num_workers_io=2 -Dpgx.max_off_heap_size=4000 -Dpgx.num_workers_analysis=3
-Xmx6000m "

[oracle@bigdatalite bin]$ sh ./start-server
Picked up _JAVA_OPTIONS: -Dpgx.num_workers_io=2
-Dpgx.max_off_heap_size=4000 -Dpgx.num_workers_analysis=3 -Xmx6000m

...

oracle@bigdatalite ~]$ cd /opt/oracle/oracle-spatial-graph/property_graph/dal/groovy/
[oracle@bigdatalite groovy]$
[oracle@bigdatalite groovy]$ sh gremlin-opg-nosql.sh

pgxServer = "http://127.0.0.1:7007/"
pgxSession = Pgx.getInstance(pgxServer).createSession("graph-session");


vfile="file:///u02/mygraph.opv"  // vertex flat file
efile="file:///u02/mygraph.ope"  // edge flat file

config = GraphConfigBuilder .forMultipleFileFormat(Format.FLAT_FILE) .setSeparator(",") .addVertexUri(vfile) .addEdgeUri(efile).addVertexProperty("first_name", PropertyType.STRING, "null").build();


opg-nosql> g = pgxSession.readGraphWithProperties(config);
==>PgxGraph[name=...,N=1600000,E=5399998,created=1499720843627]

opg-nosql> g.getMemoryMb()
==>533

opg-nosql>
 

The last two steps print the size of the graph: # of vertices, edges, and also an estimated memory consumption in MB.

Hope it helps,

Zhe

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.