X

Oracle Big Data Spatial and Graph - technical tips, best practices, and news from the product team

  • August 27, 2018

Reading Directly from Flat Files (.opv/.ope) into PGX

Alan Wu
Architect

Per a request from Karin, a good friend of mine, I am showing in this blog a quick example on reading directly from flat files into an in-memory PGX graph.

As usual, I am using a Groovy script. The following actually works for both Big Data Spatial and Graph and Oracle Spatial and Graph. The key things are to use Format.FLAT_FILE and also do not forget to set the field separator to be a comma.

opg-hbase> graphConfig = GraphConfigBuilder.forFileFormat(Format.FLAT_FILE) .setVertexUris("../../data/connections.opv").setEdgeUris("../../data/connections.ope") .setSeparator(",").addVertexProperty("name", PropertyType.STRING, "none"). build();
==>{"attributes":{},"separator":",","vertex_id_type":"integer","vertex_uris":["../../data/connections.opv"],"format":"flat_file","loading":{},"vertex_props":[{"default":"none","type":"string","name":"name"},{"type":"double","name":"base_power"}],"error_handling":{},"edge_props":[],"edge_uris":["../../data/connections.ope"]}
opg-hbase> 
opg-hbase> 
opg-hbase> pgxGraph = session.readGraphWithProperties(graphConfig)
==>PgxGraph[name=connections_3,N=78,E=164,created=1535402672326]
opg-hbase> 
opg-hbase> pgxGraph.getVertex(1)
==>PgxVertex[ID=1]
opg-hbase> pgxGraph.getVertex(1).getProperty("name")
==>Barack Obama


opg-hbase> pgxGraph.getVertex(2)
==>PgxVertex[ID=2]
opg-hbase> pgxGraph.getVertex(2).getProperty("name")
==>Beyonce
 

One can of course run PGQL against this graph.

opg-hbase> pgxResultSet = pgxGraph.queryPgql("SELECT n WHERE (n) limit 10")
==>oracle.pgx.api.ResultImpl@7ef41ca2
...
 

Further, this in-memory graph can be saved back into a pair of flat files using the pgxGraph.store() Java API.

opg-hbase> graphConfig1 = GraphConfigBuilder.forFileFormat(Format.FLAT_FILE) .setVertexUris("/tmp/new_connections.opv").setEdgeUris("/tmp/new_connections.ope") .setSeparator(",").addVertexProperty("name", PropertyType.STRING, "none").addVertexProperty("base_power", PropertyType.DOUBLE). build();
==>{"attributes":{},"separator":",","vertex_id_type":"integer","vertex_uris":["/tmp/new_connections.opv"],"format":"flat_file","loading":{},"vertex_props":[{"default":"none","type":"string","name":"name"},{"type":"double","name":"base_power"}],"error_handling":{},"edge_props":[],"edge_uris":["/tmp/new_connections.ope"]}
 

opg-hbase> pgxGraph.store(graphConfig1, true)
==>{"attributes":{},"separator":",","vertex_id_type":"integer","vertex_uris":["/tmp/new_connections.opv"],"format":"flat_file","loading":{},"vertex_props":[{"default":"none","type":"string","name":"name"},{"type":"double","name":"base_power"}],"error_handling":{},"edge_props":[],"edge_uris":["/tmp/new_connections.ope"]}
opg-hbase> :quit

[oracle@bigdatalite groovy]$ head -2 /tmp/new_connections.opv
1,name,1,Barack%20Obama,,
1,base_power,4,,0.0,


[oracle@bigdatalite groovy]$ head -2 /tmp/new_connections.ope
0,1,2,,%20,,,,
1,1,3,,%20,,,,

Cheers,

Zhe

Join the discussion

Comments ( 1 )
  • Karin Friday, August 31, 2018
    Zhe,

    This works like a charm. I have tested it in a Zeppelin Notebook and everything works as expected.

    There are 2 more things to remember when reading the graph directly from .opv and .ope:
    1. Vertex labels are currently not supported for flat file format.
    2. You need to add all vertex and edge properties to the config file.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha