X

Oracle Big Data Spatial and Graph - technical tips, best practices, and news from the product team

Adding Visualization to Python-Notebook Based BDSG Property Graph Workflow

Alan Wu
Architect

This is a joint work with Isai Barajas (primary contributor) and John Wyant.

Python Notebook is a very convenient tool for one to build a
workflow/demo based on Oracle Big Data Spatial and Graph (BDSG) property
graph. In this blog, we are going to demonstrate how to add a bit
visualization to a Python-Notebook based property graph workflow.

    Requirement: Python 2.7, JPipe 0.57 and pyopg (under /opt/oracle/oracle-spatial-graph), D3.js version 3, Big Data Lite VM 4.6, and Oracle Big Data Spatial and Graph v2.0

Detailed steps are as follows. Please cut & paste the code snippets into Python Notebook.

 Step 1: Specify a few necessary libraries and imports

import matplotlib as mpl
import matplotlib.pyplot as plt
import sys
default_stdout = sys.stdout
default_stderr = sys.stderr
reload(sys)
sys.setdefaultencoding("utf-8")
sys.stdout = default_stdout
sys.stderr = default_stderr
from pyopg.core import *
pgx_config = JPackage('oracle.pgx.config')
pgx_types = JPackage('oracle.pgx.common.types')
pgx_control = JPackage('oracle.pgx.api')
hbase = JPackage('oracle.pg.hbase')

 Step 2: Create a graph configuration

graph_builder = pgx_config.GraphConfigBuilder.forPropertyGraphHbase() \
.setName("connections").setZkQuorum("bigdatalite").setZkClientPort(2181) \
.setZkSessionTimeout(120000).setInitialEdgeNumRegions(3) \
.setInitialVertexNumRegions(3).setSplitsPerRegion(1) \
graph_builder.addEdgeProperty("weight", pgx_types.PropertyType.DOUBLE, "1000000")

Step 3: Read the "connections" graph into the in-memory analyst

opg = hbase.OraclePropertyGraph.getInstance(graph_builder.build())
pgx_param = JClass("java.util.HashMap")()
instance = JClass("oracle.pgx.api.Pgx").getInstance()
if not instance.isEngineRunning():instance.startEngine(pgx_param)
session = instance.createSession("my_recommender_session1")
analyst = session.createAnalyst()

pgxGraph = session.readGraphWithProperties(opg.getConfig(), True)
pgxGraph.getNumEdges()

Step 3.1 (optional): Read out a few vertices

for element in range(1,10,1):vertex = opg.getVertex(element)print 'Vertex ID: ' + str(element) + ' - Name: ' + vertex.getProperty("name")

Vertex ID: 1 - Name: Barack Obama
Vertex ID: 2 - Name: Beyonce
...

 Step 4: Create JSON objects (nodes, links) out of edges (and vertices) that we want to visualize

# Get Edges
edges = opg.getEdges().iterator();
edge = edges.next()
# Dictiony for Nodes and Links
nodes = []
links = []
names = []
sources = []
targets = []
values = []
# Get Nodes
for count in range(1,20,1):# Vertex ValuesoutVertexName = edge.getOutVertex().getProperty("name")outVertexRole = edge.getOutVertex().getProperty("country")inVertexName = edge.getInVertex().getProperty("name")inVertexRole = edge.getInVertex().getProperty("country")
# Add out Vertexif {"name": outVertexName, "group": outVertexRole} not in nodes:nodes.append({"name": outVertexName, "group": outVertexRole})names.append(outVertexName)
# Add in Vertexif {"name": inVertexName, "group": inVertexRole} not in nodes:nodes.append({"name": inVertexName, "group": inVertexRole})names.append(inVertexName)
# Edge Informationsources.append(outVertexName)targets.append(inVertexName)values.append(edge.getLabel())
# Next Edgeedge = edges.next()
# Get Links
for count in range(0,19,1):# Vertex ValuesoutVertexName = sources[count]inVertexName = targets[count]
# Edge Valuessource = names.index(outVertexName)target = names.index(inVertexName)value = values[count]links.append({"source": source, "target": target, "value": value})


from IPython.display import Javascript
import json
# Transform the graph into a JSON graph
data = {"nodes":nodes, "links":links}
jsonGraph = json.dumps(data, indent=4)
# Send to Javascript
Javascript("""window.jsonGraph={};""".format(jsonGraph))

Step 5: Set up a <div>...</div> for graph plotting

%%html
<div id="d3-example"></div>
<style>
.node {stroke: #fff; stroke-width: 1.5px;}
.link {stroke: #999; stroke-opacity: 5.6;}
</style>

leadsadmirescollaboratesfeudsfeudscollaboratescollaboratesadmiresfeudsfeudsfeudscollaboratesadmirescollaboratesadmiresfeudscollaboratesfeudsadmiresName: Tony Fadell
Country: United States
Name: Nest
Country: United States
Name: The Academy
Country: United States
Name: Matthew McConaughey
Country: United States
Name: Barack Obama
Country: United States
Name: Kirsten Gillibrand
Country: United States
Name: Vladimir Putin
Country: Russia
Name: Angela Merkel
Country: Germany
Name: David Koch
Country: United States
Name: Eric Holder
Country: United States
Name: Nicolas Guerekoyame Gbangou
Country: null
Name: Omar Kobine Layama
Country: null
Name: Beyonce
Country: United States
Name: Serena Williams
Country: United States
Name: Facebook
Country: United States
Name: Google
Country: United States
Name: CBS
Country: United States
Name: ABC
Country: United States
Name: Xi Jinping
Country: China
Name: Shinzo Abe
Country: Japan
Name: Tom Steyer
Country: United States
Name: Kristen Anderson-Lopez
Country: United States
Name: H.R. McMaster
Country: United States
Name: Charlie Rose
Country: United States
Name: Pope Francis
Country: Italy
Name: Hillary Clinton
Country: United States
Tony FadellNestThe AcademyMatthew McConau...Barack ObamaKirsten Gillibr...Vladimir PutinAngela MerkelDavid KochEric HolderNicolas Guereko...Omar Kobine Lay...BeyonceSerena WilliamsFacebookGoogleCBSABCXi JinpingShinzo AbeTom SteyerKristen Anderso...H.R. McMasterCharlie RosePope FrancisHillary Clinton

 Step 6: Graph processing with D3 Force-directed layout

%%javascript
// We load the d3.js library from the Web.
require.config({paths: {d3: "http://d3js.org/d3.v3.min"}});
require(["d3"], function(d3) {// The code in this block is executed when the // d3.js library has been loaded.
// First, we specify the size of the canvas containing// the visualization (size of the <div> element).var width = 800, height = 600;// We create a color scale.var color = d3.scale.category20();// We create a force-directed dynamic graph layout.var force = d3.layout.force().charge(-300).linkDistance(100).size([width, height]);// In the <div> element, we create a <svg> graphic// that will contain our interactive visualization.var svg = d3.select("#d3-example").select("svg")if (svg.empty()) {svg = d3.select("#d3-example").append("svg").attr("width", width).attr("height", height);}
// We load the JSON graph we generated from iPython inputvar graph = window.jsonGraph; plotGraph(graph);
// Graph Plot functionfunction plotGraph(graph) {// We load the nodes and links in the force-directed graph.force.nodes(graph.nodes).links(graph.links).start();// We create a <line> SVG element for each link in the graph.var link = svg.selectAll(".link").data(graph.links).enter().append("line").attr("class", "link").attr("stroke-width", 7);// Link Valuelink.append("title").text(function(d) { return d.value; });
// We create a <circle> SVG element for each node// in the graph, and we specify a few attributes.var node = svg.selectAll(".node").data(graph.nodes).enter().append("circle").attr("class", "node").attr("r", 16) // radius.style("fill", function(d) {// The node color depends on the club.return color(d.group); }).call(force.drag); // The name of each node is the node number.node.append("title").text(function(d) { var info = "Name: " + d.name + "\n" + "Country: " + d.group;return info; });// Text Over Nodesvar text = svg.append("g").selectAll("text").data(force.nodes()).enter().append("text").attr("x", function(d) { return -10 }).attr("y", 0).style("font-size","10px").text(function(d) { if (d.name.length > 15) {return d.name.substring(0, 15) + "...";}return d.name; });
// We bind the positions of the SVG elements// to the positions of the dynamic force-directed graph,// at each time step.force.on("tick", function() {link.attr("x1", function(d) { return d.source.x; }).attr("y1", function(d) { return d.source.y; }).attr("x2", function(d) { return d.target.x; }).attr("y2", function(d) { return d.target.y; });node.attr("cx", function(d) { return d.x; }).attr("cy", function(d) { return d.y; }); text.attr("transform", function(d) {return "translate(" + d.x + "," + d.y + ")";}); });}
});

Acknowledgement: thanks Jay Banerjee for his input on this blog post.



Join the discussion

Comments ( 1 )
  • sunita soni Wednesday, May 30, 2018
    This really has covered a great insight on Python. I found myself lucky to visit your page and came across this insightful read on Python tutorial. Please allow me to share similar work on Python training course . Watch and gain knowledge today.https://www.youtube.com/watch?v=XmfgjNoY9PQ
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.