Monday Jan 19, 2015

The Importance of Organizing Spatial Data By Proximity

Guest Post By: Nick Salem, Distinguished Engineer, Neustar

Note: Thanks to Nick Salem, Technical Chair of the Oracle Spatial SIG , for contributing this valuable tuning and performance tip that will be useful to most Oracle Spatial users!

The goal of this post is to shed some light on a technique that I feel is many times overlooked by users working to tune large spatial datasets: to organize the rows in a table by their spatial proximity. Dan Geringer alluded to this in the post “Tips on tuning SDO_NN (nearest neighbor) queries” . What I want to highlight is that this technique is not only beneficial for SDO_NN queries, but also for basically all queries that use any of the spatial query operators like SDO_FILTER, SDO_RELATE, SDO_WITHIN_DISTANCE, as well as the SDO_RELATE helper operators such as SDO_ANYINTERACT, SDO_INSIDE and so on. The SDO_JOIN operator by itself may not benefit from this approach because it relies solely on the spatial index, but if you decide to join the resultset of the SDO_JOIN operation back to the input tables of the join operation, then you will most likely also benefit from having the data stored by proximity.

To understand the dynamics of this, one has to understand in general how Oracle queries work. When a user issues a spatial query, a spatial index is used to find the rowids for the rows that need to be returned. Oracle uses these rowids to retrieve the database blocks from disk into the buffer cache where it can process the data. Each database block can contain one or more rows of data. In order for Oracle to retrieve data for a query, it needs to retrieve all the blocks needed to satisfy a query. The more blocks that need to be scanned, the more I/O operations are performed and the longer the query will take. For example, a spatial query resulting in the scan of one or two adjacent blocks will return a lot faster than the same spatial query needing to scan a large number of disparate blocks to process the same results. Of course, this issue is not confined to just spatial queries. In fact, most Oracle DBAs are aware of this with the concept of the index clustering factor which describes the relationship between the order of an index and its corresponding table. For non-spatial data, achieving an optimal index clustering factor can be as easy as ordering the data in a table by a scalar data type column that you plan to index and then creating that index. For spatial data, this can be a little trickier because one cannot just simply order by the SDO_GEOMETRY column. In Dan’s post “Tips on tuning SDO_NN (nearest neighbor) queries”, he shows an example using the MD.HHENCODE_BYLEVEL function to return a value that you can use to sort your spatial data by.

In this post, I will show an example of a use case I tested that highlights the impact on performance when ordering spatial data by proximity. The example will include taking the US Business listing table and creating two copies: one that is not ordered by proximity, and the other ordered by proximity using the MD.HHENCODE_BYLEVEL function. Then, I will test running a simple SDO_WITHIN_DISTANCE query on both tables to retrieve the sum of employees for an arbitrary 5 mile radius and compare the results.

Example Pre-requisites

In this example, I am starting with a 16 million row table containing all the US business locations with approx 30 columns called BUSINESS1. This table is not ordered by spatial proximity. BUSINESS1 also has a spatial index created on column GEOM.

Step 1) Next I will create table BUSINESS2, which will be an exact copy of BUSINESS1, but will be ordered by proximity using the MD.HHENCODE_BY_LEVEL function as described in Dan’s post.


CREATE TABLE business2 PCTFREE 0 NOLOGGING PARALLEL
as
SELECT /*+ parallel(8) */
  b.*
FROM
  business1 b
ORDER BY
  row_number() over (order by md.hhencode_bylevel(
     b.geom.sdo_point.x,-180,180,27,
     b.geom.sdo_point.y,-90,90,26)) ;

* Note: the MD.HHENCODE_BYLEVEL function takes an x and y coordinate and so the example works well with the US Business points. If you are working with polygons instead of points, you will need to retrieve the centroid of the shape and then pass the X/Y point coordinates to function. For more info, please refer to Dan Geringer’s original post “Tips on tuning SDO_NN (nearest neighbor) queries” to see an example of how this is done.

* Note: only use PCTFREE 0 for read only tables. In this example, the BUSINESS2 is read only and using PCTFREE 0 allows more rows to be packed into a single database block.

Step 2) add diminfo and create spatial index

begin

mdsys.sdo_meta.change_all_sdo_geom_metadata( USER, ‘BUSINESS2’, ‘GEOM’, mdsys.sdo_dim_array (
     mdsys.sdo_dim_element('X', -180, 180, .5),
     mdsys.sdo_dim_element('Y', -90, 90, .5) ),8307);
end;
/

CREATE INDEX xsp_business2 ON business2(geom)
INDEXTYPE is mdsys.spatial_index PARAMETERS(' SDO_RTR_PCTFREE=0 WORK_TABLESPACE=WORK');

* Note: it is recommended to use a WORK_TABLESPACE that is a different tablespace than the one where the index will be created. Although optional, the use of a WORK_TABLESPACE can reduce the fragmentation of your spatial index which is important for performance.

* Note: only use SDO_RTR_PCTFREE=0 for read only data. In this example, the US BUSINESS listing table is pretty much a read only dataset and gets completely replaced every month.

Step 3) Perform query comparison test

In SQL*PLUS, set timing and flush the buffer cache

SQL> SET TIMING ON;
SQL>
SQL> ALTER SYSTEM FLUSH BUFFER_CACHE;

System altered.

* Note: you should not flush the buffer cache on a production system. But in a test environment, flushing the buffer cache can help with testing comparative performance by ensuring there are no blocks in the SGA from prior queries that could skew performance results.

Run the first query on the non-spatially ordered table

select
  sum(b.num_of_employees)
from
 business1 b
where
  sdo_within_distance( b.geom,
                       mdsys.sdo_geometry(2001,8307,
                       mdsys.sdo_point_type( -117.047071, 32.75521, null),null,null),
                       'distance=5 unit=mile' ) = 'TRUE';

SUM(B.NUM_OF_EMPLOYEES)
-----------------------
                 137964

Elapsed: 00:00:35.14

Run the query on the spatially ordered business table and compare results.

select
  sum(b.num_of_employees)
from
 business2 b
where
  sdo_within_distance( b.geom,
                       mdsys.sdo_geometry(2001,8307,
                       mdsys.sdo_point_type( -117.047071, 32.75521, null),null,null),
                       'distance=5 unit=mile' ) = 'TRUE';

SUM(B.NUM_OF_EMPLOYEES)
-----------------------
                 137964

Elapsed: 00:00:01.49

Conclusion

In this example, I was able to achieve more than 20X better performance just by taking the same table and ordering by a geographic extent. The queries in the example utilized the spatial operator SDO_WITHIN_DISTANCE to return the sum of employees with a 5 mile radius around a location. Since both tables (BUSINESS1 and BUSINESS2) are identical except in storage of the order of rows of data, the spatial index performance should be pretty much the same. The difference in performance that we are seeing is due to the amount of disk and memory I/O processing caused by the different number of blocks that needed to be accessed for each of the queries. This is a substantial improvement in performance and highlights the importance of the order of spatial data by geographic proximity. And as mentioned in the beginning of the post, ordering spatial data may boost any spatial query operations, whether you are performing within distance queries, sdo relate queries, nearest neighbor or performing map visualizations. Results can vary based on size of table, the speed of disk I/O and also the order of the original dataset. It is possible that the original dataset you are working with is already organized by some geographic extent such as county or ZIP Code so additional ordering using the method described in this post could result in some performance gains but nothing as significant as the 20X I have experienced. But if the order of the table is completely random and not tied to any geographic extent, then you can expect to see greater performance gains. All in all, I definitely recommend looking into organizing larger and more frequently queried spatial data by geographic proximity as a best practice technique for optimizing your spatial data.


Tuesday Apr 01, 2014

Upcoming Webinars: MapViewer at City of Toronto for Public Safety, Customers Achieve 300x Performance Gains with Oracle Spatial and Graph

A note to share information about two upcoming Directions Media webinars on April 23 and May 6.

City of Toronto Enhances Public Safety Using Real-time Big Data and Map Rendering with Oracle and AGSI, Wed., April 23, 2:00PM US EDT

Learn how the City of Toronto Police Services can search, review and map social media traffic in real time to quickly identify and respond to incidents, improving public safety. See live demos of their system using Oracle MapViewer's HTML5 capabilities, Oracle Spatial, and a social media mapping platform from partner AGSI. Carol Palmer of Oracle will co-present this webinar with Mike Jander of AGSI, and City of Toronto, hosted by Directions Media.

Learn more and register for this free webinar -
http://www.directionsmag.com/webinars/register/city-of-toronto-enhances-public-safety-using-real-time-big-data-and-ma/389356?DM_webinars_section&utm_medium=web&utm_campaign=389356


Learn How Customers Are Experiencing 300x Performance Gains with Oracle Spatial and Graph, Tues., May 6, 2:00PM US EDT (Free Webinar)

Nick Salem of Neustar and Steve Pierce of Think Huddle will share their realized performance benchmarks using Oracle Spatial and Graph. With Oracle Spatial and Graph in Database 12c, customers can address the largest geospatial workloads and experience performance increases of 50 to 300 times for vector operations, with minimal configuration changes. Jim Steiner of Oracle will also discuss performance gains from parallel raster processing and Exadata.

Learn more and register for this free webinar -
http://www.directionsmag.com/webinars/register/learn-how-customers-are-experiencing-300x-performance-gains-with-oracl/390239?DM_webinars_section&utm_medium=web&utm_campaign=390239

Monday Feb 24, 2014

Performance Boost with Aggregating Geometries - with 12c Spatial Vector Performance Acceleration

Nick Salem of Neustar recently shared some impressive performance gains realized using Oracle Spatial 12c Vector Performance Acceleration, in a Google+ post.   He observed performance gains of 40X to 300X -- with a use case aggregating all ZIP Code geographies in California using a SDO_AGGR_UNION operations (plain and with group by/mod functions as described in 11g documented best practices). 

Read more details of his test case and results here.  https://plus.google.com/114373574274269737617/posts/2caAypKxwff

Thanks for sharing, Nick!

Friday Jan 10, 2014

New Point-in-Polygon function in Oracle Spatial and Graph 12c

By: Jim Steiner, Siva Ravada, Rick Anderson

With the increased adoption of Exadata for spatial workloads, we have been looking at ways to exploit more and more of the capabilities of this architecture to address problems faced in large scale spatial analysis. The new point-in-polygon function in Spatial can result in 100s of times faster UPDATE and INSERT operations with no degradation in query performance for large scale point-in-polygon operations. Mask operations (DISJOINT, TOUCH, INSIDE, ANYINTERACT) can be performed with the point-in-polygon function.

When working with point data and performing point-in-polygon analysis the existing spatial operators to do a fast query on the data if there is a Spatial index on the point data. However, in many cases, the data volume is very high, so creating and maintaining the index becomes very expensive. With 12c, we exploit Exadata smartscan by implementing a different model to take advantage of all the CPUs to do point in polygon operations and not have the overhead of a Spatial index. The mdsys.PointInPolygon() function returns those rows that reside within a specified polygon geometry. This parallel-enabled Point-In-Polygon function takes an arbitrary set of rows whose first column is a point's x-coordinate value and the second column is a point's y-coordinate value.

The mdsys.PointInPolygon() function API is the following:

mdsys.sdo_PointInPolygon(cur SYS_REFCURSOR,
                         geom_obj IN SDO_GEOMETRY,
                         tol IN NUMBER,
                         params IN VARCHAR2 DEFAULT NULL);

The "cur" parameter is used to select an "x" and "y" point coordinate from a

user table. The two columns must be of type NUMBER; this is NOT a geometry

parameter.

The "geom_obj" parameter is either a polygon geometry from a table, or a

Transient instance of a polygon geometry, against which all of the selected points from "cur" will be validated.

The "tol" parameter is the desired tolerance value, which must be greater than the value "0.0".



The following examples show the performance benefits of this new approach:

Here we select all rows from the "weather_sensor" table and query those rows against a transient polygon geometry instance. Only 1 weather_sensor row (out of 4) resides within the specified polygon.

SQL> SELECT *

2 FROM TABLE(mdsys.sdo_PointInPolygon(

3 CURSOR(select * from weather_sensor),

4 MDSYS.SDO_GEOMETRY(

5 2003,

6 NULL,

7 NULL,

8 MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 1),

9 MDSYS.SDO_ORDINATE_ARRAY(5, 1, 8, 1, 8, 6, 5, 7, 5, 1)),

10 0.05));

In order to utilize parallel query servers, you must either specify the

"/*+ PARALLEL(4) */" optimizer hint, or enable parallel query execution,

using the command:

alter session force parallel query;

Below is the same as above, but uses 4 parallel query servers:

SQL> SELECT /*+ PARALLEL(4) */ *

2 FROM TABLE(mdsys.sdo_PointInPolygon(

3 CURSOR(select * from weather_sensor),

4 MDSYS.SDO_GEOMETRY(

5 2003,

6 NULL,

7 NULL,

8 MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 1),

9 MDSYS.SDO_ORDINATE_ARRAY(5, 1, 8, 1, 8, 6, 5, 7, 5, 1)),

10 0.05));

There can be a huge performance benefit to using parallel query servers.

The following "worst-case" example queries 1-million rows against a transient

polygon geometry instance, using the non-parallel query execution:

SQL> -- instead of the actual data...

SQL>

SQL> -- Test "non-parallel" execution first

SQL> timing start "sdo_PointInPolygon()"

SQL> SELECT COUNT(*)

2 FROM TABLE(mdsys.sdo_PointInPolygon(

3 CURSOR(select * from pip_data),

4 MDSYS.SDO_GEOMETRY(

5 2003,

6 NULL,

7 NULL,

8 MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 1),

9 MDSYS.SDO_ORDINATE_ARRAY(5, 1, 8, 1, 8, 6, 5, 7, 5, 1)),

10 0.05));

timing for: sdo_PointInPolygon()

Elapsed: 00:05:00.73

Enabling the parallel query servers dramatically reduces the query execution time:

SQL> -- Now test using 4 parallel query servers

SQL> timing start "sdo_PointInPolygon()"

SQL> SELECT /*+ PARALLEL(4) */ COUNT(*)

2 FROM TABLE(mdsys.sdo_PointInPolygon(

3 CURSOR(select * from pip_data),

4 MDSYS.SDO_GEOMETRY(

5 2003,

6 NULL,

7 NULL,

8 MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 1),

9 MDSYS.SDO_ORDINATE_ARRAY(5, 1, 8, 1, 8, 6, 5, 7, 5, 1)),

10 0.05));

SQL> timing stop

timing for: sdo_PointInPolygon()

Elapsed: 00:02:18.18

For more information about this new feature, link to the documentation URL:
SDO_PointInPolygon


About

The official blog for the spatial features of Oracle Spatial and Graph, an option of Oracle Database - brought to you by the product managers and developers. Get technical tips, product information, and the latest news here. Visit our official product website at http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/spatialfeatures-1902020.html

Search

Categories
Archives
« May 2015
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
29
30
31
      
Today