X

Information, tips, tricks and sample code for Big Data Warehousing in an autonomous, cloud-driven world

Recent Posts

Autonomous

SQL Macros Have Arrived in Autonomous Database

If you saw last week's Autonomous Database newsletter you will have noticed that one of the new features that I announced was support for SQL Macros), see here and if you want to sign-up for the Autonomous Database newsletter then see here. Whilst the newsletter included a link to the SQL documentation, I thought it would be useful to write blog post to provides some more insight into how this new feature can make your SQL code both simpler and faster and give you yet another great reason to move your workloads to the cloud. Lets get started... What is a SQL Macro It's a new, simpler way to encapsulate complex processing logic directly within SQL. SQL Macros allow developers to encapsulate complex processing within a new structure called a "macro" which can then be used within SQL statement. Essentially there two types of SQL Macros: SCALAR and TABLE. What's the difference: SCALAR expressions can be used in SELECT list, WHERE/HAVING, GROUP BY/ORDER BY clauses   TABLE expressions used in a FROM-clause Why Do We Need SQL Macros When We Have PL/SQL? In the past, if you wanted to extend the capabilities of SQL, then PL/SQL was the usually the best way to do add additional, more complex processing logic. However, by using PL/SQL this created some underlying complexities at execution time since we needed to keep swapping between the SQL context and the PL/SQL context. This tended to have an impact on performance and most people tried to avoid doing it whenever possible.  Why Is A SQL Macro Better Than A PL/SQL Function? SQL Macros have an important advantage over ordinary PL/SQL functions in that they make the reusable SQL code completely transparent to the Optimizer – and that brings big benefits! It makes it possible for the optimizer to transform the original code for efficient execution because the underlying query inside the macro function can be merged into outer query. That means there is no context switching between PL/SQL and SQL and the query inside the macro function is now executed under same snapshot as outer query. So we get both simplicity and faster execution.   How Does Autonomous Database Support SQL Macros As of today, Autonomous Database only supports TABLE macros with support for SCALAR macros coming later. I provided a gentle introduction to SQL Macros at OpenWorld 2019 and you can download the presentation from here: A New Approach To Simplifying Complex SQL. This covers both types of scalar macros so you understand how they can be used today and what you will be able to do shortly when the SCALAR macros are supported. So what can we do today with a TABLE macro?    Two Types of Table Macros There are two types of table macros: 1 Parameterized Views:  tables used in the query are fixed inside the definition of the macro and arguments are passed in to select all or specific rows from those tables. The "shape" of the queries returned from these table is, in most cases, fixed. The most common use of these parameterized views is when you need to pass in arguments to select a subset of the rows that are then aggregated. The first code example below is a good example where I pass in a specific zip code and get the total sales revenue for that zip code. 2. Polymorphic Views: These are more interesting (at least I find them more interesting) because they have one or more table arguments (of course they can also have scalar valued arguments as well). The passed in tables are used inside the query returned by the macro which gives them a lot flexibility. The second set of code examples below show this type of macro and I can see this being of real interest to DBAs because of the overall flexibility and power it provides to create useful utilities. This is probably going to need a follow-up post at some point.   How Do I Create A Table Macro? We define a SQL macro using something like the following syntax: CREATE OR REPLACE FUNCTION total_sales(zip_code VARCHAR2) RETURN VARCHAR2 SQL_MACRO..... the keyword is 'SQL_MACRO' as the return type which is a text string containing SQL that will be substituted into the original statement at execution time. There is a lot that can be done with SQL Macros, simply because they provide so much flexibility, however, my aim here is try and keep things simple just to get us started. Maybe I will go deeper in subsequent blog posts. Let's create a simple table macro...   My First Table Macro For our first table macro let's keep things relatively simple. Suppose I want to have a function that returns the total sales revenue from my fact table SALES for a specific zip code in my CUSTOMERS table. I need to join the two tables together (SALES and CUSTOMERS), find the matching rows and sum the result. In effect, we are creating a paramterized view. The SQL Macro will look like this: create or replace function total_sales(zip_code varchar2) return varchar2 SQL_MACRO(TABLE) is begin   return q'{    select cust.cust_postal_code as zip_code,              sum(amount_sold) as revenue    from sh.customers cust, sh.sales s    where cust.cust_postal_code = total_sales.zip_code     and s.cust_id = cust.cust_id    group by cust.cust_postal_code    order by cust.cust_postal_code   }'; end; /   essentially I am encapsulating a parameter driven SQL query within a PL/SQL function which is of type SQL MACRO. To execute a query using the macro I do this: select * from total_sales('60332'); and the output is as follows: So far so good...the explain plan is very interesting because what you see is not an explain plan for TOTAL_SALES but the SQL within the MACRO which was transparently inserted by the Optimizer...   Obviously the SQL statement within the MACRO can be as simple or complex as needed. But this macro only allows you to query the SALES fact table and CUSTOMERS dimension table. What if we want a more dynamic approach where I could pass in any table to my MACRO for processing?   Going A Little Deeper... In 12c we introduced a new set of keywords to return a subset of rows from a table. Mr Optimizer (aka. Nigel Bayliss) wrote a post about this which you can read here. This allowed you to return the first 10 rows of resultset by using the syntax FETCH FIRST 10 ROWS ONLY. Let's take this new feature and wrap it up in a TABLE MACRO: create or replace function keep_n(n_rows number, t DBMS_TF.Table_t)                   return varchar2 SQL_MACRO(TABLE) is begin   return 'select *            from t           order by 1           fetch first keep_n.n_rows rows only'; end; / this may look complicated but it is relatively simple to explain. The mysterious part for many readers will be the datatype for t which is DBMS_TF.Table_t. Why not simply use a varchar2 string to capture the table name? Because that could lead to SQL injection so we force the input to be a valid table name by enforcing the type via DBMS_TF.Table_t. If you want to learn more about why and how to use DBMS_TF.table_t then I would recommend watching Chris Saxon's webcast from the recent Developer Live event - "21st Century SQL". The above SQL Macro allows me to set the number of rows I want returned from any table and the output is completely dynamic in that all columns in the specified table are returned: select * from keep_n(10, sh.customers);   returns the first 10 rows from my table SH.CUSTOMERS. The following statement returns the first 2 rows from my table SH.CHANNELS:   select * from keep_n(2, sh.channels); and the explain plan for the above query is transformed by the Optimizer to the actual query inside the macro:   select *    from sh.channels   order by 1   fetch first 2 rows only';     Going A Even Deeper.. What if we wanted to extend our table macro to return a random sample of rows from any table? That's a very simple thing to do with a macro! We just need to tweak the FETCH syntax to use PERCENT ROWS and insert an order by clause that uses the DBMS_RANDOM function. Now the macro looks like this... CREATE OR REPLACE FUNCTION row_sampler(t DBMS_TF.Table_t, pct number DEFAULT 5) RETURN VARCHAR2 SQL_MACRO(TABLE) AS  BEGIN  RETURN q'{SELECT *             FROM t            order by dbms_random.value            fetch first row_sampler.pct percent rows only}'; END; / which allows me to randomly sample 15% of rows from my CUSTOMERS table: SELECT * FROM row_sampler(sh.customers, pct=>15);     or my PRODUCTS table: SELECT * FROM row_sampler(sh.products, pct=>15); and again the explain looks as if we had written the the SQL statement in full as: SELECT *  FROM sh.products ORDER BY dbms_random.value FETCH FIRST 15 percent rows only     Conclusion Hopefully this has given you a little bit of a peek into the exciting new world of SQL Macros. This new feature is extremely powerful and I would recommend  watching Chris Saxon's video from the recent Developer Live event - "21st Century SQL" - for more information on how to use SQL Macros with other features such as SQL pattern matching (MATCH_RECOGNIZE). Don't forget to take a look at the documentation, see here.  LiveSQL will shortly be updated to the same patchset release as Autonomous Database and once that happens then I will start posting some code samples that you test and play with on that platform. Don't forget you can always sign up for a 100% free account to run your own Autonomous Database: https://www.oracle.com/cloud/free/ Have fun with SQL Macros!        

If you saw last week's Autonomous Database newsletter you will have noticed that one of the new features that I announced was support for SQL Macros), see here and if you want to sign-up for...

Autonomous

Keep your clone's data up-to-date with Refreshable Clones in Autonomous Database

One of the most widely used features in Autonomous Database on Shared Infrastructure (ADB-S) today is the ability to clone your database, no matter how big or small, with little to no effort. This week, we are souping up ADB's cloning abilities to include Refreshable Clones! A refreshable clone is a read-only clone that stays "connected" to, and has the ability to pull in data (ie. refresh) from, its source database with a simple click of a button. Until now, if you needed to update your clone's data from its source, you had two options: Move new data from the source database to the clone (via data pump, database links etc.) Create a new clone from the source database   Refreshable clones take away the friction of these above options by enabling you to refresh your clone's data by simply hitting a refresh button and providing the source database's timestamp to refresh to (aptly termed the refresh point); the clone automatically looks at your source database's logs and pulls in all the data up to your inputted timestamp. Here are some example use cases where this clone type can be useful to your team: Providing a routinely updated clone to a different business unit within your organization for reporting and analysis  Creating billing or workload separation for your databases between business units within the organization Providing up-to-date read-only test database environments to internal teams Of course, I look forward to learning about how our users make use of this functionality in various different scenarios.   With this context, let me walk you through a simple example of using a refreshable clone showing how powerful this feature can be!   Step 1: Setting up an Autonomous Database After logging into my Oracle Cloud account, I navigate to my existing ADB instance conveniently named "sourceDB". If you missed the memo and haven't created your first Autonomous Database yet, here is a quickstart tutorial on ADB to get you up to speed. To insert a line of data before we clone, I navigate to my SQL Developer Web (SDW) worksheet via the Tools tab my database's OCI console.   I then create a table named "refreshclonetests" with a single row of data in it, before we proceed to clone the database. Note that I perform this action at 12:20 am UTC.     Step 2: Creating a refreshable clone from the Autonomous Database instance I now jump into my source database's list of actions to select create clone.   I select the new Refreshable Clone option. Notice the text describing it; A refreshable clone must be refreshed every 7 days or less, else it falls too far out of sync from the source and can no longer be refreshed. For this example, I name my clone "refreshclone". As easy to remember as it gets.   I proceed to select the number of OCPUs for my refreshable clone; there is no storage selection necessary. Since this is a read-only clone that only brings in data from its source database, the amount of storage selected in TB is automatically the same as that of the source. There is also no Admin password option for the refreshable clone, as that is taken from the source when refreshed.   Clicking the create clone option starts provisioning the clone. Once provisioned, you can see useful clone information on the OCI console, including the source database that is attached to and the refresh point timestamp of the source to which the clone was refreshed.   Opening up SQL Developer Web once again, via the refreshable clone's OCI console, and querying the database shows the table refreshclonetests that I created in the source, with the single row of data that I inserted.   Step 3: Inserting additional data into the source database Switching back to the source database SDW worksheet, I insert and commit an additional row into the source database. We now have 2 rows in the source but only a single row in the refreshable clone. Notice the second row I inserted at 12:38 AM UTC.     Step 4: Refreshing the clone to view new data Now here's the fun bit; while in my refreshable clone I hit the "Refresh" button in the banner. This banner also displays the date and time before which we must refresh the clone (which is 7 days after the last refresh was performed), before it would lose the ability to sync with the source. Also, notice the "Disconnect Clone from Source Database" option under "More Actions" here. At any point, you may choose to disconnect your clone from its source making it a regular, standalone, read/write database. This, of course, is a one-way operation and after disconnecting you will no longer be able to refresh data into the clone. If 7 days have gone by and you can no longer refresh your clone, you may still disconnect your clone from its source.   The popup asks for a "refresh point" timestamp of the source database to which we want to refresh. This makes refreshes consistent and intelligible, as it definitively refreshes to an exact timestamp of the source. Since I inserted the second row into the source at about 12:38 AM UTC, I input 12:39 AM UTC as my refresh point and hit refresh clone.   While the clone is refreshing it goes into the "Updating" state. During a refresh, connections are not dropped and any running queries on the clone simply wait until the refresh is completed. The refresh may take several seconds to several minutes depending on how long it has been since the last refresh and the number of changes that have come in since then.   Once the refresh is completed, we can see exactly what timestamp of the source the clone has been refreshed to in the clone's information.   In the clone's SDW worksheet, we can now run a select query on the "refreshclonetests" table and instead of a single row, we now see both rows of data from the source! The data in the clone has been seamlessly updated to reflect that which is in the source.   Wrapping Up With this Refreshable Clones feature, you can now keep cloned databases updated without any tedious manual export process. As new data comes into your source database each day, it can easily be refreshed into all its connected refreshable clones with a few button clicks. For more on Refreshable Clones, refer to the complete documentation. As with everything OCI, your refreshable clone can also be refreshed via simple REST API calls (see API documentation). Better still, with the ability to call cloud REST APIs from a controlling ADB instance, you can quickly schedule automate your data refreshes to fit right into your data pipeline, without having to deploy any servers!     Like what I write? Follow me on the Twitter! 🐦

One of the most widely used features in Autonomous Database on Shared Infrastructure (ADB-S) today is the ability to clone your database, no matter how big or small, with little to no effort. This...

Autonomous

Autonomous Database Newsletter - September 03, 2020

a { color: white !important; }   September 03, 2020 Latest Newsletter For Autonomous Database on Shared Exadata Infrastructure       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Refreshable Clones SQL Developer QWeb Enhancements SQL Macros Note: some new features are only available with 19c. To get the most from Autonomous Database, upgrade to 19c as soon as possible. There is more information about how to upgrade existing 18c instances to 19c here, for ADW and here, for ATP. Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - Creating A Refreshable Clone Autonomous Database provides cloning where you can choose to create a full clone of the active instance, create a metadata clone, or create a refreshable clone. With a refreshable clone the system creates a clone that can be easily updated with changes from the source database. A refreshable clone allows the following: Maintain one or more copies of the source database for use as read-only databases. A clone database is available when needed, and when the data needs to be updated, simply refresh the clone from its source database. Share copies of a production database with multiple business units. For example, one business unit might use the source database for ongoing transactions and another business unit could at the same time use the refreshable clone database for read-only operations. Refreshable clones allow customers to spread the cost of database usage across multiple business units. They can bill the different units separately, based on their usage of one or more refreshable clones. Illustration shows a click-through demo for creating a new refreshable clone. If an Autonomous Database is the source for a refreshable clone or clones then you can view the list of refreshable clones via the Resources area, select "Refreshable Clones" as shown below. Illustration shows the OCI console view of refreshable clones associated with a specific ADB instance.   LEARN MORE   PDF: Overview of Refreshable Clones, including click-through demo, click here. DOC: About Refreshable Clones for ADW click here. DOC: About Refreshable Clones for ATP click here.     DO MORE   Visit the LiveLabs home page for a list of all available Autonomous Database labs - click here.       BACK TO TOP     (Shared Infrastructure Only) ENHANCEMENTS - Updates to SQL Developer Web UI Autonomous Database comes with a built-in SQL Developer Web. The UI for this web console has just been updated. Autonomous Database uses pre-set consumer groups/resource plans (high, medium, low, tp and tpurgent) which determine memory, CPU, and the degree of parallelism for queries and scripts. In the past SQL Developer Web connections were defaulted to the LOW consumer group. With this latest update it is now possible to select any of the available consumer groups when using the worksheet. Illustration shows SQL Developer Web UI for selecting a consumer group.   The latest UI also provides a simpler experience for managing users: create, edit, grant roles & privileges, REST Enable accounts, view accounts where the password is about to expire and unlock an account. Illustration shows SQL Developer Web UI for managing ADB users.   LEARN MORE   BLOG: Exploring the latest version of SQL Developer Web in the Oracle Autonomous Database, click here.         BACK TO TOP   (Shared Infrastructure Only) NEW - SQL Macros now in ADB What is a SQL Macro? It is a new and unique way for SQL developers to encapsulate complex processing which can then be reused within SQL statements. Unlike bespoke PL/SQL functions, SQL Macro code is transparent to SQL Optimizer – which brings big benefits since there is no overhead for context switching! There are two types of SQL Macros: TABLE expressions used in a FROM-clause SCALAR expressions used in SELECT list, WHERE/HAVING, GROUP BY/ORDER BY clauses Initially only TABLE macros will be available within Autonomous Database. Support for SCALAR macros is coming soon!   LEARN MORE PDF: Overview presentation for SQL Macros, click here VIDEO: 21st Century SQL: using SQL pattern matching and SQL macros - at the same time! click here. MOS: How To Identify the SQL Macros in Oracle Data Dictionary 19.7 Onwards (Doc ID 2678637.1). DOC: SQL_MACRO Clause click here.         BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  September 03, 2020 Latest Newsletter For Autonomous Database on Shared Exadata Infrastructure       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata...

Autonomous

Optimize the Orchestration of Your DevTest Lifecycle with the New Autonomous Database Rename Feature

One of the key advantages of Autonomous Database on Shared Exadata Infrastructure (ADB-S) is to be able to provision or clone a database with the click of a button. This is especially important in the context of agile development and testing (DevTest) in which the teams often need a fresh copy of the production environment in order to make sure the new projects, enhancements, or bug fixes solve the existing problems and don’t cause new issues in the production environment. Many ADB-S customers do this by frequently cloning their production databases for their DevTest teams because cloning an ADB-S instance requires very little effort and fits perfectly in the agile development lifecycle. In this blog post, we are going to explore how to further accelerate this cycle with the help of a new ADB-S capability that enables you to rename your existing ADB-S instance. Let’s put aside this new feature for a moment and see how several ADB-S customers used to take advantage of the cloning capability to keep their DevTest environments up-to-date until today. For the following example, let's assume I have a production database called adwsales and a test database called adwsalestest. In order to create a new clone of adwsales that shares the same name as adwsalestest, I will need to first terminate adwsalestest. Here's why... My new clone has to share the same name as the old one because I don’t want to change the connection string in my application code, and I have to terminate the old clone first because I cannot have two databases with the same name in the same region and tenancy. This has been a valid use case for many of our customers. However, creating a new clone and having access to your DevTest environment are serialized operations with no overlap in this approach. This is where the new rename capability comes into play. You probably can already tell how it can help improve this use case but to further explain, we can now start the new clone preparation with an intermediate name (e.g. adwsalesstage) in parallel while the old clone is still being used. Once the new clone is ready, we can rename it after terminating the old one. Again, the main motivation behind using the same name for every new clone is to avoid any application changes. The key difference between the two scenarios is that the rename capability allows replacing your existing DevTest database with a fresh copy of your production database thanks to a quick and simple metadata operation.If we go with the example above, when our new clone adwsalesstage is ready, we can terminate adwsalestest and use its name to rename adwsalesstage. Now it’s time to demonstrate it in a few simple steps: Download the regional wallet and have the production and test instances ready (adwsales and adwsalestest) Insert some data in the production (Now we need a new copy for DevTest!) Create a new clone of the production instance (adwsalesstage) Terminate the old test instance (adwsalestest) Rename adwsalesstage to adwsalestest Connect to adwsalestest Download the regional wallet and have the production and test instances ready (adwsales and adwsalestest) In this demonstration, I have two ADB-S instances: adwsales is my production database and adwsalestest is my test database that I cloned from adwsales. We are going to first download our regional wallet. The reason for using the regional wallet instead of the instance wallet is to avoid downloading a new wallet every time we create a new clone of the production since regional wallet serves all the instances that belong to the same tenancy in a given region. Let’s connect and run a simple query in our production and test instances: ctuzla-mac:instantclient_19_3 ctuzla$ ./sqlplus ADMIN/************@adwsales_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Aug 13 15:40:50 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Aug 13 2020 15:32:06 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from sales; COUNT(*) ---------- 19 ctuzla-mac:instantclient_19_3 ctuzla$ ./sqlplus ADMIN/************@adwsalestest_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Aug 13 16:29:57 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Aug 13 2020 15:32:06 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from sales; COUNT(*) ---------- 19 Insert some data in the production (Now we need a new copy for DevTest!) As we covered earlier, production databases may evolve rapidly and it’s important to have an up-to-date development and test environment. In order to simulate this scenario, let’s insert some data into our production database: SQL> insert into sales select * from sh.sales where rownum<60 order by time_id desc; 59 rows created. SQL> commit; Commit complete. SQL> select count(*) from sales; COUNT(*) ---------- 78 Create a new clone of the production instance (adwsalesstage) Since we want to maintain a fresh copy of our production database, let’s create a new clone called adwsalesstage, soon to be our new test database: Terminate the old test instance (adwsalestest) After verifying our new clone has been successfully created, we can terminate adwsalestest: Rename adwsalesstage to adwsalestest Now that the old clone is successfully terminated, we can go ahead and rename adwsalesstage to adwsalestest. We are doing this to avoid any connect string changes in the application code: Connect to adwsalestest Let's connect to our new test database and run the same query. Please note that the connect string hasn't changed and the query result reflects the insert operation we performed couple steps above: ctuzla-mac:instantclient_19_3 ctuzla$ ./sqlplus ADMIN/************@adwsalestest_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Aug 13 17:48:53 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Aug 13 2020 16:31:04 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from sales; COUNT(*) ---------- 78 It’s important to note that in this demonstration I didn’t even have to download a new wallet after creating my second clone since I’m using a regional wallet and the service names for the test database stay the same after the rename operation. To summarize, ADB-S offers quick and easy provisioning which is an integral part of agile development and testing. In this blog post, we have seen how the rapid provisioning combined with the new database rename capability can further improve the DevTest lifecycle. If you would like learn more about the rename feature, you can check out the documentation here.

One of the key advantages of Autonomous Database on Shared Exadata Infrastructure (ADB-S) is to be able to provision or clone a database with the click of a button. This is especially important in the...

Autonomous

Autonomous Database Newsletter - August 20 2020

a { color: white !important; } August 20, 2020 Latest Newsletter For Autonomous Database on Shared Exadata Infrastructure Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Launch of Autonomous JSON Database Rename Autonomous Database Data Safe Now Supports Private Endpoints Analytics Cloud 5.7 and ADB ADW Training Plans for LiveLabs Data Warehouse Blog Compilation Note: some new features are only available with 19c. To get the most from Autonomous Database, upgrade to 19c as soon as possible. There is more information about how to upgrade existing 18c instances to 19c here, for ADW and here, for ATP. Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here. (SharedInfrastructureOnly) NEW - Autonomous JSON Database - Autonomous Service For Application Developers Oracle Autonomous JSON Database is an Oracle Cloud service that is specialized for developing NoSQL-style applications that use JavaScript Object Notation (JSON) documents. Like other Oracle Autonomous Database services, it delivers: auto-scaling, automated patching, upgrading, and tuning. Autonomous JSON Database provides all of the same features as Autonomous Transaction Processing and, unlike MongoDB, allows developers to also store up to 20 GB of non-JSON data. There is no storage limit for JSON collections. Simplified animation showing basic steps to create an AJD instance LEARN MORE DOC: About Autonomous JSON Database, click here. DOC: Work with JSON Documents in Autonomous Database, click here. Q&A: Post questions about AJD on einstein.oracle.com, click here. O.COM: home page on oracle.com, click here. VIDEO: launch video from Developer Live event, click here. DO MORE Visit the LiveLabs home page for a list of all available Autonomous Database demo labs - click here. LAB: Loading, indexing and querying JSON Data - click here. VIDEO: Using ATP as a JSON document store - click here. NOTE - both demos use an ATP instance, but they also work with the new Autonomous JSON Database service.   BACK TO TOP (SharedInfrastructureOnly) NEW - Change Database Name For An Autonomous Database Autonomous Database offers quick and easy provisioning which is an integral part of an agile development framework. This agility can be further accelerated with the help of a new feature - database rename - which makes it significantly easier and faster to deploy dev/test instances, cloned from production, by avoiding the need to change connect strings in supporting application code (blog post about a real-world use case coming soon on blogs.oracle.com/datawarehousing). The rename pop-up form from the OCI ADB instance details page End-to-end flow of renaming an ADB instance LEARN MORE DOC: Rename Autonomous Data Warehouse Instance, click here. DOC: Rename Autonomous Transaction Processing Instance, click here.   BACK TO TOP (SharedInfrastructureOnly) NEW - Data Safe Now Supports ADBs Using Private Endpoints You can now register an Autonomous Database on Shared Exadata Infrastructure within a private virtual cloud network (VCN). For Oracle Data Safe to communicate with your database, you need to create an Oracle Data Safe private endpoint on the same VCN as your database's private endpoint. You can use any subnet; however, Oracle recommends that you use the same subnet as your database. Illustration shows two private endpoints communicate with each other, allowing Oracle Data Safe to communicate with your database. LEARN MORE DOC: Register ADB on Shared Exadata Infrastructure with Private VCN Access, click here. PPT: Oracle OpenWorld 2019 Data Safe Overview Presentation click here BRIEFING NOTE: Secure Critical Data with Oracle Data Safe click here FAQ: Data Safe FAQ click here DOC: Getting Started click here PORTAL: Data Safe internal home page click here   BACK TO TOP (SharedInfrastructureOnly) NEW - Democratize Machine Learning with Oracle Analytics Cloud 5.7 Machine learning models in Oracle Autonomous Database can now be registered in Oracle Analytics Cloud. These can then be applied to datasets and data connections through the Oracle Analytics Cloud interface. Not only does this reduce the cost of switching between environments, it opens the door for business users to tap into the expertise of an organization’s data scientists. Illustration shows registration process within Analytics Cloud 5.7 for enabling access to Oracle Machine Learning models. LEARN MORE DOC: How Can I Use Oracle Machine Learning Models in Oracle Analytics? click here. DOC: What's New in Analytics Cloud click here   BACK TO TOP (SharedInfrastructureOnly) NEW - ADW Training Plans For LiveLabs We have launched a new page on oracle.com to help you navigate all the data warehousing workshops available within LiveLabs. Select the main topic area, then either choose to start a specific workshop now or reserve a date and time to work through it. Customers won't need to sign up for anything - all accounts and access will be provided! Illustration shows new data warehousing training plans for content in Oracle LiveLabs LEARN MORE Training plan home page on o.com, click here   BACK TO TOP (SharedInfrastructureOnly) NEW - Compilation of Blogs For Modern Data Warehousing Capabilities Here is a compilation (in PDF format) of the most important blogs that cover our modern data warehousing capabilities, such as: Autonomous Data Warehouse, big data, query optimization, machine learning, spatial, graph, SQL and our Global Leaders program. Note: the document will be refreshed on a regular basis to include the very latest content. Illustration shows the cover of the document Compilation of Blogs For Modern Data Warehousing Capabilities & Beyond To download the PDF file, click here. BACK TO TOP oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP

August 20, 2020 Latest Newsletter For Autonomous Database on Shared Exadata Infrastructure Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This...

Big Data

Continuous ETL, Realtime Analytics, and Realtime Decisions in Oracle Big Data Service using GoldenGate Stream Analytics

Many thanks to Prabhu Thukkaram from the GoldenGate Streaming Analytics team for this post :). GoldenGate Stream Analytics is a Spark-based platform for building stream processing applications. One of the key differentiators is productivity or time-to-value, which is a direct result of the following Support for diverse messaging platforms – Kafka, OCI Streaming Service, JMS, Oracle AQ, MQTT, etc. Natively integrated with Oracle GoldenGate to process and analyze transaction streams from relational databases Interactive pipeline designer with live results to instantly validate your work Zero-code environment to build continuous ETL and analytics workflows Pattern library for advanced data transformation and real-time analytics Extensive support for processing geospatial data Secured connectivity to diverse data sources and sinks Built-in support for real-time visualizations and dashboards Automatic application state management Automatic configuration of pipelines for high availability and reliability Automatic configuration of pipelines for lower latency and higher throughput Automatic log management of pipelines for better disk space utilization Built in cache (choice of Coherence and Apache Ignite) for better pipeline performance and sharing of data across pipelines.  Caching is seamlessly integrated and no need to run additional cache clusters Continuously refresh your DataMart, Datawarehouse, and Data Lake (ADW, OCI Object Store, HDFS/Hive, Oracle NoSQL, Mongo DB, AWS S3, Azure Data Lake, Google Big Query, Redshift, etc.). Please note some of this will be arriving in 19.1.0.0.4 which is the next version. Alerts via Email, PagerDuty, Slack, etc., using OCI Notification Service This video discusses industry use cases and demonstrates the ease with which data pipelines can be built and deployed using GoldenGate Stream Analytics. If you are aware of the value proposition and interested in deploying GGSA pipelines to Big Data service, then please continue reading.   The Marketplace instance of GoldenGate Stream Analytics embeds Apache Spark and Apache Kafka so data pipelines can be built and tested without other dependencies. While this is good for developers to quickly build and test pipelines, it is not ideal for enterprise deployments and production. For production we will need a runtime platform that can scale with increasing workloads, is highly available, and one that can be secured with highest standards. This blog describes how to run GGSA pipelines in the Big Data Service runtime environment.   Topology In this example, the marketplace instance of GoldenGate Stream Analytics is created in the same regional subnet as the BDS cluster.  A marketplace instance of Stream Analytics can be provisioned from here.  You can also refer to the GGSA Marketplace user guide. Big Data Service can be provisioned from here.  Topology is shown in the diagram below and the GGSA instance is accessed using its public IP address. Please note your GoldenGate Stream Analytics instance will also act as a Bastion Host to your BDS cluster. Once you ssh to GGSA instance, you will also be able to ssh to all your BDS nodes. To do this you will need to copy your ssh private key to GGSA instance. You will also be able to access your Cloudera Manager using the same public IP of GGSA instance by following steps below. The default security list for the Regional Subnet must allow bidirectional traffic to edge/OSA node so create a stateful rule for destination port 443. Also create a similar Ingress rule for port 7183 to access the BDS Cloudera Manager via the OSA edge node.  An example Ingress rule is shown below. SSH to GGSA box and run the following port forward commands so you can access the Cloudera Manager via the GGSA instance. sudo firewall-cmd --add-forward-port=port=7183:proto=tcp:toaddr=<IP address of Utility Node running the Cloudera Manager console> sudo firewall-cmd --runtime-to-permanent sudo sysctl net.ipv4.ip_forward=1 sudo  iptables -t nat -A PREROUTING -p tcp --dport 7183 -j DNAT --to-destination  <IP address of Utility Node running the Cloudera Manager console>:7183 sudo iptables -t nat -A POSTROUTING -j MASQUERADE You should now be able to access the Cloudera Manager using the URL https://<Public IP of GGSA>:7183 Prerequisites Retrieve IP addresses of BDS cluster nodes from OCI console as shown in screenshot below. Alternatively, you can get the FQDN for BDS nodes from the Cloudera Manager as shown in the next screenshot. Patch spark-osa.jar in $OSA_HOME/osa-base/resources/modules/spark-osa.jar so dfs.client.use.datanode.hostname is set to true. Please make a copy of spark-osa.jar before replacing with the patch. This will be automatic in the next OS›A version 19.1.0.0.4. None of this workaround will be necessary.  For now you can obtain the patch from here and here are the steps to patch ssh to your OSA instance and run "wget https://objectstorage.us-phoenix-1.oraclecloud.com/p/cGDU0NMTEV0sF7uqgzH_94Ni54dP08mVwCAEwwtYab8/n/paasdevehcs/b/TEMP-GGSA/o/spark-osa.jar" Stop OSA by running “sudo systemctl stop osa” Copy /u01/app/osa/osa-base/resources/modules/spark-osa.jar ~/spark-osa.jar.backup Copy downloaded ~/spark-osa.jar to /u01/app/osa/osa-base/resources/modules/spark-osa.jar Start OSA by running “sudo systemctl start osa” Reconfigure YARN virtual cores using Cloudera Manager as shown below. This will allow many pipelines to run in the cluster and not bound by actual physical cores. Container Virtual CPU Cores :- This is the total virtual CPU cores available to YARN Node Manager for allocation to Containers. Please note this is not limited by physical cores and you can set this to a high number, say 32 even for VM standard 2.1. Container Virtual CPU Cores Minimum :- This is the minimum vcores that will be allocated by YARN scheduler to a Container. Please set this to 2 since CQL engine is a long-running task and will require a dedicated vcore. Container Virtual CPU Cores Maximum :- This is the maximum vcores that will be allocated by YARN scheduler to a Container. Please set this to a number higher than 2 say 4. Note: this change will require a restart of the YARN cluster from Cloudera Manager.   If using Kafka from OSA Marketplace instance, set advertised listeners to private IP of the OSA node and port 9092 in /u01/app/kafka/config/server.properties. E.g. advertised_listeners=PLAINTEXT://10.0.0.13:9092. Please restart the Kafka service after this change by running “sudo systemctl restart kafka”.  This will be automatic in the next OSA version 19.1.0.0.4. Deploying GGSA Pipelines to Non-Kerberized Dev/Test BDS cluster In GGSA system settings dialog, configure the following: Set Kafka Zookeeper Connection to Private IP of the OSA node. E.g. 10.0.0.59:2181 Set Runtime Server to Yarn Set Yarn Resource Manager URL to Private IP or Hostname of the BDS Master node (E.g. xxxxx.devintnetwork.oraclevcn.com) Set storage type to HDFS Set Path to <PrivateIP or Host Of Master><HDFS Path>. E.g. 10.x.x.x/user/oracle/ggsapipelines or xxxxx.devintnetwork.oraclevcn.com/user/oracle/ggsapipelines.  Set HA Namenode to Private IP or Hostname of the BDS Master node. E.g. xxxxx.devintnetwork.oraclevcn.com Set Yarn Master Console port to 8088 or as configured in BDS Set Hadoop authentication to Simple and leave Hadoop protection policy at authentication if available Set username to "oracle" and click Save.  A sample screenshot is shown below: Deploying GGSA Pipelines to Kerberized Production BDS Cluster In GGSA system settings dialog, configure the following: Set Kafka Zookeeper Connection to Private IP of the OSA node. E.g. 10.0.0.59:2181 Set Runtime Server to Yarn Set Yarn Resource Manager URL to Private IPs of all master nodes starting with the one running active Yarn Resource Manager. In the next version of OSA, the ordering will not be needed. E.g. xxxxx.devintnetwork.oraclevcn.com, xxxxx.devintnetwork.oraclevcn.com Set storage type to HDFS Set Path to <NameNode Nameservice><HDFS Path>. E.g. bdsggsasec-ns/user/oracle/ggsapipelines. The path will automatically be created if it does not exist but the Hadoop User must have write permissions. NameNode Nameservice can be obtained from HDFS configuration as shown in the screenshot below. Set HA Namenode to Private IPs or hostnames of all master nodes (comma separated) starting with the one running active NameNode server. In the next version of OSA, the ordering will not be needed. E.g. xxxxx.devintnetwork.oraclevcn.com, xxxxx.devintnetwork.oraclevcn.com Set Yarn Master Console port to 8088 or as configured in BDS Set Hadoop authentication to Kerberos Set Kerberos Realm to BDACLOUDSERVICE.ORACLE.COM Set Kerberos KDC to private IP or hostname of one of the BDS master nodes. E.g. xxxxx.devintnetwork.oraclevcn.com Set principal to bds@BDACLOUDSERVICE.ORACLE.COM.  Please see this documentation to create a Kerberos principal (e.g. bds) and add it to hadoop admin group, starting with step "Connect to Cluster's First Master Node" and  through the step "Update HDFS Supergroup". Note, you can hop/ssh to the master node using your GGSA node as the Bastion. You will need your ssh private key to be available on GGSA node though. Also, do not forget to restart your BDS cluster as instructed in the documentation. [opc@bdsggsa ~]$ ssh -i id_rsa_prabhu opc@xxxxx.devintnetwork.oraclevcn.com Make sure the newly created principal is added to Kerberos keytab file on the master node as shown  bdsmn0 # sudo kadmin.local kadmin.local: ktadd -k /etc/krb5.keytab bds@BDACLOUDSERVICE.ORACLE.COM Fetch the keytab file using sftp, etc and set Keytab field in system settings by uploading the same Set Yarn Resource Manager principal which should be in the format “yarn/<FQDN of BDS MasterNode running Active Resource Manager>@KerberosRealm”  E.g. yarn/xxxxx.devintnetwork.oraclevcn.com@BDACLOUDSERVICE.ORACLE.COM Sample screenshot shown below: Running Samples on BDS To import a sample, click “import” on any of the six samples in screenshot below and switch to Catalog or just click on the circle.  To run the sample on BDS change the hostname in Kafka connection of the sample pipeline from “localhost” to the private IP address (run hostname -i to get IP address) of the GGSA instance node. After this change you can just open the pipeline or you can publish the pipeline and then open the associated dashboard. Please see screenshots below.

Many thanks to Prabhu Thukkaram from the GoldenGate Streaming Analytics team for this post :). GoldenGate Stream Analytics is a Spark-based platform for building stream processing applications. One of...

Autonomous

How To Manually Upgrade Your Autonomous Database from 18c to 19c

In a recent newsletter (April 08, 2020) I mentioned that Oracle Database 19c is now available on Autonomous Database for all regions. The default version for all new databases is 19c and going forward, most new Autonomous Database features will only be available for databases using Oracle Database 19c. Around this time, you should have noticed some new messages appearing on your ADW console if your ADW instance was using Database version 18c. At the top of screen a new information banner appeared: and next to the Database version information there was another information message: This was to let everyone know that it's possible to manually upgrade to Database 19c right now. Of course, we will be automatically upgrading instances during September and email notifications from the Cloud Ops team will be sent shortly that will outline the plans for upgrading instances. But as my recent newsletter pointed out - most new features are only available if you are using 19c. Therefore, to get the most from your Autonomous Database it really does make sense to manually upgrade to 19c now and not wait for the automatic upgrade in September. For example, if you are using the built-in APEX environment, then you definitely want to be using the latest version 20.1 which has a lot of great new features (see July 15 newsletter) BUT your ADB instance has to be 19c. So, how do you manually upgrade your ADB instance from 18c to 19c? Fortunately it only takes a few mouse clicks. Here is the general flow: 1) Goto your console page for your ADW instance. You will see the upgrade announcement next to the database version information  and the banner at the top of the page:   2) Click on either of the upgrade links to start the upgrade process: 3) This will pop a form over your console page. It's similar to the other pop-up forms so to confirm you really do want to start the upgrade you need to enter the name of your ADB instance in the field shown below:   4) Click the blue "Upgrade" button to start the upgrade process:   That's it! Honestly, this is easiest upgrade you will ever do. It's simpler than upgrading your iPhone! Whilst the upgrade process is running your console page will look like this: ..and you will notice the status box is now orange and the status message says "UPGRADING". There is an additional banner message that says "The database remains available during the upgrade". No other database vendor is able to do this, everyone else forces their database to go offline during an upgrade which means all connections are terminated and all on-going workloads are lost. Can you track the upgrade process? Yes. The Work Request tab provides real-time status information for the upgrade process   What happens when the upgrade process is complete? As soon as the upgrade process has finished the Work Request screen will show the big green SUCCEEDED message:   That's it, it's all done and you are now running Database 19c. If you check the console page again you will notice that the banner message has now gone and the Database Version now says 19c. Conclusion Now you should be in a position to manually upgrade your own 18c ADB instances to 19c. Then you will be able to take advantage of all the new features we have recently rolled out such as the high availability feature, Autonomous Data Guard. The whole end-to-end migration process is shown here:   If you have any questions then please post them on our customer forum which is here.  

In a recent newsletter (April 08, 2020) I mentioned that Oracle Database 19c is now available on Autonomous Database for all regions. The default version for all new databases is 19c and going...

Autonomous

How to Access Non-Oracle Databases from Autonomous Database using Oracle Database Gateway

Being able to create database links from Autonomous Database on Shared Infrastructure (ADB-S) to other ADB-S instances or Oracle databases has been one of the most sought-after features that we introduced. ADB-S now supports creating database links to Oracle Database Gateways in order to access non-Oracle databases. In this blog post, we are going to explore how to access a Microsoft SQL Server database from an ADB-S instance using a database link that we will create to a gateway. Before we jump on the detailed steps, let’s do a quick recap of database links in ADB-S. As you may already know, ADB-S requires the target database to be configured with TCP/IP with SSL (TCPS) authentication for outgoing database links since it uses TCPS authentication by default. What this means is that the gateway we want to connect needs to be configured with TCPS authentication as well. We already covered how to configure an Oracle database, such as DBCS, with TCPS authentication in one of my earlier blog posts. Good news is that configuring Oracle Database Gateway with TCPS is very similar and requires couple minor additional steps. Let’s start! The non-Oracle database that I will be using for this demonstration is Microsoft SQL Server 2019 Express running on Windows Server 2019. For simplicity, I chose to install Oracle Database Gateway 19c on the same Windows VM. The installation of the gateway using the Oracle Universal Installer is extremely quick and simple. All you need to provide is a Windows user that has no admin privileges (gateway owner), which non-Oracle database you are planning to connect to and the details of that database such as host name, port number, database name etc. The installer also partially configures a listener for the gateway during the installation and it’s possible to choose TCPS authentication in the UI (I will explain what I mean by 'partially' in a moment). Here’s how my listener.ora looked like after the installation (note the TCPS protocol in the listener description): # listener.ora Network Configuration File: C:\app\GW\product\19.0.0\tghome_1\NETWORK\ADMIN\listener.ora # Generated by Oracle configuration tools. LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = ctuzlaWindows)(PORT = 1522)) ) ) SID_LIST_LISTENER = (SID_LIST= (SID_DESC= (SID_NAME=dg4msql) (GLOBAL_DBNAME=dg4msql_svc) (ORACLE_HOME=C:\app\GW\product\19.0.0\tghome_1) (PROGRAM=dg4msql) ) ) In the listener.ora above, LISTENER was automatically created by the installer; however, I had to manually add the SID_LIST_LISTENER variable. It’s important to note that SID_NAME is the SID of the gateway and GLOBAL_DBNAME is the service name of the gateway. The value of the GLOBAL_DBNAME variable can be anything you specify and this is the value that we will be using as the target database service name when we create our database link from our Autonomous Database. Additionally, as you can tell from the ORACLE_HOME path, GW is our Windows user who is also the gateway owner. Even though our listener already has the TCPS endpoint, we still need to perform couple additional steps so that our ADB-S instance can successfully communicate with the gateway. Here are the steps needed to complete the TCPS configuration in our gateway: Create wallets with self signed certificates for server and client Exchange certificates between server and client wallets (Export/import certificates) Add wallet location in the server network files Create wallets with self signed certificates for server and client As part of enabling TCPS authentication, we need to create individual wallets for the server and the client. Each of these wallets has to have their own certificates that they will exchange with one another. For the sake of this example, I will be using a self signed certificate. The client wallet and certificate can be created in the client side; however, I'll be creating my client wallet and certificate in the server and moving them to Object Store later on. See Configuring Secure Sockets Layer Authentication for more information. Directories to be used for the wallets and certificates C:\Users\GW\Desktop\server\wallet C:\Users\GW\Desktop\client\wallet C:\Users\GW\Desktop\tmp Create a server wallet with the GW user C:\Users\GW\Desktop\server\wallet>orapki wallet create -wallet ./ -pwd ************ -auto_login Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a server certificate with the GW user C:\Users\GW\Desktop\server\wallet>orapki wallet add -wallet ./ -pwd ************ -dn "CN=windows" -keysize 1024 -self_signed -validity 3650 -sign_alg sha256 Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a client wallet with the GW user C:\Users\GW\Desktop\client\wallet>orapki wallet create -wallet ./ -pwd ************ -auto_login Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a client certificate with the GW user C:\Users\GW\Desktop\client\wallet>orapki wallet add -wallet ./ -pwd ************ -dn "CN=ctuzla" -keysize 1024 -self_signed -validity 3650 -sign_alg sha256 Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Exchange certificates between server and client wallets (Export/import certificates) Export the server certificate with the GW user C:\Users\GW\Desktop\server\wallet>orapki wallet export -wallet ./ -pwd ************ -dn "CN=windows" -cert C:\Users\GW\Desktop\tmp\server.crt Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Export the client certificate with the GW user C:\Users\GW\Desktop\client\wallet>orapki wallet export -wallet ./ -pwd ************ -dn "CN=ctuzla" -cert C:\Users\GW\Desktop\tmp\client.crt Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Import the client certificate into the server wallet with the GW user C:\Users\GW\Desktop\server\wallet>orapki wallet add -wallet ./ -pwd ************ -trusted_cert -cert C:\Users\GW\Desktop\tmp\client.crt Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Import the server certificate into the client wallet with the GW user C:\Users\GW\Desktop\client\wallet>orapki wallet add -wallet ./ -pwd ************ -trusted_cert -cert C:\Users\GW\Desktop\tmp\server.crt Oracle PKI Tool Release 19.0.0.0.0 - Production Version 19.3.0.0.0 Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Add wallet location in the server network files We now need to modify the server network files so that they point to the server wallet location and they are ready to use the TCPS protocol. Here's how those files look in my case: Server-side $ORACLE_HOME/network/admin/sqlnet.ora # sqlnet.ora Network Configuration File: C:\app\GW\product\19.0.0\tghome_1\NETWORK\ADMIN\sqlnet.ora # Generated by Oracle configuration tools. SSL_SERVER_DN_MATCH= (ON) WALLET_LOCATION = (SOURCE = (METHOD = File) (METHOD_DATA = (DIRECTORY = C:\Users\GW\Desktop\server\wallet) ) ) Server-side $ORACLE_HOME/network/admin/listener.ora # listener.ora Network Configuration File: C:\app\GW\product\19.0.0\tghome_1\NETWORK\ADMIN\listener.ora # Generated by Oracle configuration tools. WALLET_LOCATION = (SOURCE = (METHOD = File) (METHOD_DATA = (DIRECTORY = C:\Users\GW\Desktop\server\wallet) ) ) LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = ctuzlaWindows)(PORT = 1522)) ) ) SID_LIST_LISTENER = (SID_LIST= (SID_DESC= (SID_NAME=dg4msql) (GLOBAL_DBNAME=dg4msql_svc) (ORACLE_HOME=C:\app\GW\product\19.0.0\tghome_1) (PROGRAM=dg4msql) ) ) Notes About the Target Environment If your target database is hosted on a Windows VM, make sure the firewall is turned off or configured in a way to not block the incoming traffic. As you can see in my listener description above, my listener is configured on port 1522. Since my VM is hosted in OCI, I needed to add an ingress rule for that port that in the security list of my virtual cloud network (VCN). Without this step, I wouldn’t be able to reach the listener from my Autonomous Database. Here’s how the HS parameters in my gateway agent init file looks like: # This is a customized agent init file that contains the HS parameters # that are needed for the Database Gateway for Microsoft SQL Server # # HS init parameters # HS_FDS_CONNECT_INFO=ctuzlaWindows:8000//mssfinance HS_FDS_TRACE_LEVEL=ODBC HS_FDS_RECOVERY_ACCOUNT=RECOVER HS_FDS_RECOVERY_PWD=RECOVER Please note that my SQL Server database (mssfinance) is configured to have TCP enabled on port 8000 as shown above. Confirm the listener status: C:\Users\GW>lsnrctl status LSNRCTL for 64-bit Windows: Version 19.0.0.0.0 - Production on 24-JUL-2020 09:27:30 Copyright (c) 1991, 2019, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=ctuzlaWindows)(PORT=1522))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for 64-bit Windows: Version 19.0.0.0.0 - Production Start Date 17-JUL-2020 20:35:35 Uptime 6 days 12 hr. 51 min. 59 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File C:\app\GW\product\19.0.0\tghome_1\network\admin\listener.ora Listener Log File C:\app\GW\diag\tnslsnr\ctuzlaWindows\listener\alert\log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=ctuzlaWindows)(PORT=1522))) Services Summary... Service "dg4msql_svc" has 1 instance(s). Instance "dg4msql", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully Create a Database Link from ADB-S to the Gateway We now have a working TCPS authentication in the gateway. Here are the steps from the documentation that we will follow to create a database link from ADB-S to the gateway: Copy the client wallet (cwallet.sso) that we created in C:\Users\GW\Desktop\client\wallet to Object Store. Create credentials to access your Object Store where you store the cwallet.sso. See CREATE_CREDENTIAL Procedure for details. Create a directory to store the target database wallet: SQL> create directory wallet_dir as 'walletdir'; Directory WALLET_DIR created. Upload the wallet to the wallet_dir directory on ADB-S using DBMS_CLOUD.GET_OBJECT: SQL> BEGIN DBMS_CLOUD.GET_OBJECT( credential_name => 'OBJ_STORE_CRED', object_uri => 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/adbstraining/b/target-wallet/o/cwallet.sso', directory_name => 'WALLET_DIR'); END; / PL/SQL procedure successfully completed. On ADB-S, create credentials to access the target database. The username and password you specify with DBMS_CLOUD.CREATE_CREDENTIAL are the credentials for the target database that you use to create the database link: BEGIN DBMS_CLOUD.CREATE_CREDENTIAL( credential_name => 'SSFINANCE_LINK_CRED', username => 'mssadmin', password => '************'); END; / PL/SQL procedure successfully completed. Create the database link to the target database using DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK: BEGIN DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK( db_link_name => 'FINANCELINK', hostname => '149.343.90.113', port => '1522', service_name => 'dg4msql_svc', ssl_server_cert_dn => 'CN=windows', credential_name => 'SSFINANCE_LINK_CRED', directory_name => 'WALLET_DIR', gateway_link => TRUE); END; / PL/SQL procedure successfully completed. Note that DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK procedure has a new parameter called gateway_link that needs to be set to TRUE when creating a database link to an Oracle Database Gateway. Use the database link you created to access the data on the target database: select count(*) from CUSTOMERS@FINANCELINK; COUNT(*) -------- 4 This is all you need in order to access a non-Oracle database from your Autonomous Database! In this post, we have gone through gateway installation, TCPS configuration in gateways, target environment tips and database link creation in order to access a SQL Server database. The same steps apply to other non-Oracle databases that are supported by Oracle Database Gateway as well.

Being able to create database links from Autonomous Database on Shared Infrastructure (ADB-S) to other ADB-S instances or Oracle databases has been one of the most sought-after features that we...

Autonomous

Using OCI Resource Manage to deploy a complete end-to-end data warehouse

A while back, I announced in the Autonomous Database Newsletter (June 24 edition) that we now had a series of reference architecture patterns for Autonomous Data Warehouse. Currently, the following patterns are available (note: more patterns are being built and will be added shortly): Departmental data warehousing - an EBS integration example Departmental data warehousing/data marts - consolidate spreadsheets Enterprise data warehousing - an integrated data lake example Set up a data science environment that uses Oracle Machine Learning Each reference architecture pattern comes with the following: best practices framework; a set of recommendations which are a starting point for implementing the pattern and a terraform script for deploying the end-to-end set of cloud services on OCI. That was great news for everyone because you now had a very precise system architecture plan for each data warehouse use case that could be expanded to incorporate additional use cases as your data warehouse project evolved over time. These patterns were the result of a some amazing work by our Enterprise Architecture, OCI andADB PM teams. There are a lot more patterns to come and I will let you know as new ones get added to the list. I have used a couple of the patterns myself and, if I am honest, while the Terraform scripts are great it was hard work to setup all the required keys etc and get everything in to place so I could run one of the pre-built Terraform scripts. But then again, Terraform is a completely new area for me! Then this week our OCI team announced they have added one of our data warehouse patterns to the list of pre-built stacks within OCI Resource Manager. Here's a screen grab of the list of built-in stacks that are ready for deploying:   Right now, we have the "Departmental Data Warehouse" pattern available as a stack and this deploys an instance of Autonomous Database and an instance on Oracle Analytics Cloud for you. So how do you get started? Here is a walk-through of the main screens: Step 1) - I have logged into my Always Free cloud account and arrived at the main OCI home page: Step 2 - I have opened the main menu tree and scrolled down to Resource Manager and then selected "Stacks" Step 3 - As this is the first stack I have built my home page is currently empty. Obviously we click on the blue Create Stack button. Step 4 - I can build my own stack deployment process or I can select to use a pre-built stack and this is what we need to use in this case...   Step 5 - this pops a form which lists all the available pre-built stacks as I showed above. In this case I want to build a departmental data warehouse so let's select that box Step 6 - now I can set the name and description for my deployment "stack" along with the compartment I want to use in my deployment (if you are not sure what a compartment is or does then spend a few minutes looking at this page in the documentation). Once you are happy with the information on this page, click the Next button in the bottom left corner. Step 7 - this brings us to the form which is going to capture the information about our ADW and OAC instances. In the screen below you can see that I have set the number of CPUs for my ADW instance as 1, I have set the password for my ADMIN user, I have changed the name and display name for my new database, I have elected to build a 19c autonomous data warehouse and I have enabled auto-scaling. For storage, I am going to start with 1 TB. Just point out....both the number of OCPUs and the amount of storage and can be scaled independently and with zero downtime.  Step 8 - scrolling down....I can set my license type. We allow you to bring your on-premise licenses to the cloud or you can simply buy additional licenses as required. With this departmental warehouse project I know I am going to include personally identifiable information about some of our employees so I need to keep that data totally safe and secure. Therefore, I am going to register my ADW instance with our Data Safe service (which is free to use with Autonomous Database). ...and to further protect my ADW instance this pre-built stack automatically uses Access Control Lists (ACLs). If you want more information about to use ACLs then click here. This process within Resource Manager will automatically configure the ACLs so that OAC can connect to my ADW instance so that's all taken care of but I need to allow my desktop computer to access my ADW instance as well. Therefore, I have to enter the public IP address for my desktop computer. If you don't include this information then you will not be able to use any of the desktop tools or built-in tools that come with ADW. Step 9 -  Setup the details for my OAC instance. Again, I have opted for 1 OCPU and I am going to take the ENTERPRISE_ANALYTICS deployment option since this gives me access to all the features I need. If you use the pulldown menu you will there are two options: 1)Self-service analytics deploys an instance with data visualization, and 2) Enterprise analytics deploys an instance with enterprise modeling, reporting, and data visualization.   Step 10 - and now we are at the end of the process and ready to deploy our end-to-end departmental data warehouse. The review page gives us a chance to make sure everything is correct before we hit the Create button.  The Jobs tab allows you to monitor progress of your deployment and at the end of this process you should have a new ADW instance and a new OAC instance within your tenancy.   Congratulations You should now have your new departmental data warehouse ready for use, What's Next Next you can start loading data into your ADW instance. Take a look at the Chapter 3 in the ADW documentation which covers loading data with Autonomous Data Warehouse - https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/load-data.html#GUID-1351807C-E3F7-4C6D-AF83-2AEEADE2F83E Then you can launch the Oracle Analytics home page in your browser, connect to your ADW instance and start building reports and dashboards. The OAC product management team has put together a great series of labs that will get you started with OAC and ADW, see here: https://www.oracle.com/uk/business-analytics/data-visualization/tutorials.html Hope this was useful and I will let you know as we add more patterns to the solution library and more patterns into the pre-built stacks within OCI Resource Manager.                                   

A while back, I announced in the Autonomous Database Newsletter (June 24 edition) that we now had a series of reference architecture patterns for Autonomous Data Warehouse. Currently, the following...

Autonomous

Autonomous Database Newsletter - July 22 2020

Autonomous Database on Shared Exadata Infrastructure July 22, 2020   Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Using Autonomous Data Guard MDW Patterns in OCI Resource Manager Note: some new features are only available with 19c. To get the most from Autonomous Database, upgrade to 19c as soon as possible. There is more information about how to upgrade existing 18c instances to 19c here, for ADW and here, for ATP. Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - Creating a Standby Database with ADG When you enable Autonomous Data Guard on ADB, the system creates a standby database that stays "connected" to the primary database and automatically refreshes its data from the primary database. This allows the following, depending on the state of the primary database: If your primary database goes down, Autonomous Data Guard converts the standby database to the primary database with minimal interruption. After failover completes, Autonomous Data Guard automatically creates a new standby database. You can perform a switchover operation, where the primary database becomes the standby database, and the standby database becomes the primary database. Note: Autonomous Database does not provide access to the standby database. ×   // Get the modal var modal = document.getElementById("myModal"); // Get the image and insert it inside the modal - use its "alt" text as a caption var img = document.getElementById("myImg"); var modalImg = document.getElementById("img01"); var captionText = document.getElementById("caption"); img.onclick = function(){ modal.style.display = "block"; modalImg.src = this.src; captionText.innerHTML = this.alt; } // Get the element that closes the modal var span = document.getElementsByClassName("close")[0]; // When the user clicks on (x), close the modal span.onclick = function() { modal.style.display = "none"; } Simplified animation showing basic steps to create an ADG standby instance for ADW   LEARN MORE   DOC: Key features of Autonomous Data Guard for ADW, click here. DOC: Autonomous Data Guard notes for ADW, click here. DOC: Key features of Autonomous Data Guard for ATP, click here. DOC: Autonomous Data Guard notes for ATP, click here.       BACK TO TOP   (Shared Infrastructure Only) NEW - Using OCI Resource Manager to Deploy Data Warehouse Reference Architecture Patterns The list of pre-built Stacks within OCI Resource Manager has been expanded to include some of the new reference architecture patterns. Currently, the Departmental Data Warehouse pattern is available as a pre-built stack and the other Modern Data Warehouse patterns will be available shortly. ×   // Get the modal var modal = document.getElementById("myModal02"); // Get the image and insert it inside the modal - use its "alt" text as a caption var img = document.getElementById("myImg02"); var modalImg = document.getElementById("img02"); var captionText = document.getElementById("caption02"); img.onclick = function(){ modal.style.display = "block"; modalImg.src = this.src; captionText.innerHTML = this.alt; } // Get the element that closes the modal var span = document.getElementsByClassName("close02")[0]; // When the user clicks on (x), close the modal span.onclick = function() { modal.style.display = "none"; } Simplified animation showing basic steps for using Resource Manager pre-built stacks to deploy a departmental data warehouse.   LEARN MORE DOC: Architecture Center, click here. DOC: Pattern for Departmental Data Warehouse, click here. DOC: Pattern for Enterprise Data Warehouse with HCM data enrichment, click here. DOC: Pattern for Departmental Data Warehouse with EBS integration click here. DOC: Pattern for Enterprise Data Warehouse with an integrated data lake click here. DOC: Pattern for data science environment using Oracle Machine Learning, click here.       BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

Autonomous Databaseon Shared Exadata InfrastructureJuly 22, 2020   Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Announcing Autonomous Data Guard!

Just over a week ago, you may have heard Larry mention Autonomous Data Guard at the live-streamed Oracle Live event. Today, I am happy to announce this much-awaited disaster recovery (DR) feature in Autonomous Database on Shared Infrastructure (ADB-S). Autonomous Data Guard (ADG) gives users the ability to enable a standby database for each ADB instance in a just one-click; As always in the Autonomous world - It is completely automated! Note that this is an advanced feature available only in 19c and above ADB instances and not available in the Always-Free tier. If you are still running database version 18c, consider upgrading to 19c in one-click as well. While ADB already runs on highly available Exadata infrastructure, this feature further protects your business' data against unforeseen disaster scenarios (think earthquakes, fires, floods, major network outages etc.)  by automatically failing over to a standby database when the primary database goes down. Currently, ADB-S supports ADG within the same region (physically separated across Availability Domains) and aims to support ADG across regions later this year. Before we jump right into using this feature, let's brush up on some (simplified) disaster recovery terminology as it relates to ADG: Peer Databases: Two or more databases that are peered (linked and replicated) between each other. They consist of a Primary database and Standby (copy of the primary) databases. Primary or Source Database: The main database that is actively being used to read from and write to by a user or application. Standby Database: A replica of the primary database which is constantly and passively refreshing (ie. replicating) data from the primary. This standby database is used in case of failure of the primary. In the case of ADG, we keep this standby on a different physical Exadata machine (in a different Availability Domain in regions that have more than one) for the highest level of protection. Recovery Point Objective (RPO): An organization's tolerance for data loss, after which business operations start to get severely impacted, usually expressed in minutes. We want this to be as low as possible. Recovery Time Object (RTO): An organization's tolerance for the unavailability (or downtime) of a service after which business operations start to get severely impacted, usually expressed in minutes. We want this to be as low as possible. Armed with the above knowledge, let's dive right into using Autonomous Data Guard. You already know how to navigate to your Autonomous Database console in OCI, by clicking on your database name in your list of database instances. As before, you should be using a 19c version database. Like magic, you now see Autonomous Data Guard as an option on your console. Click the "Enable" button on your console, followed by clicking the "Enable Autonomous Data Guard" confirmation button. (Okay, I lied... It's 2 clicks, not 1). You're done, that's all you need to do to protect your database! You will now see your peered standby database in a Provisioning state until it becomes Available; depending on the size of your primary database this may take several minutes. Note that when you switch your standby to make it your primary as described below, you will not need a new wallet - Your existing wallet will continue to work seamlessly! Everything that follows describes features you will use either in a disaster type scenario, or to test your applications / mid-tiers against your Autonomous Database with ADG,   Failover - When your Primary database goes down, you are automatically protected by your Standby database If a disaster were to occur and your primary database is brought down, you can "Failover" to your standby. A failover is a role change, switching from the primary database to the standby database when the primary is down and Unavailable, while the standby is Available. An Automatic Failover is automatically triggered (no user action needed) by the Autonomous Database when a user is unable to connect to their primary database for a few minutes. Since this is an automated action, we err on the side of caution and allow auto-failovers to succeed only when we can guarantee no data loss will occur.  For automatic failover, RTO is 2 minutes and RPO is 0 minutes. In the rare case when your primary is down and Automatic Failover was unsuccessful, the Manual Failover button may be triggered by you, the user. During a manual failover, the system automatically recovers as much as data as possible minimizing any potential data loss; there may be a few seconds or minutes of data loss. You would usually only perform a Manual Failover in a true disaster scenario, accepting the few minutes of potential data loss to ensure getting your database back online as soon as possible. For manual failover, the RTO is 2 minutes and RPO is 5 minutes.   After a failover, a new standby for your primary will automatically be provisioned.       Switchover - Test your applications with Autonomous Data Guard when both databases are healthy When your standby is provisioned, you will see a "Switchover" option on the console while your primary and standby databases are both healthy (ie. in the Available or Stopped states). Clicking the Switchover button performs a role change switching from the primary database to the standby database, which may take a couple of minutes, with no data loss guaranteed. You would usually perform a Switchover to test your applications against this role change behaviour of the primary.     We talked above about the buttons on the UI console. Of course, as with the rest of Oracle's cloud platform, there are Autonomous Database APIs for all actions including Switchover and Failover that can be called at any time. You may also subscribe to Events to be notified of your standby's operations.   Wrapping Up While we ideated over Autonomous Data Guard, our primary focus has been that it should just work, with nothing for a user to do. My hope is that every user with a production database out there takes advantage of this simplicity and is able to further minimize their business' operational downtime and data loss. For all the details about Autonomous Data Guard features please refer to the official documentation and for updated pricing refer to the service description. Stay tuned for Part 2 of this post later this year, once we are ready to announce Autonomous Data Guard across regions!

Just over a week ago, you may have heard Larry mention Autonomous Data Guard at the live-streamed Oracle Live event. Today, I am happy to announce this much-awaited disaster recovery (DR) feature in...

Autonomous

Autonomous Database Newsletter - July 15 2020

  Autonomous Database on Shared Exadata Infrastructure July 15, 2020   Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Connecting to non-Oracle Databases Modify Network Configuration Timezone Support Modifying User Profiles Updated Quick Start Guides APEX 20.1 is coming to ADB Global Leaders Update Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - Connecting to Non-Oracle Databases Autonomous Database now supports using connections that are enabled via Oracle Database Gateway. Gateways are designed for accessing a specific non-Oracle Database such as DB2, Teradata, SQL Server etc. Oracle Database Gateway provide queries and applications with access to data anywhere in a distributed database system without knowing either the location of the data or how it is stored. This removes the need to customize applications to access data from non-Oracle systems.   LEARN MORE   DOC: Create Database Links to an Oracle Database Gateway to Access Non-Oracle Databases (ADW), click here. DOC: Create Database Links to an Oracle Database Gateway to Access Non-Oracle Databases (ATP), click here. DOC: Introduction to Heterogeneous Connectivity, click here. DOC: Install guides for Database Gateways, click here.       BACK TO TOP   (Shared Infrastructure Only) NEW - Modify Network Settings for ADB Instances Autonomous Database now supports switching network access type. This feature enables customers to perform an in-place switch from public endpoint to private endpoint (and vice versa). The feature will reduce customer effort required for switching between public and private endpoints and make it easier for customers to adopt private endpoint.     LEARN MORE   DOC: Networking Management Tasks in OCI doc, click here. DOC: Change from Private to Public Endpoints with Autonomous Database (ADW), click here. DOC: Change from Private to Public Endpoints with Autonomous Database (ATP), click here.       DO MORE   VIDEO: Autonomous Database Shared - Switching from public to private endpoint, click here. BLOG: Getting Up to Speed on Using Private Endpoints for Autonomous Database with Shared Exadata Infrastructure, click here. DOC: Reference architecture pattern: Deploy an autonomous database with a private endpoint click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - Timezone Files Automatically Updated For time zone support, Oracle Database uses time zone files that store the list of all time zones. The time zone files for Autonomous Database are periodically updated to reflect the latest time zone specific changes. An Autonomous Database instance will automatically pick up updated time zone files depending on the lifecycle state of the instance: - Stopped: At the next start operation the update is automatically applied. - Available: After a restart the update is automatically applied. When a load or import operation results in a timezone related error, restart the ADB instance and try again.   LEARN MORE   DOC: Manage Time Zone Data Updates on ADW, click here. DOC: Manage Time Zone Data Updates on ATP, click here. EINSTEIN: How Does Time Zone Upgrade Work in Autonomous Database on Shared Exadata Infrastructure (ADB-S), click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - Create/Modify User Profile and Set Password Rules Autonomous Database now supports the ability to create/alter user profiles and import existing user profiles from another environment with Oracle Data Pump Import. Cloud Admins can now manage the rules for password complexity on Autonomous Database. Administrators can can create a Password Verify Function (PVF) and associate the PVF with a profile to manage the rules for password complexity.   LEARN MORE   DOC: Manage User Profiles with ADW, click here. DOC: Manage password complexity rules on ADW, click here. DOC: Manage User Profiles with ATP, click here. DOC: Manage password complexity rules on ATP, click here.     BACK TO TOP   (Shared Infrastructure Only) UPDATE - Quick Start Guides Quickstart Tutorial 1: Autonomous Database Quickstart Learn about Autonomous Database on Shared Infrastructure and learn how to create an instance in just a few clicks. Then load data into your database, query it, and visualize it.   Quickstart Tutorial 2: Analyzing your data with Autonomous Database Connect using secure wallets and monitor your Autonomous Database instances, use Oracle Analytics to visualize data and then use Oracle Machine Learning (OML) to try your hand at predictive analytics.   LEARN MORE   DOC: Getting Started Guides for ADW, click here. DOC: Getting Started Guide for ATP, click here.       BACK TO TOP   NEW - Collateral For Autonomous Database Great news - Oracle APEX 20.1 is coming to Autonomous Database! Always Free customers received the upgrade on July 8, 2020 and other customers will receive APEX 20.1 beginning July 17, 2020. To receive the APEX 20.1 upgrade, customers must be on Oracle Database 19c. For more information about how to upgrade an existing ADB instance to 19c please follow the instructions in the documentation: - click here, for ADW - click here, for ATP.   Can I Validate My APEX Applications Before Upgrading? Yes you can! There is a unique capability in APEX on Autonomous Database that enables customers to validate their applications prior to the APEX upgrade. Please see this blog post for detailed instructions.     BACK TO TOP     ORACLE GLOBAL LEADERS       EMEA Summer 2020 Event - Replay Now Available We have just held our first Oracle Global Leaders EMEA Online Conference through Zoom with 300 registrations and an average of 130 participating each day. Over 20 customers and partner shared their data management and analytics projects use-cases on six panels. We also had Mike Dietrich, Engin Şenel, Barb Lundhild, Philippe Lions, Mark Hornick and Joel Kallman provide updates and roadmaps for their products. Cetin Ozbutun delivered the Executive Keynote on our Data Management Roadmap. This is the first time we have managed to make the event material publicly available, so please share this link, click here, with your customers and prospects. (Note - some attendees content is not included: PDF docs from Caixa Bank and ADNOC (pending)).   Americas Summer 2020 Event - Sept 2 - 3 During this online event you will have the unique opportunity to hear from customers, Oracle product management and product development for Autonomous Database, big data, data warehousing, analytics and general data management strategies and implementations. Registration for this event is now open, so please share this link, click here, with your customers and prospects.   Recent Webcasts Forth Smart transforms Thai Finance with Autonomous Database and Oracle Analytics Cloud Learn how Forth Smart, a payment services provider, uses Oracle technology for easy-access, real-time data. Share this on-demand link, click here, with your customers and prospects. Sharda University future proofs student learning with Oracle Autonomous Database Find out how Sharda University uses Autonomous Database to manage data and uncover insights for student success. Share this on-demand link, click here, with your customers and prospects.     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous Databaseon Shared Exadata InfrastructureJuly 15, 2020   Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Autonomous

Autonomous Database Newsletter - June 24 2020

a { color: white !important; }   Autonomous Database on Shared Exadata Infrastructure June 24, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Support for ORC files Automatic upgrade to 19c AWR Reports in Performance Hub Changes to file transfer limits into Object Store Procedures for using Database Real Application Security Reference architecture patterns for data warehousing Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - ADB now supports ORC files ADB has expanded object storage support in two important ways: it can now query ORC files as well as complex data types found in Parquet, ORC and Avro files. The Optimized Row Columnar (ORC) file is a highly efficient way to store Hive data. It was designed to overcome limitations of the other Hive file formats. Like Parquet, ORC data is stored in a compressed columnar format and provides metadata to enable efficient reads. ORC, Parquet and Avro file types provide the ability to capture fields containing complex data types, including arrays, maps, unions and objects (or structs). These complex types can now be queried in ADB using Oracle’s extensive JSON query support. Note: support for ORC and complex data types are only available in 19c.   LEARN MORE DOC: DBMS_CLOUD Package ORC to Oracle Data Type Mapping for ADW, click here. DOC: DBMS_CLOUD Package ORC, Parquet and Avro Complex Types for ADW, click here. DOC: DBMS_CLOUD Package ORC, Parquet and Avro to Oracle Column Name Mapping for ADW, click here. DOC: DBMS_CLOUD Package ORC to Oracle Data Type Mapping for ATP, click here. DOC: DBMS_CLOUD Package ORC, Parquet and Avro Complex Types for ATP, click here. DOC: DBMS_CLOUD Package ORC, Parquet and Avro to Oracle Column Name Mapping for ATP, click here.     DO MORE   BLOG: Query ORC files and complex data types in Object Storage with Autonomous Database, click here. BLOG: Autonomous Database - Access Parquet Files in Object Stores, click here.     BACK TO TOP   (Shared Infrastructure Only) NOTE - Customer Announcement About Automatic Upgrade to 19c An email announcement about the automatic upgrade to Database 19c has been sent to ADB customers (see below). The automatic upgrade of Autonomous Databases will start in September. At the date of the scheduled upgrade, Autonomous Database instances will be automatically upgraded in-place to Oracle Database 19c. Please note that to better manage the customer experience during this upgrade process we are recommending that customers contact support by logging a service request if they have any questions about the upgrade process. To view the options for manually upgrading an exsting ADW instance to Database 19c - click here. To view the options for manually upgrading an exsting ATP instance to Database 19c - click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - Download AWR Reports From Performance Hub Automatic Workload Repository (AWR) report is a vital tool to collect historical performance statistics of an Oracle Database by DBAs and System Performance engineers. Performance Hub on the OCI console now allows customers to download an AWR report for a specific time period.   LEARN MORE   DOC: Using Performance Hub to Analyze Database Performance, click here. DOC: Active Session History Reports, click here.     DO MORE   DOC: Interpreting Results from Active Session History Reports... After generating an ASH report, you can use the following sections to review its contents to help identify possible causes of transient performance problems: Top Events Top SQL Top PL/SQL Top Sessions Activity Over Time click here.   BACK TO TOP   (Shared Infrastructure Only) UPDATE - Maximum File Size Increased For Transfer Object Store The maximum file size limit for DBMS_CLOUD.PUT_OBJECT procedure when transferring files to OCI Object Storage has been increased to 50 GB. Note - there are different size restrictions when using other object stores such as Amazon S3 and Microsoft's Azure Blob Storage. See doc links below for more details.   LEARN MORE   DOC: Details of PUT_OBJECT procedure for ADW, click here. DOC: Details of PUT_OBJECT procedure for ATP, click here.     BACK TO TOP   (Shared Infrastructure Only) UPDATE - Procedures For Using Database Real Application Security With ADB Real Application Security works the same on Autonomous Database as on an on-premise database except you need to perform some additional administrator tasks before using Real Application Security on Autonomous Database. These tasks are now detailed in the documentation.   LEARN MORE   DOC: Procedures for using Oracle Database Real Application Security with ADW, click here. DOC: Procedures for using Oracle Database Real Application Security with ATP, click here.     BACK TO TOP   (Shared Infrastructure Only) UPDATE - Reference Architecture Patterns and Solution Playbooks Now Available The Architecture Center has been updated to include a new series of data warehouse patterns: Each reference architecture pattern will come with the following: best practices framework; a set of recommendations which are a starting point for implementing the pattern and a terraform script for deploying the end-to-end set of cloud services on OCI. Currently, the following patterns are available (note: more patterns are being built and will be added shortly): Departmental data warehousing - an EBS integration example Departmental data warehousing/data marts - consolidate spreadsheets Enterprise data warehousing - an integrated data lake example Set up a data science environment that uses Oracle Machine Learning     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous Databaseon Shared Exadata InfrastructureJune 24, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Autonomous

Query ORC files and complex data types in Object Storage with Autonomous Database

Apache ORC is a columnar file type that is common to the Hadoop ecosystem.  It is similar in concept to Apache Parquet; Hortonworks Hadoop clusters often leveraged ORC – while Cloudera clusters utilized Parquet.  Like parquet, ORC is a database file designed for efficient reads. Files embed a schema, data is stored in a compressed columnar format, predicate pushdown enables data pruning – sound familiar?  See the Parquet blog post if you want a refresher :) In addition, ORC provides the ability to capture complex data types – including arrays, maps, unions and objects (or structs).  This capability too is similar to both Parquet and Avro. Autonomous Database now supports querying object store data that is captured in ORC format – in addition to text, Avro and Parquet.  And, across the structured file types – you can now also query complex data types.  Let’s take a look at an example.  We’ll extend the movie file that was used in our previous Avro post (we downloaded this data from Wikipedia) - this time using the ORC file type with an extra column. The movie file has the following schema: id int original_title string overview string poster_path string release_date string vote_count int runtime int popularity double genres array<struct<id:int,name:string> Notice that each movie is categorized by multiple genres (an array of genres).  This array is an array of objects - or structs:  each genre has an id (integer) and a name (string).  Let's take a look at how to access this data in Autonomous Database.  After creating a credential object that enables access to Oracle Object Storage, create a table using dbms_cloud.create_external table: begin   DBMS_CLOUD.create_credential (     credential_name => 'OBJ_STORE_CRED',     username => abc@oracle.com',     password => '12345'   ) ;     dbms_cloud.create_external_table (     table_name =>'movie_info',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'https://objectstorage.us-phoenix-1.oraclecloud.com/n/abcd/b/movies/o/movie-info.orc',     format =>  '{"type":"orc",  "schema": "first"}'     ); end; / Notice you don’t have to specify the shape of the data.  The columns and data types are automatically derived by reading the metadata contained in the ORC file.  This created a table with the following DDL:  CREATE TABLE "ADMIN"."MOVIE_INFO"     ( "ID" NUMBER(10,0),       "ORIGINAL_TITLE" VARCHAR2(4000 BYTE) COLLATE "USING_NLS_COMP",       "OVERVIEW" VARCHAR2(4000 BYTE) COLLATE "USING_NLS_COMP",       "POSTER_PATH" VARCHAR2(4000 BYTE) COLLATE "USING_NLS_COMP",       "RELEASE_DATE" VARCHAR2(4000 BYTE) COLLATE "USING_NLS_COMP",       "VOTE_COUNT" NUMBER(10,0),       "RUNTIME" NUMBER(10,0),       "POPULARITY" BINARY_DOUBLE,       "GENRES" VARCHAR2(4000 BYTE) COLLATE "USING_NLS_COMP"    )  DEFAULT COLLATION "USING_NLS_COMP"     ORGANIZATION EXTERNAL      ( TYPE ORACLE_BIGDATA       DEFAULT DIRECTORY "DATA_PUMP_DIR"       ACCESS PARAMETERS       ( com.oracle.bigdata.credential.name=OBJ_STORE_CRED         com.oracle.bigdata.fileformat=ORC   )       LOCATION        ( 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/abcd/b/movies/o/movie-info.orc'        )     )    REJECT LIMIT UNLIMITED    PARALLEL ; So, how will you handle the complex data?  Simple… data will be returned as JSON.  Thus, despite the fact that different file types store the same data in different ways, your query can be totally agnostic of this fact.  You see the same JSON output for all common complex types.  Use Oracle's rich JSON query processing capabilities against the table. Running a simple query against this table yields the following: select original_title, release_date, genres from movie_info where release_date > '2000' order by original_title; original_title release_date genres (500) Days of Summer 2009 [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 10,000 BC 2008 [{"id":6,"name":"Comedy"}] 11:14 2003 [{"id":9,"name":"Thriller"},{"id":14,"name":"Family"}] 127 Hours 2010 [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"}] 13 Going on 30 2004 [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 1408 2007 [{"id":45,"name":"Sci-Fi"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":6,"name":"Comedy"},{"id":18,"name":"War"}] Notice that the genres returned as a json array.  Let's make that JSON data more useful.  The JSON can be transformed using Oracle's JSON functions - including the simple "." notation as well as the more powerful transform functions like JSON_TABLE.  The example below queries the table - turning each value of the array into a row in the result set: select original_title,        release_date,        m.genre_name,        genres from movie_info mi,      json_table(mi.genres, '$.name[*]'        COLUMNS (genre_name VARCHAR2(25) PATH '$')                                  ) AS m where rownum < 10; original_title release_date genre_name genres (500) Days of Summer 2009 Drama [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] (500) Days of Summer 2009 Comedy [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] (500) Days of Summer 2009 Horror [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] (500) Days of Summer 2009 Western [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] (500) Days of Summer 2009 War [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] (500) Days of Summer 2009 Romance [{"id":3,"name":"Drama"},{"id":6,"name":"Comedy"},{"id":17,"name":"Horror"},{"id":19,"name":"Western"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 10,000 BC 2008 Comedy [{"id":6,"name":"Comedy"}] 11:14 2003 Family [{"id":9,"name":"Thriller"},{"id":14,"name":"Family"}] 11:14 2003 Thriller [{"id":9,"name":"Thriller"},{"id":14,"name":"Family"}] 127 Hours 2010 Comedy [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"}] 127 Hours 2010 Drama [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"}] 13 Going on 30 2004 Romance [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 13 Going on 30 2004 Comedy [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 13 Going on 30 2004 War [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}] 13 Going on 30 2004 Drama [{"id":6,"name":"Comedy"},{"id":3,"name":"Drama"},{"id":18,"name":"War"},{"id":15,"name":"Romance"}]   JSON_TABLE was used in this case to 1) create a row for each value of the array (think outer join) and 2) the struct was parsed to extract the name of the genre. Autonomous Database continues to expand its ability to effectively query external source.  Look for more to come in this area soon!

Apache ORC is a columnar file type that is common to the Hadoop ecosystem.  It is similar in concept to Apache Parquet; Hortonworks Hadoop clusters often leveraged ORC – while Cloudera clusters...

Autonomous

Oracle Database Vault on Autonomous Database: Quickly and Easily

We often talk about how to protect our data from external threats by implementing various practices such as encrypting our data, only allowing connections via TCPS authentication or even further, deploying our databases in a private network. However, these security practices might not be sufficient to eliminate the threats that can originate from unauthorized privileged user access or unauthorized database changes. Good news is that Oracle Database Vault offers comprehensive access control capabilities to prevent those internal threats. What’s better is that you can use it in your Oracle Autonomous Database on Shared Exadata Infrastructure (ADB-S) as well! In this blog post, we are going to focus on how to take advantage of Database Vault in ADB-S by just following a few simple steps: Enable Database Vault Control Privileged User Access Create a Realm to Protect Your Data Disable Database Vault Enable Database Vault To configure and enable Oracle Database Vault on Autonomous Database, we will follow the steps described in our documentation: Create two users, one to be the Database Vault owner while the other one to be the Database Vault account manager. This is the first step towards implementing separation of duties. ctuzla$ ./sqlplus ADMIN/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 3 21:07:29 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Jun 03 2020 20:55:08 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> create user mydvowner identified by ************; User created. SQL> create user mydvacctmgr identified by ************; User created. Configure Database Vault: SQL> EXEC DBMS_CLOUD_MACADM.CONFIGURE_DATABASE_VAULT('mydvowner', 'mydvacctmgr'); PL/SQL procedure successfully completed. Enable Database Vault: SQL> EXEC DBMS_CLOUD_MACADM.ENABLE_DATABASE_VAULT; PL/SQL procedure successfully completed. Restart your ADB-s instance. Confirm Database Vault is enabled: SQL> SELECT * FROM DBA_DV_STATUS; NAME STATUS ------------------ ------------------- DV_APP_PROTECTION NOT CONFIGURED DV_CONFIGURE_STATUS TRUE DV_ENABLE_STATUS TRUE Control Privileged User Access In the previous section, we have seen how to set up separate users to be the designated Database Vault owner and account manager. This was an important step in order to achieve separation of duties; however, we are not done yet. In Autonomous Database, ADMIN user has DV_OWNER and DV_ACCTMGR roles by default. What this means is that even after configuring and enabling Database Vault with different users, ADMIN user can still perform certain operations such user management thanks to these roles. Even though ADMIN can be considered as a backup account for both Database Vault owner and account manager, it’s fairly easy to revoke these roles from ADMIN as we will see below. Connect to your ADB-S instance as the designated Database Vault account manager and revoke DV_ACCTMGR role from ADMIN: ctuzla$ ./sqlplus MYDVACCTMGR/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 3 22:42:00 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Jun 03 2020 21:18:48 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> revoke dv_acctmgr from ADMIN; Revoke succeeded. Connect to your ADB-S instance as the designated Database Vault owner and revoke DV_OWNER role from ADMIN: ctuzla$ ./sqlplus MYDVOWNER/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 3 22:43:30 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Sun May 31 2020 21:38:21 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> revoke dv_owner from ADMIN; Revoke succeeded. Confirm ADMIN user cannot perform any user operations: ctuzla$ ./sqlplus ADMIN/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 3 23:00:05 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Jun 03 2020 21:30:55 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> create user testuser identified by ************; create user testuser identified by ************ * ERROR at line 1: ORA-01031: insufficient privileges SQL> alter user dvdemo identified by ************; alter user dvdemo identified by ************ * ERROR at line 1: ORA-01031: insufficient privileges SQL> drop user dvdemo; drop user dvdemo * ERROR at line 1: ORA-01031: insufficient privileges Create a Realm to Protect Your Data In this section, we will explore how to create a realm to protect tables of a schema. For this demonstration, I will be using my DVDEMO schema that contains a table named PRODUCTS. Check if ADMIN can access the DVDEMO.PRODUCTS table: ctuzla$ ./sqlplus ADMIN/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 3 23:00:05 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Jun 03 2020 21:30:55 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from dvdemo.products; COUNT(*) ---------- 72 As you can see, ADMIN can access the data that belongs to another schema since it’s not protected by a realm yet. Create a realm for all tables of the DVDEMO schema as Database Vault owner: SQL> begin DBMS_MACADM.CREATE_REALM ( realm_name => 'DVDEMO Realm', description => 'Realm for DVDEMO schema', enabled => DBMS_MACUTL.G_YES, audit_options => DBMS_MACUTL.G_REALM_AUDIT_OFF, realm_type => 1); end; / PL/SQL procedure successfully completed. SQL> begin DBMS_MACADM.ADD_OBJECT_TO_REALM( realm_name => 'DVDEMO Realm', object_owner => 'DVDEMO', object_name => '%', object_type => 'TABLE'); end; / PL/SQL procedure successfully completed. Authorize DVDEMO to access the realm: SQL> begin DBMS_MACADM.ADD_AUTH_TO_REALM( realm_name => 'DVDEMO Realm', grantee => 'DVDEMO', auth_options => DBMS_MACUTL.G_REALM_AUTH_OWNER); end; / PL/SQL procedure successfully completed. Try to access PRODUCTS table as ADMIN and DVDEMO users: ctuzla$ ./sqlplus ADMIN/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 4 00:04:48 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Wed Jun 03 2020 23:00:06 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from dvdemo.products; select count(*) from dvdemo.products * ERROR at line 1: ORA-01031: insufficient privileges ctuzla$ ./sqlplus DVDEMO/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 4 00:05:44 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Sun May 31 2020 20:57:16 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> select count(*) from dvdemo.products; COUNT(*) ---------- 72 As expected, only DVDEMO can access the PRODUCTS table. Disable Database Vault As the final step, let’s take a look at how to disable Database Vault. Disable Database Vault using the following API: ctuzla$ ./sqlplus MYDVOWNER/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 4 00:13:10 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Jun 04 2020 00:08:31 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> EXEC DBMS_CLOUD_MACADM.DISABLE_DATABASE_VAULT; PL/SQL procedure successfully completed. Restart your ADB-s instance. Confirm Database Vault is disabled: ctuzla$ ./sqlplus ADMIN/************@tuzladv_low SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 4 00:16:50 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Last Successful login time: Thu Jun 04 2020 00:04:48 -07:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.5.0.0.0 SQL> SELECT * FROM DBA_DV_STATUS; NAME STATUS ------------------- -------------------------- DV_APP_PROTECTION NOT CONFIGURED DV_CONFIGURE_STATUS TRUE DV_ENABLE_STATUS FALSE Check if ADMIN can access the DVDEMO.PRODUCTS table now: SQL> select count(*) from dvdemo.products; COUNT(*) ---------- 72 It is pretty simple, isn’t it? In just a few steps, we have gone through how to enable Database Vault, how to limit privileged user access via a realm and how to revert everything back to square one by disabling Database Vault. It’s important to note that this is just a small subset of all the capabilities Database Vault offers. If you’d like to learn more about Database Vault, please check out Database Vault Administrator’s Guide.

We often talk about how to protect our data from external threats by implementing various practices such as encrypting our data, only allowing connections via TCPS authentication or even further,...

Autonomous

Autonomous Database Newsletter - June 4-2020

  Autonomous Database on Shared Exadata Infrastructure June 04, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Customer Managed Oracle REST Data Services Workload metrics on performance Hub Timezone selector on Perf Hub Latency metrics now available Easy access to tenancy details New ADB collateral Global Leaders Update Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - ADB now supports Customer Managed Oracle REST Data Services When you use the default ORDS on Autonomous Database, you cannot modify any of the ORDS configuration options. For example, with the default configuration connections for ORDS are preconfigured to use the LOW database service. You can now use a customer managed environment if you want manual control of the configuration and management of Oracle REST Data Services.   LEARN MORE   DOC: About Customer Managed Oracle REST Data Services on ADW, click here. DOC: About Customer Managed Oracle REST Data Services on ATP, click here. DOC: Installing and Configuring Customer Managed ORDS on Autonomous Database, click here. VIDEO: ORDS 101, click here.     DO MORE   BLOG: Examples of how to use REST, click here (see section marked "Examples"). GITHUB: Sample code repository, click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - Workload Metrics on Performance Hub Performance Hub has been expanded to include a new tab, "Workload". This can be used to monitor the workload in the database and helps the user identify performance spikes and bottlenecks. There are 4 regions in this tab: CPU Statistics, Wait Time Statistics, Workload Profile, and Sessions. Each region contains a group of charts that indicate the characteristics of the workload and the distribution of the resources, as shown below. IMAGE - view the screenshot of the new Workload tab containing the four regions and their associated charts are displayed - click here.   LEARN MORE   DOC: See section on "To navigate to Performance Hub in the Oracle Cloud Infrastructure Console interface of an Autonomous Database", click here   BACK TO TOP   (Shared Infrastructure Only) NEW - Performance Hub Now Has a Timezone Selector A time zone control has been added to Performance Hub. This will allow users to view data using an alternate time zone to UTC. They will have the choice of viewing data using either UTC (default), the time zone of their browser client, or the time zone of the database.     BACK TO TOP   (Shared Infrastructure Only) NEW - Latency Metrics Now Available in Service Metrics The Database Service Metrics information help Cloud Admins measure useful quantitative data about their Autonomous Databases. This list of available metrics now includes two new latency metrics: 1) Connection latency - Time taken (in milliseconds) to connect to an Autonomous Database that uses shared Exadata infrastructure in each region from a Compute service virtual machine in the same region. 2) Query latency/response time - Time taken (in milliseconds) to display the results of a simple query on the user's screen. Note - the tests to measure connection latency and query latency are not run against any customer instances. Within the Service Metrics console, it is possible to define and publish alarms against these new metrics. IMAGE - view the screenshot of the new latency metrics graphs on Service Metrics console- click here.   LEARN MORE   Overview of Events Service, click here. List of all the Autonomous Database events, click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - SQL access to tenancy details for ADB When you file a service request for Autonomous Database, you need to provide the tenancy details for your instance. Tenancy details for the instance are available on the Oracle Cloud Infrastructure console. However, if you are connected to the database, you can now obtain these details by querying the CLOUD_IDENTITY column of the V$PDBS view. For example: SELECT cloud_identity FROM v$pdbs; ...will generate something similar to the following: {"DATABASE_NAME" : "DBxxxxxxxxxxxx", "REGION" : "us-phoenix-1", "TENANT_OCID" : "OCID1.TENANCY.REGION1..ID1", "DATABASE_OCID" : "OCID1.AUTONOMOUSDATABASE.OC1.SEA.ID2", "COMPARTMENT_OCID" : "ocid1.tenancy.region1..ID3"}   LEARN MORE   DOC: Obtain tenancy details for ADW to File a Service Request, click here. DOC: Obtain tenancy details for ATP to File a Service Request, click here.     BACK TO TOP   NEW - Collateral For Autonomous Database TRAINING: Autonomous Database Workshops on GitHub 1) Autonomous Database Quickstart - Learn about Autonomous Database on Shared Infrastructure and learn how to create an instance in just a few clicks. To run the workshop click here. 2) Analyzing Your Data with Autonomous Database - Connect using secure wallets, monitor your instance, use Analytics Desktop to visualize data and then use machine learning to try your hand at predictive analytics. To run the workshop click here. VIDEO: Oracle Analytics Channel - Oracle Autonomous Database Experience George Lumpkin discusses getting value from an analytics system and leveraging end-to-end solutions in an ever-changing market. To watch the video click here. VIDEO: Using ATP as a JSON Document Store This video demonstrates how to use ATP as a JSON document store. The video is divided into the following sections: Overview, Accessing JSON collections using a document-store API, querying JSON collections using SQL, loading JSON from object storage and viewing and analyzing data from JSON collections using a notebook. To watch the video click here.     BACK TO TOP     ORACLE GLOBAL LEADERS   EVENT - Global Leaders EMEA 2020 Online Summit: June 23 - June 24   For June the Oracle Global Leaders program is going online, with the goal of keeping you connected with your peers so you can hear the latest Oracle customer and product stories. Gain from valuable real life experience on projects with Autonomous Database, data warehousing, big data, analytics and Oracle Cloud Infrastructure. The focus for this live online event will be a series of customer panels based on the following topics: Oracle Cloud Infrastructure Oracle Autonomous Database Oracle Exadata for Analytics Oracle Analytics Oracle Machine Learning Oracle Database Development To register for this event, click here and to view the agenda, click here. If you have any questions, email Reiner Zimmermann In association with intel® WEBCAST: The MineSense Story: Smart Shovelling with ADW and APEX - June 30 Hear how Oracle Autonomous Data Warehouse + APEX is powering smart and efficient Mining and enabling the “Scale Up” of startup company MineSense. Learn how MineSense reduced implementation time from weeks or months to days, experienced over 2x performance improvement and implemented a secure, scalable infrastructure. To register for this Zoom webinar, click here.     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous Databaseon Shared Exadata InfrastructureJune 04, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Big Data

Oracle Big Data SQL 4.1 is now available

I'm pleased to announce that Oracle Big Data SQL 4.1 is now available.  This is a significant release that enables you to take advantage of new Oracle Database 19c capabilities plus core Big Data SQL enhancements. Big Data SQL enables organizations to easily analyze data across Apache Hadoop, Apache Kafka, NoSQL, object stores and Oracle Database leveraging their existing Oracle SQL skills, security policies and applications with extreme performance.  Oracle Database 19c introduces the following features to make this support even better: Enhanced Information Lifecycle Management with Hybrid Partitioned Tables   a single partitioned table can now source data from both Oracle data files and external stores (object storage, hive, HDFS and local text files).  Store the external data in native big data formats (parquet, orc, csv, avro, etc.) - enabling that data to be shared with Spark and other data lake applications In-Memory External Tables:  load frequently accessed external data in-memory to dramatically improve performance Core Big Data SQL reader functionality for object storage has also been enhanced - offering support for new sources, file types and complex columns: ORC columnar file type support has been added to existing parquet, avro and text file support Complex data types (e.g. arrays, records, etc.) can now be queried using JSON functions Vastly improved support for text file attributes  Azure Blob Storage added to existing Oracle Object Storage and Amazon S3 support Apache Kafka support has also been enhanced with Oracle SQL Access to Kafka.  This new capability removes the requirement for the Oracle Kafka hive storage handler and provides a more systematic approach to querying Kafka streams. Enjoy the new release!

I'm pleased to announce that Oracle Big Data SQL 4.1 is now available.  This is a significant release that enables you to take advantage of new Oracle Database 19c capabilities plus core Big Data SQL...

Autonomous

Autonomous Database Newsletter - April 30-2020

  Autonomous Database on Shared Exadata Infrastructure April 30, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Service Level Agreement for ADB Per-second billing Support for SODA documents and collections ADB Extensions for IDEs Stateful Rule Support in Private Endpoints Enhancements for Data Pump Global Leaders Program - Update Quick Links Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. To find out how to subscribe to this newsletter click here.   (Shared Infrastructure Only) NEW - Autonomous Database now comes with an SLA The service level agreement for ADB has just been published and it applies only to the deployment of an Autonomous Database on shared infrastructure. The pillar document listed below in the section headed LEARN states that: Oracle will use commercially reasonable efforts to have the Oracle Cloud Infrastructure – Autonomous Database Services on Shared Infrastructure with a Monthly Uptime Percentage of at least 99.95% during any calendar month (the "Service Commitment"). In the event an Autonomous Database Service on Shared Infrastructure does not meet the "Service Commitment" then customers will be eligible to receive Service Credits. Note - there are specific definitions of key terms in the pillar document (listed below in the section headed LEARN) that you should read and understand.   LEARN MORE PDF: Oracle PaaS and IaaS Public Cloud Services Pillar Document (April 2020),  click here. Autonomous Database information is covered in Category 7, which starts on page 4 with ADB covered on pages 17 and 18.   BACK TO TOP   (Shared Infrastructure Only) NEW - Per-Second Billing For Autonomous Database Autonomous Database instance CPU and storage usage is now billed by the second, with a minimum usage period of one minute. Previously Autonomous Database billed in one-hour minimum increments and partial usage of an hour was rounded up to a full hour. This change benefits the following types of use cases: Where you only run your ADB instances during work hours and stop instances at end of working day Where you start-stop your instances to run overnight batch jobs Where you have ADB instances which typically have a short lifecycle – sandboxes, QA, training, integration testing instances.   LEARN MORE DOC: Per-second billing announcement for ADW,  click here DOC: Per-second billing announcement for ATP,  click here DOC: OCI Per-second billing general announcement which covers ADB,  click here BLOG: Review of per-second billing on OCI,  click here   BACK TO TOP   (Shared Infrastructure Only) NEW - Support for SODA Documents and Collections   Autonomous Database now supports loading and using Simple Oracle Document Architecture (SODA) documents and collections. SODA allows developers to store, search, and retrieve document collections, typically JSON documents, without using SQL. It is possible to also access document collections with SQL/JSON operators which means that ADB now combines the simplicity of a document database with the power of a relational database. The Oracle Cloud Infrastructure console has been updated with a link to access the SODA drivers for several languages and frameworks including: Java, Node.js, Python, Oracle Call Interface, and PL/SQL, as shown below.   LEARN MORE DOC: Using JSON Documents with ADW,  click here DOC: Using JSON Documents with ATP,  click here Documentation covers the following key topics: Work with Simple Oracle Document Access (SODA) in Autonomous Database SODA Collection Metadata on Autonomous Database Work with JSON Documents Using SQL and PL/SQL APIs on Autonomous Database Load JSON Documents with Autonomous Database   DO MORE   For examples of using SODA see the following whitepaper:  Developing schemaless applications using the Simple Oracle Document Access APIs.     BACK TO TOP   (Shared Infrastructure Only) NEW - ADB Extensions for IDEs ADB extensions enable developers to connect to, browse, and manage Autonomous Databases directly from their IDEs. There are extensions for the following: Eclipse, Microsoft Visual Studio and Visual Studio Code. Users with permissions to manage the databases can perform a number of actions (depending on their IDE), such as: Sign up for Oracle Cloud Connect to a cloud account using a simple auto-generated config file and key file Create new or clone existing Always Free Autonomous Database, Autonomous Database Dedicated, and Autonomous Database Shared databases Automatically download credentials files (including wallets) and quickly connect, browse, and operate on Autonomous Database schemas Change compartments and regions without reconnecting Start, stop, or terminate Autonomous Databases Scale up/down Autonomous Database resources Restore from backup Update instance credentials and/or license type used Rotate wallets Convert Always Free ADBs into paid ADBs   LEARN MORE   DOC: IDE Extensions in ADW,  click here DOC: IDE Extensions in ATP,  click here   BACK TO TOP   (Shared Infrastructure Only) NEW - Stateful Rule Support in Private Endpoints When configuring a private endpoint one of the steps is to define a security rule(s) in a Network Security Group. This creates a virtual firewall for the Autonomous Database and allows connections to the Autonomous Database instance. The Network Security Group can now be configured with stateful rules.   LEARN MORE   DOC: Configure Private Endpoints with ADW, click here DOC: Configure Private Endpoints with ATP, click here BLOG: Announcing Private Endpoints in Autonomous Database on Shared Exadata Infrastructure, click here.     BACK TO TOP   (Shared Infrastructure Only) UPDATED - Enhancements to Data Pump Support Two specific improvements relating to how Data Pump dump files can be used with ADB. It is now possible to: Query Data Pump dump files in the Cloud by creating an external table Use Data Pump dump files in the Cloud as source files to load data   LEARN MORE DOC: Query external Data Pump dump files with ADW, click here DOC: Load Data Pump dump files into an existing table with ADW, click here.   DOC: Query external Data Pump dump files with ATP, click here DOC: Load Data Pump dump files into an existing table with ATP, click here.     BACK TO TOP     ORACLE GLOBAL LEADERS PROGRAM   WEBCAST - How to Scale Your Services with Autonomous Database To Infinity 29 April 2020 at 15:00 CET Join this Oracle Global Leaders webcast to discover directly from Peter Merkert, CTO & Co-Fouder, Retraced GmbH how they are using Oracle Autonomous Database and the benefit they are getting. To register click here. To learn more, also see the video: How Autonomous & Blockchain Provide A Sustainable Transparent Clothing Supply Chain?     WEBCAST - From BillBoards to Dashboards using ADW and OAC 13 May 2020 at 12:00 PM ET Join this Oracle Global Leaders webcast to discover directly from Derek Hayden, SVP, Data Strategy and Analytics at OUTFRONT Media, about the role of Autonomous Data Warehouse in the new data platform for OUTFRONT Media moving in the digital age. Derek will discuss initial success with ADW, OAC and how this platform is being used for OUTFRONT's growth, efficiencies and data management in the out of home advertising industry. To register click here.       WEBCAST - Building low code application using Oracle Autonomous Database and Application Express 19 May 2020 at 15:00 CET Join this Oracle Global Leaders webcast to discover directly from Dr. Holger Friedrich, CEO, SumIT how they are using Oracle Autonomous Database and the benefit they are getting. Oracle Application Express is a powerful low-code web application development framework built-in and ready-to-use with Oracle Autonomous Database. It is instantly available in every provisioned service instance. In this webcast we will explain and demonstrate how sumIT and partner document2relation employ APEX and ATP to provide quick migration services and application development for migrating legacy Lotus Notes applications to the Cloud. To register click here.     WEBCAST - Smart Shovelling with ADW & APEX 13 May 2020 at 12:00 PM ET Join our Global Leaders webinar to hear from Frank Hoogendoorn, CDO of MineSense: a technology solution provider for the Mining Industry that combines IoT and machine learning to create "Smart Shovel" technologies that optimize mine operations, all powered by Oracle solutions! In this Webinar, Frank will explain how MineSense has transitioned from "start-up" to a "scale-up", and how Autonomous Data Warehouse with built-in APEX is enabling rapid business growth with international Mining customers that include some of the world’s largest Copper and Zinc producers. Backed by APEX and ADW, their disruptive ‘smart shovel’ technology collects and analyzes orebody data in real-time, where machine learning based algorithms drive processing decisions to boost mine productivity while reducing energy consumption, all of which leads to more sustainable mining operations. With the flexibility and low cost setup of ADW, the ease of application development and deployment with APEX, and superb on-demand data processing performance, MineSense will talk about how Oracle Solutions, (some even running right on mining shovels!), have been key in setting up a pivotal digital platform, where IT is no longer seen as a business cost but rather a key driver for growth. To register click here.     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous Databaseon Shared Exadata InfrastructureApril 30, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

The simplest guide to exporting data from Autonomous Database directly to Object Storage

One of the best ways to export data out of your Autonomous Database (ADB), whether to transfer data to a database lying in a different region or even keep a copy of the data in a single place for other teams in your organization to access, is via Oracle Data Pump. Recently we released a much requested feature enabling the ability to export data from your ADB Instance directly to your Oracle Object Storage bucket. For this walkthrough, I assume you have:  A Oracle Cloud Account (Sign Up for a trial if you don't have one. It's totally free!) An Autonomous Database Instance with some data in it (Follow Labs 1 & 3 in this ADB Workshop if you don't have one) Once you have logged in to your Oracle Cloud account with an ADB Instance, lets dive right in to these steps below to export your data.   Step 1: Create an Object Storage bucket Using the hamburger menu on the left, navigate to your Object Storage. Select a compartment which you would like to put your data in and hit "Create Bucket". The default options work well; In the popup, enter a bucket name (we will use 'examplebucket') and continue on to create the bucket . This bucket will hold your exported dump files. Note: Data Pump Export supports both OCI Object Storage and OCI Classic Object Storage.   Step 2: Create a Credential: Authorization between your ADB instance and Object Storage You will need to generate an auth token password for a new credential, navigate the left side menu to Identity -> Users -> Your User. In your selected user, on the bottom left, you will see "Auth Tokens". When clicked, you will see an option generate a token. Note it down somewhere secure, since you cannot view this later. Note: You must use a Swift auth token based credential here, and cannot use an OCI native Signing Key credential. Navigate to your existing Autonomous Database instance (it may be ADW or ATP) via the left-hand menu. In your instance's console, go to the "Tools" tab and navigate to SQL Developer Web. Login with your ADMIN Database User. Copy the following script, in your worksheet. Fill in your Cloud User's username and Auth Token as password, and run it to create your user credential. BEGIN   DBMS_CLOUD.CREATE_CREDENTIAL(     credential_name => 'EXPORT_CRED',     username => 'someone@oracle.com',     password => 'erioty6434900z……'   ); END;   Step 3: Set your credential as the Default Credential (for Data Pump v19.6 and below) Next, in SQL Developer Web, run the following command after replacing in your database user: ALTER DATABASE PROPERTY SET DEFAULT_CREDENTIAL = '<Database User Name>.EXPORT_CRED' You have now set your database's default credential. Note: I am assuming here you are running Data Pump version 19.6 or below since that is what is currently available on most platforms. In versions 19.7 and above, you will not need to set a default credential and be able to run your exports with a credential parameter!   Step 4: Install Oracle Data Pump If you're ahead in the game and already have a recent installation of Oracle Data Pump, skip to Step 5. If not, the easiest method to install Data Pump is via Instant Client. Download that as well as the Tools Package, which includes Oracle Data Pump, for your platform from Oracle Instant Client Downloads. Follow the documentation for step-by-step installation details.   Step 5: Run Oracle Data Pump Export The Data Pump utility is standalone from the database. Once installed, in your terminal / command line we will run the command below. Notice, we are using the "default_credential" from Step 3 as the first argument in the 'dumpfile' parameter.  The second argument is pointing to your Object Storage bucket's URI. You may use a Swift URI if you have one, instead of an OCI Native URI as is in the example below. In this URI, replace the: <region ID> with the region you are using. You can identify this by looking at the top right corner of your Oracle Cloud page. Go into "Manage Regions" or look at this page to get your region's identifier. <namespace> with Object Storage namespace. This can be found in Profile -> Tenancy (top right corner of the Oracle Cloud page). Look for your "Object Storage Namespace", it may be the same as your tenancy name or a different ID. ​​​​​ Here we used the bucket name "examplebucket", if you used a different bucket name also replace that in the URI. With encryption enabled, you will be prompted for a password. For more information on the export parameters and file size limits have a look at the documentation. expdp admin/password@ADWC1_high \ filesize=5GB \ dumpfile=default_credential:https://objectstorage.<regionID>.oraclecloud.com/n/<namespace>/b /examplebucket/o/exampleexport%U.dmp \ parallel=16 \ encryption_pwd_prompt=yes \ logfile=export.log \ directory=data_pump_dir Of course, depending on the size of your database, this database export can take minutes or hours.   Final Step & Thoughts That's a wrap! Once completed successfully, you can navigate back to your Object Storage bucket we created in Step 1 and you will see your newly exported dump files. The file names begin with "exampleexport". Since exporting to object store uses chunks for optimum performance, you may see additional files other than your dump ".dmp" files in your bucket. Don't delete them, but you may ignore them and should only have to interact with your dump files to access your data. Note: If you followed this guide to a T but are seeing an authorization or a URI not found error, you (or your Cloud Admin) may need to grant object storage access to your Cloud User or Group.   Here, we looked at exporting your data to the object store using the popular Data Pump utility running on your machine or server, which makes for a simple workflow to include in your data pipeline. In coming weeks, we will also support running Data Pump import/export (using the ORACLE_DATAPUMP driver) from within the database using your PL/SQL scripts using DBMS_CLOUD, so keep your eyes peeled for more Data Pump goodness! ?

One of the best ways to export data out of your Autonomous Database (ADB), whether to transfer data to a database lying in a different region or even keep a copy of the data in a single place for...

Autonomous

Autonomous Database Newsletter - April 08-2020

  Autonomous Database on Shared Exadata Infrastructure April 08, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. Options for upgrading to Database 19c Enhancements to Data Pump New menu option to restart ADB instances Enhacements to wallets Global Leaders Program - Update Day-with-Development Program Webcast Program Quick Links     (Shared Infrastructure Only) NEW - Upgrade an Autonomous Database Instance to Oracle Database 19c Oracle Database 19c is now available on Autonomous Database for all regions. In select regions, customers can choose between Oracle Database 18c and Oracle Database 19c for new databases. The default version for all new databases is 19c and going forward, most new Autonomous Database features will only be available for databases using Oracle Database 19c. Please start planning for the upgrade of your applications to Database 19c. For most applications, there should not be any changes required. However, it is recommended you should test your applications with Database 19c. You can easily test your applications by creating a clone of your current database to Oracle Database 19c, and testing your application using this clone. Oracle Database 18c is scheduled to be available in Autonomous Database until September 2020. You can choose to upgrade your database at any time during this period. After that time, all existing databases will be automatically upgraded to Oracle Database 19c. There are several approaches to upgrading your database which are described in the documentation, see links below. Note: Autonomous Database plans to offer a new upgrade path starting in April, with a single button that provides a one-way, in-place upgrade.   LEARN MORE   DOC: Upgrade an ADW instance to Database 19c, click here DOC: Upgrade an ATP instance to Database 19c, click here   DO MORE   BLOG: More information about ADB cloning feature is here     BACK TO TOP   (Shared Infrastructure Only) NEW - Enhancements to Data Pump 1) Support For Pre-Authenticated URLs This enhancement allows Oracle Data Pump to use Oracle Cloud Infrastructure pre-authenticated URIs for source files on Oracle Cloud Infrastructure Object Storage. Note: Customers should carefully assess the business requirement for and the security ramifications of pre-authenticated access. There is more guidance available in the documentation.   LEARN MORE   DOC: Import Data Using Oracle Data Pump on ADW, click here DOC: Import Data Using Oracle Data Pump on ATP, click here   DO MORE   VIDEO: Database Migration with MV2ADB - a tool that provides support for moving existing databases to Oracle Autonomous Database, click here O.COM: Oracle Database Cloud Migration Solutions page, click here     (Shared Infrastructure Only) 2) Data Pump Export to Object Store It is now possible for Oracle Data Pump to export directly to Oracle Object Store. This simplifies the process of moving data between Autonomous Data Warehouse and other Oracle databases. This export method is supported with Oracle Cloud Infrastructure Object Storage and Oracle Cloud Infrastructure Object Storage Classic.   LEARN MORE   DOC: Move Data with Data Pump Export to Object Store on ADW, click here DOC: Move Data with Data Pump Export to Object Store on ATP, click here   BACK TO TOP   (Shared Infrastructure Only) NEW - Menu Option to Restart ADB Instance A new menu option has been added to the OCI console to automate the process of stopping and then starting an ADB instance. The new menu option is labelled "Restart" as shown below:   LEARN MORE   DOC: How to restart an ADW instance, click here DOC: How to restart an ATP instance, click here   BACK TO TOP   (Shared Infrastructure Only) NEW - README file added to Wallet The Oracle client credentials wallet zip file now contains a README file. This file provides information about the wallet expiry date. An example of the contents of the new readme file is shown below:   Wallet Expiry Date ----------------------- This wallet was downloaded on 2020-04-03 10:19:43.4 UTC. The SSL certificates provided in this wallet will expire on 2025-03-31 21:26:18.928 UTC. In order to avoid any service interruptions due to an expired SSL certificate, you must re-download the wallet before this date.   LEARN MORE   DOC: Download client credentials (wallets) for an ADW instance, click here DOC: Download client credentials (wallets) for an ATP instance, click here     BACK TO TOP   VIDEO Oracle Global Leaders - Day With Development A recording of a recent Oracle Global Leaders Day with Development workshop is now available for viewing. The workshop covers general architecture overview of Data Management Services on OCI and how the following services work together: Big Data Service Autonomous Data Warehouse Data Integration Service Streaming Service, Data Flow Oracle Analytics Cloud To watch the video click here.     WEBCAST - Global Leaders Customers: Autonomous Database Use Cases Our Global Leaders team will be hosting a series of webcasts over the next six months where various customers will talk publicly about how they are using Autonomous Database. The schedule for April is: April 29, 2020 at 15:00 CET Oracle Global Leaders Webcast: Oracle Autonomous Database from the view of a CIO. Join this webcast to discover directly from Rok Planinsec, CIO of Unior, SI how they are using Oracle Autonomous Database and the benefit they are getting. At Unior Oracle Autonomous Data Warehouse and Oracle Analytics enable real-time operational reporting from months to seconds for more strategic decision-making with zero database administration.     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Shared Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous DatabaseonShared ExadataInfrastructureApril 08, 2020       Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Big Data

Now Available! Oracle Big Data Service

  After a successful controlled GA, we are excited to announce general availability for Oracle Big Data Service in the following regions:  Ashburn, Sao Paulo, Frankfurt, Tokyo, Seoul, London and Phoenix.  Oracle Big Data Service is an automated cloud platform service designed for a diverse set of big data use cases and workloads. From agile, short-lived clusters used to tackle specific tasks to long-lived clusters that manage large data lakes, Big Data Service scales to meet all big data requirements at a low cost and with the highest levels of security. Create new big data clusters or efficiently extend your on-premise big data solutions – and leverage the full capabilities of Cloudera Enterprise Data Hub along with Oracle Big Data analytics capabilities. Take advantage of Oracle Cloud SQL to enable new and existing applications to gain insights from data across the big data landscape using Oracle’s advanced SQL dialect – including data sourced from Hadoop, Object Storage, NoSQL and Kafka. And, use the languages of your choice – including Python, Scala, R and more – for machine learning, graph and spatial analytics. Go to Big Data Service on oracle.com for more information about the new service

  After a successful controlled GA, we are excited to announce general availability for Oracle Big Data Service in the following regions:  Ashburn, Sao Paulo, Frankfurt, Tokyo, Seoul, London...

Newsletter

Autonomous Database Newsletter - February 26-2020

Autonomous Database on Shared Exadata Infrastructure February 26, 2020 Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the following new features and topics: Support for Private Endpoints Global Leaders Program - Update Quick Links Don't forget to checkout our "What's New" page for a comprehensive archive of all the new features we have added to Autonomous Database - click here. (Shared Infrastructure Only) NEW - Autonomous Database Now Supports Private Endpoints It is now possible to restrict access to Autonomous Database by specifying a private endpoint within a Virtual Cloud Network (VCN). Configuration of private access is done when provisioning or cloning an Autonomous Database - allowing all traffic to and from an Autonomous Database to be kept off the public internet. During provisioning or cloning, it is possible to specify private access for an Autonomous Database by selecting Virtual cloud network within the Choose network access section of the create/clone dialog, as shown below: Description of the illustration adb_private_vcn.png The configuration process for enablishing private endpoints involves three steps which must be done before provisioning or cloning an Autonomous Database: Create a VCN within the region that will contain your Autonomous Database. Configure a subnet within the VCN, configured with default DHCP options. Specify at least one network security group (NSG) within the VCN - used to specify the ingress and egress rules for the Autonomous Database. Note: Private Endpoints is currently rolling out across all ou data centers. As of today, Feb 26, it is live in the following data centers: Amsterdam Ashburn Frankfurt Jeddah London Melbourne Osaka Phoenix Seoul Tokyo Toronto It will be available shortly in the remaining data centers. LEARN MORE   DOC: Configuring Private Endpoints with ADW: click here DOC: Configuring Private Endpoints with ATP: click here. BLOG: Announcing Private Endpoints in Autonomous Database on Shared Exadata Infrastructure: click here.   DO MORE The documentation includes two sample network scenarios: Sample 1: Connecting from Inside Oracle Cloud Infrastructure VCN Sample 2: Connecting from Your Data Center to Autonomous Database DOC: Private Endpoint Configuration Examples on ADW: click here DOC: Private Endpoint Configuration Examples on ATP: click here. BACK TO TOP EVENTS Oracle Global Leaders - Day With Development As part of our Global Leaders program, Oracle Product Management will be hosting a series of workshops for Architects, DBAs, Application Developers and for those interested in learning more about Oracle Information Management. Content will cover a general architecture overview, various cloud service capabilities (such as Autonomous Data Warehouse, Oracle Data Integration and Oracle Analytics Cloud) and how these services work together. All Services will be shown live so you will experience the look and feel of each product along with how to work with our complete range of data warehouse services. The schedule for this series of "Day With Development" events in the US is:   March 26        Irving   April 20       Nashville   April 21       Louisville   May 21       New York The schedule for this series of "Day With Development" events in the EMEA is:   March 3       Madrid, Spain   March 5       Roma, Italy   March 9 (Morning)       Dubai, United Arab Emirates   March 9 (Afternoon)       Dubai, United Arab Emirates   March 12       Istanbul, Turkey   March 16       Warszawa, Poland   March 19       Colombes cedex, France   March 23       Moscow, Russian Federation2 Please use the above links to register for an event in your area and get more information the general Terms and Conditions.   VIDEO - Rosendin Electric Rosendin Electric uses Oracle Autonomous Data Warehouse with Oracle Analytics Cloud to capture data in various formats and sources, making that data available to the right people to extract meaningful insights and drive smarter decision making. Note - click on the above image to watch the video or click here. Thanks to our Global Leaders team for helping to deliver this great customer video WEBCAST - Global Leaders Customers: Autonomous Database Use Cases Our Global Leaders team will be hosting a series of webcasts over the next six months where various customers will talk publicly about how they are using Autonomous Database. The schedule for March and April is: March 4, 2020 at 15:00 CET Oracle Global Leaders Webcast: Managing 1 PB of data with Oracle Autonomous DW. Join this webcast to discover directly from Manuel Marquez, Openlab Coordinator at CERN how they are using Oracle Autonomous Database and the benefit they are getting. Since 1982, CERN has been successfully adopting Oracle technology. Now, CERN is investigating Autonomous Data Warehouse to improve the performance of its research infrastructures. The Partnership with Oracle is key in monitoring large quantities of data. April 29, 2020 at 15:00 CET Oracle Global Leaders Webcast: Oracle Autonomous Database from the view of a CIO. Join this webcast to discover directly from Rok Planinsec, CIO of Unior, SI how they are using Oracle Autonomous Database and the benefit they are getting. At Unior Oracle Autonomous Data Warehouse and Oracle Analytics enable real-time operational reporting from months to seconds for more strategic decision-making with zero database administration. BACK TO TOP oracle.com ADW ATP Documentation ADW ATP TCO Calculator Serverless Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database Customer Forums ADW ATP BACK TO TOP Oracle Autonomous Database BACK TO TOP

Autonomous DatabaseonShared Exadata InfrastructureFebruary 26, 2020 Welcome to our latest customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This newsletter covers the...

Announcing Private Endpoints in Autonomous Database on Shared Exadata Infrastructure

Access to a database using private IP addresses has been one of the most common requests for Autonomous Database on Shared Exadata Infrastructure, especially from enterprise customers. Today, we are announcing support for private IP addresses in Autonomous Database on Shared Exadata Infrastructure. With the Private Endpoints functionality Autonomous Database customers will now be able to assign a private IP address and a private hostname to their database in their Virtual Cloud Network. This completely disables the public endpoint for the database, ensuring that no client can access the database from the public internet. For a detailed explanation of this feature including configuration examples, please see this blog post. Private Endpoints functionality further enhances the security of the Autonomous Database for customers with on-premises and Virtual Cloud Network connectivity requirements. We believe the Private Endpoints functionality will be especially beneficial for customers whose security standards mandate private IP addresses for their applications and databases. Autonomous Database continues to offer access via public endpoints as well. Note that private endpoints and public endpoints are mutually exclusive. If you want your databases to be accessible from the public internet in addition to your Virtual Cloud Network and your on-premises network, you should choose to use a public endpoint. Even with public endpoints, you can configure your database so that it is only accessible from trusted clients or networks. You can also ensure the network traffic between the database and your clients in your Virtual Cloud Network or your on-premises network stays private and does not traverse the public internet. To configure databases with public endpoints, you can use the following features: The Service Gateway for connecting your clients running in your Virtual Cloud Network to the database privately without going through the public internet. FastConnect or VPN Connect for connecting your on-premises clients to the database privately without going through the public internet. Network access control lists (ACLs) for restricting access to your database from only specific client IP addresses or networks so that untrusted clients cannot reach the database from the public internet. Note that Autonomous Database on Shared Exadata Infrastructure always uses SSL authentication and encryption between clients and the database - for both public and private endpoints. All users of a database must have a wallet containing the required connectivity files like SSL certificates, SQL*Net and JDBC configuration files. Any user who does not have access to these files and a database username/password is not able to connect to the database. SSL certificates ensure that, even with a public endpoint, only authorized users are able to attempt to connect to a database. Stay tuned for more posts on networking and connectivity with Autonomous Database!

Access to a database using private IP addresses has been one of the most common requests for Autonomous Database on Shared Exadata Infrastructure, especially from enterprise customers. Today, we are...

How to level up and invoke an Oracle Function (or any Cloud REST API) from within your Autonomous Database

  Recently, we released functionality in the Autonomous Database Shared Infrastructure (ADB-S) to enable a user to call REST API Endpoints using simple PL/SQL scripts that run directly in the database. ADB supports REST API calls from the 3 major cloud providers, Oracle Cloud, AWS and Azure. This feature gets rid of the need to deploy a running server someplace and then call a cloud REST API via a programmable tool such as oci-curl; instead, you simply call the DBMS_CLOUD.SEND_REQUEST procedure with familiar PL/SQL syntax and it runs straight out of your database server. This procedure makes use of the underlying UTL_HTTP database package. This also means you can invoke your REST API call script with existing in-built database features like triggers and job schedulers that your existing code is likely already using! Below, I give an example of using the DBMS_CLOUD package to invoke an Oracle Function. Oracle Functions are fully managed, highly scalable, Functions-as-a-Service platform, available in the Oracle Cloud  (OCI) and powered by the Fn Project open-source engine. An Oracle Function is intended to be a single unit of work deployed as a serverless, callable function in OCI, which is billed only for the resources consumed during the function's execution.   1)  Create and deploy an Oracle Function If you already have an Oracle Function deployed and ready to go in Oracle Cloud, jump to (2) Before we can jump right into deploying an Oracle Function, we must perform the following steps to get set up. Since there are several steps, we will defer to the well-structured Functions documentation: Configure your Tenancy to create an Oracle Function Configure your Client Environment for Oracle Function Development   Next, we will be deploying a basic Oracle Function that accepts a parameter during invocation and returns "Hello <parameter>!" as a response. This involves registering a Docker image with the helloworld-func function code, and then deploying the function to an application in Oracle Functions. Create and deploy your Oracle Function     2) Invoke your Oracle Function using DBMS_CLOUD   Next, we will use the new DBMS_CLOUD functionality in ADB to invoke the function we just deployed. Open up SQL Developer Web from ADB instance Service Console and run the following scripts as directed. If you don't yet have an ADB instance and need a guide on how to set one up click here. Create a user credential that is required for authentication to call OCI APIs by filling in your user_ocid, tenancy_ocid, private_key and fingerprint. Click here if you are unsure where to find this information. BEGIN DBMS_CLOUD.CREATE_CREDENTIAL (        credential_name => 'OCI_KEY_CRED',        user_ocid       => 'ocid1.user.oc1..aaaaaaaam2...',        tenancy_ocid    => 'ocid1.tenancy.oc1..aaaaaaaakc...',        private_key     => 'MIIEogIBAAKCAQEAtU...',        fingerprint     => 'f2:db:d9:18:a4:aa:fc:83:f4:f6..'); END; /   And finally, we use the SEND_REQUEST procedure to invoke the deployed function using the function endpoint. You may identify your function's invoke endpoint using this CLI command or simply copying it from your Oracle Cloud UI under Developer Services     Replace the uri parameter below with your Function's invoke endpoint and, if you like, your own custom name in the body parameter.   SET SERVEROUTPUT ON   DECLARE     resp DBMS_CLOUD_TYPES.resp;   BEGIN     --HTTP POST Request     resp := DBMS_CLOUD.send_request(                credential_name => 'OCI_KEY_CRED',                uri => 'https://5pjfkzq5fhq.ca-toronto-...actions/invoke',                method => DBMS_CLOUD.METHOD_POST,                body => UTL_RAW.cast_to_raw('Nilay')             );         -- Response Body in TEXT format   DBMS_OUTPUT.put_line('Body: ' || '------------' || CHR(10) ||   DBMS_CLOUD.get_response_text(resp) || CHR(10));      -- Response Headers in JSON format   DBMS_OUTPUT.put_line('Headers: ' || CHR(10) || '------------' || CHR(10) ||   DBMS_CLOUD.get_response_headers(resp).to_clob || CHR(10));     -- Response Status Code   DBMS_OUTPUT.put_line('Status Code: ' || CHR(10) || '------------' || CHR(10) ||   DBMS_CLOUD.get_response_status_code(resp)); END; /  If all went according to plan, you should see a 200 OK Response Code and the response text "Hello Nilay!" (or the name you passed into the function) as in the screen below. Note: While invoking Oracle Functions, use the troubleshooting guide to help resolve issues. In addition to that, here are two places I stumbled so you won't have to: If you are still seeing access errors after already uploading your public key to your User, or creating the policies to give Functions access to your user, you may want to wait a few minutes. It sometimes may take a little while for the changes to propagate. If you are seeing a 502 error and are using a public subnet in your VCN, you may need to create an internet gateway and set up its routing table (you can use CIDR 0.0.0.0/0 as default) to give public internet access to your gateway. Click here for more information about networking or to use the simplified Virtual Networking Wizard.     While this is a simple example to walk you through the necessary steps, Oracle Functions is an extremely powerful service and the ability to call Functions from the database expands Autonomous Database functionality to essentially any custom functionality you desire. You may of course also use the SEND_REQUEST procedure to call any other Cloud Service REST API, such as object storage operations, instance scaling operations, Oracle Streams, and many more!

  Recently, we released functionality in the Autonomous Database Shared Infrastructure (ADB-S) to enable a user to call REST API Endpoints using simple PL/SQL scripts that run directly in the database....

Newsletter

Autonomous Database Newsletter - February 18-2020

  Autonomous Database on Shared Exadata Infrastructure February 18, 2020       Welcome to our first customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This first newsletter covers the following new features and topics: Database Vault now available APEX 19.2 now available in Always Free ADB Multiple DB support New Graph Server supports ADB Easier migrations New data centers Global Leaders Program - Day With Development Quick Links   Don't forget to checkout our "What's New" page for Autonomous Database on oracle.com, click here.   BACK TO TOP   (Shared Infrastructure Only) NEW - Using Oracle Database Vault with Autonomous Database Oracle Database Vault implements powerful security controls for your Autonomous Database. These unique security controls restrict access to application data by privileged database users, reducing the risk of insider and outside threats and addressing common compliance requirements.     LEARN MORE   DOC: Using Oracle Database Vault with ADW: click here DOC: Using Oracle Database Vault with ATP: click here. PPT: OOW2018 - Autonomous and Beyond: Security in the Age of the Autonomous Database: click here DOC: Database 19c documentation for Database Vault, click here.     SEE MORE   VIDEO: Database Vault Overview, click here. PDF: Database Vault FAQ - click here PDF: Database Vault Data Sheet - click here PDF: Oracle Database Vault Overview - click here PDF: Oracle Database Vault Best Practices - click here PDF: Oracle Database Vault DBA Best Practices - click here     DO MORE   VIDEO: Database Vault - Enforcing separation of duties, click here. VIDEO: Database Vault - Enforcing Trusted Path Access Control, click here. VIDEO: Database Vault Advanced Use Cases II - Operations Control, Database Vault Simulation Mode, click here.       BACK TO TOP   (Shared Infrastructure Only)   NEW - Always Free Autonomous Database now includes APEX 19.2 Oracle APEX 19.2 on Always Free Autonomous Database provides a preconfigured, fully managed and secured environment to both build and deploy world-class data-centric applications. Oracle APEX applications developed on-premise can be easily deployed to Oracle APEX on the free version Autonomous Database.     LEARN MORE   What is Application Express: click here. DOC: APEX 19c Getting Started Guide: click here. DOC: Creating Applications with Oracle Application Express in ADW: click here. DOC: Creating Applications with Oracle Application Express in ATP: click here.     DO MORE   Create a simple Human Resources (HR) application for the fictitious AnyCo Corp which manages departmental and employee information stored in database tables, click here. VIDEO: Learn how to create an APEX Application using a spreadsheet in just 2 minutes! click here. HOL: List of APEX hands-on labs: click here.       BACK TO TOP   (Shared Infrastructure Only) NEW - Multiple Database Versions and Availability by Region Depending on the region where you provision or clone your database, Autonomous Database now supports one or more Oracle Database versions. The version availability depends on the region - note that some regions do not support multiple Oracle Database versions. When multiple database versions are available, you choose an Oracle Database version when you provision or clone a database   LEARN MORE   DOC: Oracle Database versions and availability by region for ADW: click here. DOC: Oracle Database versions and availability by region for ATP: click here. If you are using ADB with Oracle Database 19c there are additional database features available in the database: DOC: ADW Oracle Database 19c features, click here. DOC: ATP Oracle Database 19c features, click here.     BACK TO TOP   (Shared Infrastructure Only) NEW - Availability of Oracle Graph Server and Client 20.1 Oracle Graph Server and Client 20.1 is a software package that works with Autonomous Database. It includes the in-memory analytics server (PGX) and client libraries required to work with the Property Graph feature in Autonomous Database. With graph analytics you can explore and discover connections and patterns in social networks, IoT, big data, data warehouses, and complex transaction data for applications such as fraud detection in banking, customer 360, and smart manufacturing.   LEARN MORE   Download is available on edelivery.oracle.com click here. DOC: Installation instructions click here. See section 1.7 Using Oracle Graph with the Autonomous Database. WEB: Graph Server page on oracle.com click here.     SEE MORE   PPT: Demystifying Graph Analytics for the Non-expert, click here PPT: Using Graph Analytics for New Insights, click here     BACK TO TOP   (Shared Infrastructure Only)   NEW - Features to Simplify Migrations To ADB 1. Database Resident Connection Pool (DRCP) Using DRCP provides you with access to a connection pool in your ADB that enables a significant reduction in key database resources required to support many client connections. See Use Database Resident Connection Pooling with Autonomous Database for more information. DOC: ADW click here. DOC: ATP click here.   2. Set MAX_STRING_SIZE value By default the Autonomous Data Warehouse database uses MAX_STRING_SIZE set to the value EXTENDED. To support migration from older Oracle Databases or applications you can set MAX_STRING_SIZE to the value STANDARD. See Checking and Setting MAX_STRING_SIZE for more information. DOC: ADW click here. DOC: ATP click here.   3. Number of concurrent statements The maximum number of concurrent statements is increased. See Predefined Database Service Names for Autonomous Data Warehouse for more information. DOC: ADW click here. DOC: ATP click here.       BACK TO TOP   (Shared Infrastructure Only) NEW - More Data Centers Now Online Autonomous Database is now available in the following data centers: Jeddah, Saudi Arabia Osaka, Japan Melbourne, Australia Amsterdam, Netherlands For more information on the status of services in each data center click here. For more information about how to subscribe to these new regions, click here. To switch to the new region, use the Region menu in the Console. See Switching Regions for more information, click here.     BACK TO TOP   EVENTS Oracle Global Leaders - Day With Development As part of our Global Leaders program, Oracle Product Management will be hosting a series of workshops for Architects, DBAs, Application Developers and for those interested in learning more about Oracle Information Management. Content will cover a general architecture overview, various cloud service capabilities (such as Autonomous Data Warehouse, Oracle Data Integration and Oracle Analytics Cloud) and how these services work together. All Services will be shown live so you will experience the look and feel of each product along with how to work with our complete range of data warehouse services. The schedule for this series of "Day With Development" events in the US is:   Feb 21       Chesterfield/St Louis   Feb 28       Vancouver   March 26        Irving   April 20       Nashville   April 21       Louisville   May 21       New York The schedule for this series of "Day With Development" events in the EMEA is:   March 3       Madrid, Spain   March 5       Roma, Italy   March 9 (Morning)       Dubai, United Arab Emirates   March 9 (Afternoon)       Dubai, United Arab Emirates   March 12       Istanbul, Turkey   March 16       Warszawa, Poland   March 18       Colombes cedex, France   March 23       Moscow, Russian Federation2 Please use the above links to register for an event in your area and get more information the general Terms and Conditions.     BACK TO TOP   oracle.com ADW ATP Documentation ADW ATP TCO Calculator Serverless Dedicated Cloud Cost Estimator Autonomous Database ADB New Features Autonomous Database Schema Advisor Autonomous Database       Customer Forums ADW ATP       BACK TO TOP  

  Autonomous DatabaseonShared ExadataInfrastructureFebruary 18, 2020       Welcome to our first customer newsletter for Autonomous Database on Shared Exadata Infrastructure. This first newsletter...

Newsletter

Announcing Our New Customer Newsletter for Autonomous Database - Don't Miss Out!

Welcome I am happy to announce that I will be posting a regular customer newsletter for Autonomous Database (on Shared Exadata Infrastructure) on this blog. It will cover the latest features added to Autonomous Database with links to additional collateral such as specific doc pages, videos, whitepapers, etc. Typically we are delivering new features every 3-4 weeks so this newsletter will appear on a regular basis throughout the year. How Do I Subscribe? If you want to subscribe to an RSS feed that will automatically refresh when each newsletter is posted then here are the steps... 1) Goto blogs.oracle.com/datawarehousing 2) Click on the three horizontal lines next to a magnifying glass in the top-right corner of the page to open the menu showing the list of categories for this blog 3) Now select the category "Newsletter" This will take you to the following page...https://blogs.oracle.com/datawarehousing/newsletter (yes, you are quite correct - I could have simply given you this link at the top of the page but this way you get to view the different categories available on the data warehouse blog so you can easily register for other areas). 4) Now click on the last icon on the right in the group of four...this is the RSS feed icon 5) If you just want the RSS feed to include the newsletters then select the second option "Only This Category's Posts" All done! You will probably get a pop-up to authorise access to your local application for viewing RSS feeds.  When Is The First Edition The first edition will be out very soon! Yes, it's free. Hope you enjoy our newsletter.  

Welcome I am happy to announce that I will be posting a regular customer newsletter for Autonomous Database (on Shared Exadata Infrastructure) on this blog. It will cover the latest features added to...

Newsletter

Announcing Our New Customer Newsletter for Autonomous Database - Don't Miss Out!

I am happy to announce that I will be posting a regular customer newsletter for Autonomous Database (on Shared Exadata Infrastructure) on this blog. It will cover the latest features added to Autonomous Database with links to additional collateral such as specific doc pages, videos, whitepapers, etc. Typically we are delivering new features every 3-4 weeks so this newsletter will appear on a regular basis throughout the year. How Do I Subscribe? If you want to subscribe to an RSS feed that will automatically refresh when each newsletter is posted then here are the steps... 1) Goto blogs.oracle.com/datawarehousing 2) Click on the three horizontal lines next to a magnifying glass in the top-right corner of the page to open the menu showing the list of categories for this blog 3) Now select the category "Newsletter" This will take you to the following page...https://blogs.oracle.com/datawarehousing/newsletter (yes, you are quite correct - I could have simply given you this link at the top of the page but this way you get to view the different categories available on the data warehouse blog so you can easily register for other areas). 4) Now click on the last icon on the right in the group of four...this is the RSS feed icon 5) If you just want the RSS feed to include the newsletters then select the second option "Only This Category's Posts" All done! You will probably get a pop-up to authorise access to your local application for viewing RSS feeds.  When Is The First Edition Out? The first editions out now, see here: https://blogs.oracle.com/datawarehousing/autonomous-database-newsletter-february-18-2020! Yes, it's free. Hope you enjoy our newsletter.  

I am happy to announce that I will be posting a regular customer newsletter for Autonomous Database (on Shared Exadata Infrastructure) on this blog. It will cover the latest features added to...

Autonomous

How to Send an Email using UTL_SMTP in Autonomous Database

Autonomous Database now supports UTL_HTTP and UTL_SMTP PL/SQL packages. You may already be familiar with these as they are commonly used in various scenarios. In this blog post, we will focus on how to send an email using the UTL_SMTP package. Before we dive into the details, it's worth noting that both packages are subject to certain restrictions. Even though we'll cover some of those here, you might still want to see PL/SQL Packages with Restrictions in the documentation. The UTL_SMTP package is designed for sending emails over Simple Mail Transfer Protocol (SMTP) and it provides numerous interfaces to the SMTP commands (See UTL_SMTP for more details). Thanks to these interfaces, it's in fact quite simple to send an email from within the database. However, as mentioned earlier, there are some restrictions that we need to be aware of. For example, the only supported email provider currently is Oracle Cloud Infrastructure (OCI) Email Delivery service. In other words, we need to have a working Email Delivery configuration before we can start sending emails on Autonomous Database (ADB). Let's start! Here's the list of steps that we are going to follow to successfully send an email on ADB: Configure Email Delivery Service Allow SMTP Access for ADMIN via an Access Control Entry (ACE) Create a PL/SQL Procedure to Send Email Send a Test Email Configure Email Delivery Service Oracle Cloud Infrastructure Email Delivery is an email sending service that provides a fast and reliable managed solution for sending high-volume emails (See Overview of the Email Delivery Service for more details). In this step, we are going to configure Email Delivery in OCI console as shown below: Generate SMTP credentials for a user. Open the navigation menu. Under Governance and Administration, go to Identity and click Users. Locate the user in the list that has permissions to manage email, and then click the user's name to view the details. Click SMTP Credentials. Click Generate SMTP Credentials. Enter a Description of the SMTP Credentials in the dialog box. Click Generate SMTP Credentials. A user name and password is displayed. Note: Whether you create a new user (See Adding Users) or choose to use an existing user for these steps, you need to make sure the user is assigned to a group with permissions to manage approved-senders and suppressions (See Set Up Permissions for more details). For example, our user in this example is assigned to a group that has the following policies for approved-senders and suppressions: Allow group <Your Group Name> to manage approved-senders in tenancy Allow group <Your Group Name> to manage suppressions in tenancy Create an approved sender for Email Delivery. We need to do this for all email addresses we use as the "From" with UTL_SMTP.MAIL (See Managing Approved Senders for more information). Open the navigation menu. Under Solutions and Platform, go to Email Delivery and click Email Approved Senders. Click Create Approved Sender within the Approved Senders view. Enter the email address you want to list as an approved sender in the Create Approved Sender dialog box. Click Create Approved Sender. The email address is added to your Approved Senders list. Allow SMTP Access for ADMIN via an Access Control Entry (ACE) Now we need to append an access control entry (ACE) using the DBMS_NETWORK_ACL_ADMIN package for ADMIN user to access SMTP for a specific host and port: Note: You can find your SMTP endpoint (host) and eligible ports by opening the navigation menu following Email Delivery --> Email Configuration. begin -- Allow SMTP access for user ADMIN dbms_network_acl_admin.append_host_ace( host =>'smtp.us-ashburn-1.oraclecloud.com', lower_port => 587, upper_port => 587, ace => xs$ace_type( privilege_list => xs$name_list('SMTP'), principal_name => 'ADMIN', principal_type => xs_acl.ptype_db)); end; / Create a PL/SQL Procedure to Send Email CREATE OR REPLACE PROCEDURE SEND_MAIL ( msg_to varchar2, msg_subject varchar2, msg_text varchar2 ) IS mail_conn utl_smtp.connection; username varchar2(1000):= 'ocid1.user.oc1.username'; passwd varchar2(50):= 'password'; msg_from varchar2(50) := 'adam@example.com'; mailhost VARCHAR2(50) := 'smtp.us-ashburn-1.oraclecloud.com'; BEGIN mail_conn := UTL_smtp.open_connection(mailhost, 587); utl_smtp.starttls(mail_conn); UTL_SMTP.AUTH(mail_conn, username, passwd, schemes => 'PLAIN'); utl_smtp.mail(mail_conn, msg_from); utl_smtp.rcpt(mail_conn, msg_to); UTL_smtp.open_data(mail_conn); UTL_SMTP.write_data(mail_conn, 'Date: ' || TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || UTL_TCP.crlf); UTL_SMTP.write_data(mail_conn, 'To: ' || msg_to || UTL_TCP.crlf); UTL_SMTP.write_data(mail_conn, 'From: ' || msg_from || UTL_TCP.crlf); UTL_SMTP.write_data(mail_conn, 'Subject: ' || msg_subject || UTL_TCP.crlf); UTL_SMTP.write_data(mail_conn, 'Reply-To: ' || msg_to || UTL_TCP.crlf || UTL_TCP.crlf); UTL_SMTP.write_data(mail_conn, msg_text || UTL_TCP.crlf || UTL_TCP.crlf); UTL_smtp.close_data(mail_conn); UTL_smtp.quit(mail_conn); EXCEPTION WHEN UTL_smtp.transient_error OR UTL_smtp.permanent_error THEN UTL_smtp.quit(mail_conn); dbms_output.put_line(sqlerrm); WHEN OTHERS THEN UTL_smtp.quit(mail_conn); dbms_output.put_line(sqlerrm); END; / Notes: username: Specifies the SMTP credential username. passwd: Specifies the SMTP credential password. msg_from: Specifies one of the approved senders. mailhost: Specifies the SMTP connection endpoint. Send a Test Email In order to verify that we configured everything accurately and created a working procedure (SEND_MAIL), we will send a test email: execute send_mail('taylor@example.com', 'Email from Oracle Autonomous Database', 'Sent using UTL_SMTP'); To summarize, UTL_SMTP package is now supported in Autonomous Database and we just explored how to send an email by taking advantage of this package the OCI Email Delivery service. If you'd like to learn more about Email Delivery or UTL_SMTP in ADB, please follow the documentation links referenced above.  

Autonomous Database now supports UTL_HTTP and UTL_SMTP PL/SQL packages. You may already be familiar with these as they are commonly used in various scenarios. In this blog post, we will focus on how...

How to Launch a Virtual Cloud Network Using the Networking Quickstart Wizard and Connect to Your Autonomous Database

A virtual cloud network (VCN) is a virtual, private network you set up in Oracle data centers. It is very similar to a traditional network with firewall rules and various gateways. When you work with Oracle Cloud Infrastructure (OCI), setting up a VCN for your cloud resources is usually one of the first things that you might be doing. However, configuring a VCN can be a bit involving when you think about all the sub-components that need attention such as subnets, route tables, gateways, security lists, etc. (See OCI Networking documentation for more details). The good news is that you can now launch a VCN with connectivity to the internet and Oracle Services Network in just a couple steps thanks to the new OCI Networking Quickstart wizard.  The wizard basically creates a VCN with regional public and private subnets, a NAT gateway, service gateway and an internet gateway along with the necessary route table and security list rules (including SSH access). It only prompts you to specify the IP CIDR block for the VCN and subnets. This reduces the number of steps and amount of time it takes to setup your network to 1-2 minutes. In this blog post, we are going to explore how to take advantage of this wizard to quickly launch a VCN as well as creating a compute instance in this VCN to connect to our Autonomous Database. Here's the outline of the steps that we are going to follow: Create a VCN Using the Networking Quickstart Wizard Provision a Compute Instance (Virtual Machine) Connect to our ADW Instance Create a VCN Using the Networking Quickstart Wizard We have two options to launch the wizard:  From Oracle Cloud Console home page under 'Quick Actions': In the navigation menu, follow 'Networking' --> 'Virtual Cloud Networks': In the wizard dialog, we will select 'VCN with Internet Connectivity' and click 'Start Workflow': In the next page, we will enter the VCN name and specify the compartment, VCN and subnet CIDR blocks: We will click 'Next' to review our configuration: As the final step, we'll hit 'Create' and watch all the components being configured: Provision a Compute Instance (Virtual Machine) In the previous step, we have seen how easy it is to launch a VCN in just a couple minutes. As you may remember, our end goal is to access our ADW instance from within that VCN and we are almost there! Now, all we have to do is to provision a virtual machine (VM) in the VCN that we just created. Our VM can either be on a public subnet or a private subnet. A VM on a public subnet has the option to have a public IP address; on the other hand, a VM on a private subnet only has a private IP address that can be accessed within the same VCN. Just to demonstrate how we can access to our ADW instance from both public and private subnets, we will create two VMs as shown below. On public subnet: On private subnet: Connect to our ADW Instance So far we launched our VCN and created two VMs in it. At this point it's important to remember that one of our VMs (ctuzla-vcnpublic) is on a public subnet while the other VM (ctuzla-vcnprivate) is on a private subnet, meaning it doesn't have an assigned public IP address and cannot be accessed via the internet. Connecting to ADW from our VM on the public subnet is fairly easy. After copying our ADW wallet into our VM, we can just SSH into the VM using the public IP address and connect to the ADW instance in SQL Plus. In order to connect to ADW from our VM on the private subnet, we need to first connect to the VM itself and it requires couple additional steps. Since ctuzla-vcnprivate is on a private subnet, we will first connect to ctuzla-vcnpublic and ssh into ctuzla-vcnprivate using its private IP address (please note that we need to copy the private SSH key of ctuzla-vcnprivate into ctuzla-vcnpublic to be able to do this). Let's see all these steps in action (The steps below assume that we already have Oracle Instant Client set up and our ADW wallet available in both VMs): Connection from a public subnet (ctuzla-vcnpublic): ctuzla-mac$ ssh -i /Users/ctuzla/id_rsa opc@129.143.200.127 Last login: Wed Dec 11 18:43:03 2019 [opc@ctuzla-vcnpublic ~]$ [opc@ctuzla-vcnpublic instantclient_19_5]$ ./sqlplus ADMIN/************@adw_high SQL*Plus: Release 19.0.0.0.0 - Production on Wed Dec 11 19:51:33 2019 Version 19.5.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Connected to: Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production Version 18.4.0.0.0 SQL> select * from dual; D - X Connection from a private subnet (ctuzla-vcnprivate): Please note that we will first connect to ctuzla-vcnpublic then jump into ctuzla-vcnprivate ctuzla-mac$ ssh -i /Users/ctuzla/id_rsa opc@129.143.200.127 Last login: Wed Dec 11 19:38:41 2019 [opc@ctuzla-vcnpublic ~]$ [opc@ctuzla-vcnpublic ~]$ ssh -i /home/opc/id_rsa opc@10.0.1.3 Last login: Wed Dec 11 19:33:30 2019 [opc@ctuzla-vcnprivate ~]$ [opc@ctuzla-vcnprivate instantclient_19_5]$ ./sqlplus ADMIN/************@adw_high SQL*Plus: Release 19.0.0.0.0 - Production on Wed Dec 11 20:13:40 2019 Version 19.5.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Connected to: Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production Version 18.4.0.0.0 SQL> select * from dual; D - X In this blog post, we have explored how to launch a VCN using the Networking Quickstart wizard, create a compute instance (on both public and private subnet), and connect to an ADW instance from within this VCN. As we have seen, the wizard turns the VCN creation into a much simpler and quicker process. If you would like to learn more about the new Networking Quickstart wizard, please check out the documentation here.

A virtual cloud network (VCN) is a virtual, private network you set up in Oracle data centers. It is very similar to a traditional network with firewall rules and various gateways. When you work with...

Announcing: Big Data Appliance X8-2

Big Data Appliance X8-2 is the 7th hardware generation of Oracle's leading Big Data platform continuing the platform evolution from Hadoop workloads to Big Data, SQL, Analytics and Machine Learning workloads. Big Data Appliance combines dense IO with dense Compute in a single server form factor. The single form factor enables our customers to build a single data lake, rather then replicating data across more specialized lakes.  What is New? The current X8-2 generation is based on the latest Oracle Sun X8-2L servers, and leverages that infrastructure to deliver enterprise class hardware for big data workloads. The latest generation sports more cores, more disk space and the same level of memory per server. Big Data Appliance retains its InfiniBand internal network, support by a multi-homed Cloudera CDH cluster set up.  Why a Single Form Factor? Many customers are embarking on a data unification effort, and the main data management concept used in that effort is the data lake. Within this data lake, we see and recommend a set of workloads to be run as is shown in this logical architecture:   In essence what we are saying is that the data lake will host the Innovation or Discovery Lab workloads as well as the Execution or production workloads on the same systems. This means that we need an infrastructure to both deal with large data volumes in a cost effective manner and deal with high compute volumes on a regular basis. Leveraging the hardware footprint in BDA, enables us to run both these workloads. The servers come with 2 * 24 cores AND 12 * 14TB drives enabling very large volumes of data and CPUs spread across a number of workloads. So rather then dealing with various form factors, and copying data from the main data lake to a side show Discovery Lab, BDA X8-2 consolidates these workloads. The other increasingly important data set in the data lake is streaming into the organization, typically via Apache Kafka. Both the CPU counts and the memory footprints can provide a great Kafka cluster, connecting it over InfiniBand to the main HDFS data stores. Again, while these nodes are very IO dense for Kafka, the simplicity of using the same nodes for any of the workloads makes Big Data Appliance a great Big Data platform choice. What is in the Box? Apart from the hardware specs, the software that is included in Big Data Appliance enables the data lake creation in a single software & hardware combination. Big Data Appliance comes with the full Cloudera stack, enabling the data lake as drawn above, with Kafka, HDFS, Spark all included in the cost of the system. The specific licensing for Big Data Appliance makes the implementation cost effective, and added to the simplicity of a single form factor makes Big Data Appliance an ideal platform to implement and grow the data lake into a successful venture. Where I can do test and development for BDA? Quite frequently our customers need to run some tests once and don't need to retain environment for longer. Another use case is customizable environment, when customers need to try some Hadoop reconfiguration, but risk of mistake is quite high and it's challenging to redeploy all BDA environment in case of fail. Cloud solutions will solve these problems and our team is working hard to release soon Big Data Service, which can help our customers to setup hybrid environments (with BDA on the ground and Big Data Service in the cloud)

Big Data Appliance X8-2 is the 7th hardware generation of Oracle's leading Big Data platform continuing the platform evolution from Hadoop workloads to Big Data, SQL, Analytics and Machine Learning...

Autonomous

Is your data in parts and lying in other object stores? No worries, Autonomous Database has you covered!

Whether you personally believe in the multi-cloud lifestyle that’s in vogue or not, the reality is you may be dealing with different departments or orgs within your company that store their data in their cloud of choice, and partitioned in various ways. Autonomous DB doesn't judge, we support querying or ingesting your data whether it lies in Oracle Cloud Object Store, Azure, or AWS!  Let me walk you through a simple example of how you may query your multi-part data lying in any of these object stores. We will be using the new partitioned external table functionality in the DBMS_CLOUD package, which creates a partitioned table structure over flat files lying in your external storage. (ie. a query on this table fetches data directly from the flat files in external storage, instead of the database's own datafiles). You may use the other functions in this package if you need to apply this to a non-partitioned external table, or to actually copy data into your database. Let's begin with some cloud concepts we will be using further in the post: In general, an Object Store is a cloud-based, scalable, serverless storage platform that offers storage of “objects”. These objects can be unstructured data of any content type, including analytic data and rich content, like images and videos. Oracle’s Object Storage, AWS S3 and Azure Blob Storage are all examples of object stores. A private object store URL is a URL that requires authentication via a credential to be accessed. Pre-Authenticated URLs and Public URLs don’t require any credentials to be accessed. Having the complete URL will give someone access to that data. Next, download the 2 part file in a zip here. In keeping with some of my previous posts, we will use the small weather history data set from Charlotte, NC in Comma Separated Values (CSV) format. The file which has "Part 1" in its name has July 2014 weather data, and that with "Part 2" will be the following month’s, Aug 2014 weather data.   Now select the cloud object store that you will be using to jump to the appropriate section:               .                 Using Oracle Cloud Object Storage   Step 1: Unzip and upload the two downloaded files to your Oracle Cloud Object Storage bucket Create an Object Store bucket and unzip and upload the two Weather history files to your object store. You can refer to this hands-on lab for a detailed walkthrough of how to do this in your Oracle Cloud tenancy.     Keep note of the "Visibility" (you can use Private or Public) of the bucket to which you are uploading the files. This will come in handy in Step 3.   Step 2: Create a credential to access your object store for your Private URLs Dive into SQL Developer that is connected to your Autonomous Database, and create a credential for your object store like below. The username is your Oracle Cloud Infrastructure username. The password is your Oracle Cloud Infrastructure generated auth token: BEGIN   DBMS_CLOUD.CREATE_CREDENTIAL (     credential_name => 'OBJ_STORE_CRED',     username => '<OCI username>',     password => '<OCI auth token>'   ); END; / Note: If you are unsure about how to create an auth token, refer to Step 8 in in the same lab tutorial mentioned above. While we have used a username and password for simplicity here, we recommend that you use native authentication for your production systems added security and versatility.   Step 3: Create a Partitioned External Table on top of your two-part Weather History data Now that we have a credential ready, run the PL/SQL script below using your Object Store URLs in the location parameters. You may either construct the URLs yourself by looking at this URL format or find the URL from the details of your file in the object store console as in the following screens.       Since we have a month's worth of data in each file, we call the DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE function to create the partitioned external table with 2 partitions using the REPORT_DATE column. Note: We are using a credential parameter in this call to DBMS_CLOUD to illustrate the use of files that lie in a private object storage bucket (ie. that have private URLs). If we use public bucket or pre-authenticated URLs, we can omit the credential parameter entirely. We can also use a mix of private and public URLs from the same object store if required.   BEGIN  DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(     table_name =>'WEATHER_REPORT_EXTERNAL_PART',     credential_name =>'OBJ_STORE_CRED',     format => json_object('type' value 'csv', 'skipheaders' value '1',    'dateformat' value 'mm/dd/yy'),   column_list => 'REPORT_DATE DATE,   ACTUAL_MEAN_TEMP NUMBER,   ACTUAL_MIN_TEMP NUMBER,                      ACTUAL_MAX_TEMP NUMBER,   AVERAGE_MIN_TEMP NUMBER,                      AVERAGE_MAX_TEMP NUMBER,   AVERAGE_PRECIPITATION NUMBER',   partitioning_clause =>     'partition by range (REPORT_DATE)           (partition p1 values less than (to_date(''01-AUG-2014'',''DD-MON-YYYY'')) location               ( ''https://objectstorage.us-ashburn-1.oraclecloud.com/                   n/adwctraining5/b/partitiontest/o/Charlotte_NC_Weather_History_Part1.csv'')            ,            partition p2 values less than (to_date(''01-SEP-2014'',''DD-MON-YYYY'')) location               ( ''https://objectstorage.us-ashburn-1.oraclecloud.com/                   n/adwctraining5/b/partitiontest/o/Charlotte_NC_Weather_History_Part2.csv'')            )'   ); END; /   ↓ Click here to Jump to Step 4     |    Click here to Jump to object store selection ↑   Using Microsoft Azure Blob Storage   Step 1: Unzip and upload the two downloaded files to your Azure Blob Storage container Navigate to your Azure Blob Storage account, create a container and unzip and upload the two Weather history files. (Refer to these detailed steps on how to do this if necessary).     Keep note of the "Access Level" of your container (you can use a Private or Public container) in which you uploaded the files. This will come in handy in Step 3.   Step 2: Create a credential to access your Blob storage for your Private URLs Dive into SQL Developer that is connected to your Autonomous Database, and create a credential for your blob store like below. The username is your Azure Storage account name. The password is your Azure storage account access key: BEGIN   DBMS_CLOUD.CREATE_CREDENTIAL (     credential_name => 'OBJ_STORE_CRED',     username => '<Azure Storage account name>',     password => '<Azure Storage account access key>'   ); END; / Note: If you are still unsure about how to find your Azure storage account access key, follow detailed steps here.   Step 3: Create a Partitioned External Table on top of your two-part Weather History data Now that we have a credential ready, run the PL/SQL script below using your Azure Blob store URLs in the location parameters. You may either construct the URLs yourself by looking at the URL format or find the URL from the details of your file in the blob store console as in the following screen:     Since we have a month's worth of data in each file, we call the DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE function to create the partitioned external table with 2 partitions using the REPORT_DATE column. Note:  We are using a credential parameter in this call to DBMS_CLOUD to illustrate the use of files that lie in a private Azure blob storage container (ie. that have private URLs). If we use a public access container, we can omit the credential parameter entirely. We can also use a mix of private and public URLs from the same object store if required. We don't currently support pre-authenticated URLs for Azure, that's coming soon. Feel free to comment below if your business relies heavily on this.   BEGIN  DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(     table_name =>'WEATHER_REPORT_EXTERNAL_PART',     credential_name =>'OBJ_STORE_CRED',     format => json_object('type' value 'csv', 'skipheaders' value '1',    'dateformat' value 'mm/dd/yy'),   column_list => 'REPORT_DATE DATE,   ACTUAL_MEAN_TEMP NUMBER,   ACTUAL_MIN_TEMP NUMBER,                      ACTUAL_MAX_TEMP NUMBER,   AVERAGE_MIN_TEMP NUMBER,                      AVERAGE_MAX_TEMP NUMBER,   AVERAGE_PRECIPITATION NUMBER',   partitioning_clause =>     'partition by range (REPORT_DATE)           (partition p1 values less than (to_date(''01-AUG-2014'',''DD-MON-YYYY'')) location               ( ''https://nilaysobjectstore.blob.core.windows.net/                   externaltest/Charlotte_NC_Weather_History_Part1.csv'')            ,            partition p2 values less than (to_date(''01-SEP-2014'',''DD-MON-YYYY'')) location               ( ''https://nilaysobjectstore.blob.core.windows.net/                   externaltest/Charlotte_NC_Weather_History_Part2.csv'')            )'   ); END; /   ↓ Click here to Jump to Step 4     |    Click here to Jump to object store selection ↑   Using AWS S3 Storage   Step 1: Unzip and upload the two downloaded files to your AWS S3 Storage container Navigate to your AWS S3 storage account, create a bucket and unzip and upload the two Weather history files. (Refer to these detailed steps on how to do this if necessary).     Keep note of the Permissions of your bucket (you can use a Private or Public access) in which you uploaded the files. This will come in handy in Step 3.   Step 2: Create a credential to access your AWS S3 storage for your Private URLs Dive into SQL Developer that is connected to your Autonomous Database, and create a credential for your S3 store like below. The username is your AWS Access key ID. The password is your AWS secret access key: BEGIN   DBMS_CLOUD.CREATE_CREDENTIAL (     credential_name => 'OBJ_STORE_CRED',     username => '<AWS Access key ID>',     password => '<AWS secret access key>'   ); END; / Note: If you are still unsure about how to find your AWS access key ID and secret access key, follow detailed steps here.   Step 3: Create a Partitioned External Table on top of your two-part Weather History data Now that we have a credential ready, run the PL/SQL script below using your AWS S3 URLs in the location parameters. You may either construct the URLs yourself by looking at the URL format or find the URL from the details of your file in the S3 bucket console as in the following screen:     Since we have a month's worth of data in each file, we call the DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE function to create the partitioned external table with 2 partitions using the REPORT_DATE column. Note:  We are using a credential parameter in this call to DBMS_CLOUD to illustrate the use of files that lie in a private AWS S3 storage bucket (ie. that have private URLs). If we use a public access bucket, we can omit the credential parameter entirely. We can also use a mix of private and public URLs from the same object store if required. We don't currently support pre-authenticated URLs for AWS, but that's coming soon. Feel free to comment below if your business relies heavily on this. BEGIN  DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(     table_name =>'WEATHER_REPORT_EXTERNAL_PART',     credential_name =>'OBJ_STORE_CRED',     format => json_object('type' value 'csv', 'skipheaders' value '1',    'dateformat' value 'mm/dd/yy'),   column_list => 'REPORT_DATE DATE,   ACTUAL_MEAN_TEMP NUMBER,   ACTUAL_MIN_TEMP NUMBER,                      ACTUAL_MAX_TEMP NUMBER,   AVERAGE_MIN_TEMP NUMBER,                      AVERAGE_MAX_TEMP NUMBER,   AVERAGE_PRECIPITATION NUMBER',   partitioning_clause =>     'partition by range (REPORT_DATE)           (partition p1 values less than (to_date(''01-AUG-2014'',''DD-MON-YYYY'')) location              (''https://nilaystests.s3-us-west-1.amazonaws.com/Charlotte_NC_Weather_History_Part1.csv'')            ,            partition p2 values less than (to_date(''01-SEP-2014'',''DD-MON-YYYY'')) location              (''https://nilaystests.s3-us-west-1.amazonaws.com/Charlotte_NC_Weather_History_Part2.csv'')            )'   ); END; / Step 4: Validate and Query your data We're done already! You may run the "DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE" function to validate that your partitioned external table was created correctly and can access its underlying data. (Validation is an optional but recommended step)   EXEC DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE('WEATHER_REPORT_EXTERNAL_PART'); If things went awry and you receive an error, you may look at the logfile and badfile to easily troubleshoot any errors you see. (Refer to this lab tutorial's Step 12 for more detail on this). If you sailed through with no errors, you may now query the table as a whole or by partition (ie. in this case, by month) with the script below. Notice that in the third query we select data between July 5th and 21st 2014, all of which lies in partition P1. This query therefore makes use of the performance benefits of partitioned external tables, and only needs to physically access the file named "Part 1" file under the hood!   SELECT REPORT_DATE, AVERAGE_PRECIPITATION FROM WEATHER_REPORT_EXTERNAL_PART; SELECT REPORT_DATE, AVERAGE_MIN_TEMP FROM WEATHER_REPORT_EXTERNAL_PART PARTITION (P2); SELECT * FROM WEATHER_REPORT_EXTERNAL_PART where REPORT_DATE between '05-JUL-2014' and '21-JUL-2014';     So, there you have it! Autonomous Database aims to not only simplify the provisioning and management of your database in the cloud, but also aims to provide easy to use functions to make your entire interaction with the database seamless and intuitive.   ↑ Click here to jump back to object store selection to try a different one

Whether you personally believe in the multi-cloud lifestyle that’s in vogue or not, the reality is you may be dealing with different departments or orgs within your company that store their data in...

Autonomous

Keeping Your Autonomous Data Warehouse Secure with Data Safe - Part 2

In the part 1 of this series we looked at how to get your ADW and OCI environment ready for using Data Safe. If you missed part 1 or need a quick refresher then the blog post is here: https://blogs.oracle.com/datawarehousing/keeping-your-autonomous-data-warehouse-secure-with-data-safe-part-1. In this post we are going to explore the process of connecting and Autonomous Data Warehouse instance to our newly deployed Data Safe environment. Remember that you deploy your Data Safe control center within a specific OCI regional data center - as is the case with all our other cloud services. Therefore, if you switch to a different data center then you will need to deploy a new Data Safe environment. Hope that makes sense! Launching the Data Safe Service Console In part 1 we got to the point of enabling Data Safe in the Frankfurt data center. Now when we login to Oracle Cloud using our newly created OCI credentials we can pop open the hamburger menu and select Data Safe: and arrive on the Data Safe landing pad. The next step is to launch the Service Console (you may be wondering...why doesn't the Service Console just open automatically since the landing pad page is empty, apart from the Service Console button! Great question and we will come back to this towards the end of the series of posts when the landing pad page will show a lot more information). After clicking on the Service Console button a new window pops open which looks like this:   Right now, there is no information showing on any of our graphs or any of the other pages. This is because we have not registered our data warehouse instance so that's the next step. Registering an ADW with Data Safe We need to register our ADW with Data Safe before we can generate any of the reports that are part of the Data Safe library. To do that we need to go to the tab marked "Target" in the horizontal menu at the top of the page: Clicking on the "Register" button will pop open a form where we can input the connection details for our ADW... Data Safe is not limited to just working with Autonomous Databases and we could register any of the following: Nnote that Data Safe supports only serverless deployments for Autonomous Database. "Dedicated" is not currently supported. There is more information here: https://docs.oracle.com/en/cloud/paas/data-safe/udscs/supported-target-databases.html For ADW (and ATP) we first need to change the connection type to TLS which will add some additional fields to the form - this will be most recognisable if you have been spent time configuring connections to ADW from DI/ETL or BI tools: it looks as if a lot of information is now required to register our ADW instance but the good news is that just about all the information we need is contained within a small zip file which we can download from our OCI ADB console. Essentially we need the wallet file for our instance. But first let's quickly complete the fields in the top part of the form:   Ok, now we need the information about our ADW instance and here's how you get it: Collecting Connection Information For ADW If we flip over to the OCI console page for our ADW instance we can see that there is a line for something called "OCID" which is the first piece of information we need to collect .There are two links next to it: "Show" and "Copy". Click on copy and then flip over to our Data Safe page and paste in the OCID reference. Now we need things like hostname, port, service name and target distinguished name along with various secure wallet files. To get this information we need to download the wallet file which can be accessed by clicking on the "DB Connection" button. On the pop-up form click the "Download Wallet" button and enter a password...note this down because we are going to need it again shortly... Once the file has been downloaded, find the file on your filesystem and unzip it. The result will be a folder containing the following files: Ok, back to our Target registration form on Data Safe....the data for the next four fields can all be found in the tnsnames.ora file. We are going to use the "low service" for this connection because running Data Safe reports is not an urgent, rush-rush, workload. If you have no idea what a "low service" is then it might be a good idea to quickly read through the section on "Managing Concurrency and Priorities on Autonomous Data Warehouse" in section 12 of the ADW documentation. In simple terms...when we connect to an ADW instance we need to select a service (low, medium or high). These services map to LOW, MEDIUM, and HIGH consumer groups which have the following characteristics: HIGH: Highest resources, lowest concurrency. Queries run in parallel. MEDIUM: Fewer resources, higher concurrency. Queries run in parallel. LOW: Least resources, highest concurrency. Queries run serially. Anyway....as long as the jobs run, then we are going to be happy. Therefore, we need to find the details for our low-service connection in the tnsnames.ora file which will look something like this: adwdemo_low = (description=(address=(protocol=tcps)(port=1522)(host=xxxxxx.oraclecloud.com))(connect_data=(service_name=xxxxx_adwdemo_low.xxxxxxx))(security=(ssl_server_cert_dn="CN=xxxxxx.oraclecloud.com,OU=Oracle,O=Oracle Corporation,L=Redwood City,ST=California,C=US"))) You can copy & paste the host, port, service_name and ssl_server_cert_dn into the four fields below the "TLS" pulldown menu entry. So now our form looks like this... now the last few steps....make sure the wallet type is set to "JKS Wallet". For the Certificate/Wallet find the "truststore.jks" file from our downloaded and unzipped connection file. In the same directory/folder we can pick "keystone.jks" for the "Keystore Wallet". The next field needs the password we used on the OCI ADW console page when we downloaded the connection zip file so paste that in... Lastly add the ADW instance username/password that we created in Part 1of this series of blog posts - our user was called DATASAFE.   before you click on the "Test Connection" button we need to run a PL/SQL script to give our new DATASAFE database user some privileges that will allow Data Safe to run though it's library of checks... Click on the download button then search for the PL/SQL script dscs_privileges.sql. Using SQL Developer (or any other tool) we need to login as our standard ADMIN user and run that script (copy & paste will do the trick). Check the log for the script and you should see something like this: Enter value for USERNAME (case sensitive matching the username from dba_users) Setting USERNAME to DATASAFE Enter value for TYPE (grant/revoke) Setting TYPE to GRANT Enter value for MODE (audit_collection/audit_setting/data_discovery/masking/assessment/all) Setting MODE to ALL Granting AUDIT_COLLECTION privileges to "DATASAFE" ...  Granting AUDIT_SETTING privileges to "DATASAFE" ...  Granting DATA_DISCOVERY role to "DATASAFE" ...  Granting MASKING role to "DATASAFE" ...  Granting ASSESSMENT role to "DATASAFE" ...  Done. Disconnected from Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production Version 18.4.0.0.0 NOW, we can test the connection... and everything should work and we should see a nice big green tick.    Finally we are ready to click on "Register Target". Now our target page shows the details of our newly registered database...     Of course that's all we have done - register our ADW as a new target - so all the other pages are still empty, including the home page:     Wrap up for Part 2 In part 1 we setup our environment ready for using Data Safe and we enabled Data Safe within our regional data center - in this case the Frankfurt Data Center. In this post, Part 2, we have successfully added our existing ADW instance as a new target database in Data Safe.   Coming up in Part 3 In the next post we will start to explore some of the data discovery and data masking features that are part of Data Safe.   Learn More... Our security team has created a lot of great content to help you learn more about Data Safe so here are my personal bookmarks: Documentation - start here: https://docs.oracle.com/en/cloud/paas/data-safe/udscs/oracle-data-safe-overview.html Data Safe page on Oracle.com - https://www.oracle.com/database/technologies/security/data-safe.html Database Security Blog: https://blogs.oracle.com/cloudsecurity/db-sec https://blogs.oracle.com/cloudsecurity/keep-your-data-safe-with-oracle-autonomous-database-today https://blogs.oracle.com/cloudsecurity/keeping-your-data-safe-part-4-auditing-your-cloud-databases        

In the part 1 of this series we looked at how to get your ADW and OCI environment ready for using Data Safe. If you missed part 1 or need a quick refresher then the blog post is here: https://blogs.ora...

Autonomous

Keeping Your Autonomous Data Warehouse Secure with Data Safe - Part 1

One of the big announcements at OpenWorld 2019 in San Francisco was Oracle Data Safe - a totally new cloud-based security control center for your Oracle Data Warehouse (ADW) and it's completely FREE!    So what exactly does it do? Well, in summary Data Safe delivers essential data security capabilities as a service on Oracle Cloud Infrastructure. Essentially, it helps you understand the sensitivity of your data, evaluate risks to data, mask sensitive data, implement and monitor security controls, assess user security, monitor user activity, and address data security compliance requirements.   Maybe a little video will help...     Data Safe Console The main console dashboard page for Data Safe looks like this:  giving you a fantastic window directly into the types of data sets sitting inside Autonomous Data Warehouse. it means you can... Assess if your database is securely configured Review and mitigate risks based on GDPR Articles/Recitals, Oracle Database STIG Rules, and CIS Benchmark recommendations Assess user risk by highlighting critical users, roles and privileges Configure audit policies and collect user activity to identify unusual behavior Discover sensitive data and understand where it is located Remove risk from non-production data sets by masking sensitive data So let's see how easy it is to connect an existing Autonomous Data Warehouse to Data Safe and learn about the types of security reviews you can run on your data sets... Getting ADW ready to work with Data Safe To make this more useful to everyone I am going to take an existing ADW instance and create a new user called LOCAL_SH. Then I am going to copy the supplementary demographics, countries and customer tables from the read-only sales history demo schema to my new local_sh schema. This will give me some "sensitive" data points for Data Safe to discover when I connect my ADW to Data Safe. CREATE USER local_sh IDENTIFIED BY "Welcome1!Welcome1"; GRANT DWROLE TO local_sh; CREATE TABLE local_sh.supplementary_demographics AS SELECT * FROM sh.supplementary_demographics; CREATE TABLE local_sh.customers AS SELECT * FROM sh.customers; CREATE TABLE local_sh.countries AS SELECT * FROM sh.countries; So what does the customer table look like: and as you can see that there are definitely some columns that would help to personally identify someone. Those types of columns need to be hidden or masked from our development teams and business users...   Now we know that we have some very sensitive data! If you are not quite as lucky as me and you are starting from a completely clean ADW and need to load some of your data then checkout the steps in our documentation guide that explains how to load your own data into ADW: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/tasks_load_data.html   Next step is to create a user within my ADW instance that I will use for in my Data Safe connection process so that my security review process is not tied to one of my application users. CREATE USER datasafe IDENTIFIED BY "Welcome1!DataSafe1"; GRANT DWROLE TO datasafe; What you are going to see later is that we have to run an installation script, which is available from the Data Safe console when we register a database. This user is going to own the required roles and privileges for running Data Safe, which is why I don't want to tie it to one of my existing application users.   Do I have the right type of Cloud Account? If you are new to Oracle Cloud or maybe created your account within the last 12 months then you can probably skip this section. For some of you who have had cloud accounts on the Oracle Cloud for over two years then when you try to access Data Safe you may get an interesting warning message about federated accounts... Accessing Data Safe After you log into your cloud account click on the hamburger (three horizontal lines) menu in the top left corner. The pop-out menu will have Data Safe listed underneath Autonomous Transaction Processing...   Click on Data Safe and this screen might appear! If this message doesn't appear then jump ahead a paragraphs to "Enabling Data Safe".   Don't panic all is not lost at this point. All you need to do is create a new OCI user. In the same hamburger menu list scroll down to the section for "Governance and Administration", select "Identity" and then select "Users"... Click on the big blue "Create User" button at the top of the screen and then fill in the boxes to create a new OCI user which we will use as the owner of your Data Safe environment. After you create the new user a welcome email should arrive with a link to reset the password...something similar to this: Having created a completely new user it's important to enable all the correct permissions so that Data Safe can access the resources and autonomous database instances within your tenancy. For my user, called DataSafe, I already have an “administrators” group within my tenancy that contains all the require privileges needed for Data Safe.  In the real world it's probably prudent to setup a new group just for Data Safe administrators and then assign OCI privileges to just that group. There is more information about this process here: https://docs.oracle.com/en/cloud/paas/data-safe/udscs/required-permission-enabling-oracle-data-safe.html Quick Recap We have an existing Autonomous Data Warehouse instance setup. I have used SQL Developer to create a new user to own a small data set containing potentially sensitive information, copied some tables from an existing schema that I know has sensitive data points so I now have a working data set and we have setup a new OCI user to own our Data Safe deployment.   Enabling Data Safe Now we are ready to start working with Data Safe. From the hamburger menu select "Data Safe" and you should see this screen if it's the first time you used it in your tenancy-region. You can see below that I am working in our Frankfurt Data Center and this is the first time I have logged into Data Safe: so to get to the next stage all we need to do is click on the big blue button to enable Data Safe. At which point we get the usual "working..." screen   followed by the "ready to get to work" screen...   Wrap-up for Part 1 This is a wrap for this particular post. In the next instalment we will look at how to register an Autonomous Data Warehouse instance and then run some of the security reports that can help us track down those objects that contain sensitive data about our customers. Learn More... Our security team has created a lot of great content to help you learn more about Data Safe so here are my personal bookmarks: Documentation - start here: https://docs.oracle.com/en/cloud/paas/data-safe/udscs/oracle-data-safe-overview.html Data Safe page on Oracle.com - https://www.oracle.com/database/technologies/security/data-safe.html Database Security Blog: https://blogs.oracle.com/cloudsecurity/db-sec https://blogs.oracle.com/cloudsecurity/keep-your-data-safe-with-oracle-autonomous-database-today https://blogs.oracle.com/cloudsecurity/keeping-your-data-safe-part-4-auditing-your-cloud-databases        

One of the big announcements at OpenWorld 2019 in San Francisco was Oracle Data Safe - a totally new cloud-based security control center for your Oracle Data Warehouse (ADW) and it's completely FREE!   ...

Autonomous

Key Highlights from an Autonomous OpenWorld 2019

It's taken longer than expected (too many things to do post-OpenWorld!) but I have finally finished and published my review of OpenWorld 2019 and it is now available in the Apple iBooks store.   Why do you need this book? Well as usual there were so many sessions and hands-on labs at this year's conference it really is hard to know where to start when you click on the link (https://events.rainfocus.com/widget/oracle/oow19/catalogow19?) to access the content catalog guide. There are some filters that help you narrow down the huge list but it can take time to search and download all the most important content linked to Autonomous Database. To save you all that time searching, I have put together a complete review in beautiful iBook format! And if you didn't manage to get to San Francisco then here is the perfect way to learn about the key messages, announcements, roadmaps, features and hands-on training that will help you get the most from your Autonomous Database experience.   So what's in the book? The guide includes all the key sessions, labs and key announcements from this year's Oracle OpenWorld conference broken down into the following sections Chapter 1 - Welcome and key video highlights  Chapter 2 -  List of key sessions, labs and videos with links to download the related presentations Chapter 3 - Details of all the links you need to keep up to date on Oracle’s strategy and products for Data Warehousing and Big Data. This covers all our websites, blogs and social media pages. Chapter 4 - Everything you need to justify being at OpenWorld 2020 Where can I get it? If you have an Apple device then you will definitely want to get the iBook version which is available here: http://books.apple.com/us/book/id1483761470 If you are still stuck on Windows or stuck using an Android device then the PDF version is the way to go. The download link for this version is here: https://www.dropbox.com/s/7ejlmhmwhwpbdpw/ADB-Review-oow19.pdf?dl=0 Feedback If you think anything is missing then let me know via this blog by leaving a comment or send me an email (keith.laker@oracle.com). Hope you find the review useful and look forward to seeing you all at Moscone Center next year for OpenWorld 2020.  

It's taken longer than expected (too many things to do post-OpenWorld!) but I have finally finished and published my review of OpenWorld 2019 and it is now available in the Apple iBooks store.   Why do...

Autonomous

Getting A Sneak Preview into the Autonomous World Of 19c

If you have been autonomous data warehouse for a while now you will know that when you create a new data warehouse your instance will be based on Database 18c. But you also probably spotted that Database 19c is also available via some of our other cloud services and LiveSQL also runs Database 19c. Which probably makes you wonder when Autonomous Data Warehouse will be upgraded to 19c? The good news is that we are almost there! We have taken the first step by releasing a "19c Preview" mode so you can test your applications and tools against this latest version of the Oracle Database in advance of ADW being autonomously upgraded to 19c. So if you are using Autonomous Data Warehouse today then now is the time to start testing your data warehouse tools and apps using the just released  "19c Preview" feature. Where is "19c Preview"  available? The great news is that you can enable this feature in any of our data centers! We are rolling it out right now so if you don't see the exact flow outlined below when you select your data center then don't panic because it just means we haven't got to your data center yet but we will, just give us a bit more time! If you can see the option to enable preview (I will show you how to do this in the next section) then let's work through the options to build your first 19c Preview instance. How to Enable 19c Preview Scenario 1 - creating a new instance based on 19c Let's assume that in this case you want to create a completely new instance for testing your tools and apps. The great news is that process is almost identical to the existing process for creating a new data warehouse. We have just added two additional mouse clicks. So after you login to the main OCI console you will navigate to your autonomous database console: Note that in this screenshot I am using our US, Ashburn data center. This is my default data center. Since this feature is available across all our data centers it doesn't matter which data center you use. Click on the big blue create "Autonomous Database" button to launch the pop-up create form... this should look familiar...I have add a display name for my new instance "ADW 19c Preview" and then I set the database name:  in this case "ADW19CP". Looks nice and simple so far! Next we pick the workload type which in my example I selected the "Data Warehouse" workload... ...now this brings us to one of our other recent additions to ADW - you can now opt for a "Serverless" vs. "Dedicated". Configuration. So what's the difference? Essentially... Serverless is a simple and elastic deployment choice. Oracle autonomously operates all aspects of the database lifecycle from database placement to backup and updates. Dedicated is a private cloud in public cloud deployment choice. A completely dedicated compute, storage, network and database service for only a single tenant. You get customizable operational policies to guide Autonomous Operations for workload placement, workload optimization, update scheduling, availability level, over provisioning and peak usage. There is more information in one my recent blog posts "There's a minor tweak to our UI - DEDICATED" In this demo I am going to select "Serverless"... and we are almost there...just note that as you scroll down the "Auto Scaling" box has been automatically selected which is now the default behaviour for all new data warehouse instances. Of course if you don't want auto scaling enabled simply untick the box and move on to the next step... Finally we get to the most important tick box on the form! the text above the tick box says "New Database Preview Version 19c Available" and all you need to do is tick the "Enable Preview Mode" box. Of course as with everything in life you need to pay attention to the small print so carefully read the text in the yellow information box: You can't actually move forward until you agree to the T&Cs related to using preview mode. The most important part is that preview mode is time boxed and ends on December 15th 2019.  NOTE: Obviously, we reserve the right to extend the date based on customer demand, however, the console will always show the correct date. Once you confirm agreement to the T&Cs you can scroll down, add your administrator password and finally click on the usual big blue "Create Autonomous Database" button. Notice that the instance page on the service console now has a yellow banner telling you when your preview period will end. There is marker on the OCI console page so you can easily spot a "preview" instance in your list of existing autonomous database instances...   Now let's move on to scenario 2 and create a 19c clone - if you have no idea what a "clone" is then this blog post might help: "What is cloning and what does it have to do with Autonomous Data Warehouse?" Scenario 2 - cloning an existing instance to 19c You may already have an existing data warehouse instance and want to check that everything that's working today (ETL jobs, scripts, reports etc) will still work when ADW moves to Database 19c. Easiest way to do this to simply clone your existing instance and transform it into a 19c instance during the cloning process.  Let's assume that you are on the service console page for your instance... click on the "Actions" button and select "Create Clone" The first step is to select the type of clone you want to create? For testing purposes it's likely that you will want to have the some data in your 19c clone as your original data warehouse as this will make it easier to test reports and scripts. This is what I have done below by selecting "Full Clone".. Of course if you just want to make sure that your existing tools and applications can connect to your new 19c ADW then a metadata clone could well be sufficient. The choice is yours! The rest of the form is as per the usual cloning process until you get towards the bottom where you will spot the new section to enable "19c Preview Mode". Click to enable the preview mode and then agree to the T&Cs and your done! Simply add your administrator password and finally click on the usual big blue "Create Autonomous Database" button. That's it! Welcome to the new world of Autonomous Database 19c. Happy testing! If you want more information about preview versions for Autonomous Database then checkout the overview page in the documentation which is here:  https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/autonomous-preview.html   What's New in 19c Autonomous Database Here is a quick summary of some of the default settings in Oracle Database Preview Version 19c for Autonomous Database: Real-Time Statistics: Enabled by default. Real-Time Statistics enables the database to automatically gather real-time statistics during conventional DML operations. Fresh statistics enable the optimizer to produce more optimal plans. See Real-Time Statistics for more information. High-Frequency Automatic Optimizer Statistics Collection: Enabled by default. High-Frequency Automatic Optimizer Statistics Collection enables the database to gather optimizer statistics every 15 minutes for objects that have stale statistics. See About High-Frequency Automatic Optimizer Statistics Collection for more information. High-Frequency SQL Plan Management Evolve Advisor Task: Enabled by default. When enabled, the database will assess the opportunity for automatic SQL plan changes to improve the performance for known statements every hour. A frequent execution means that the Optimizer has more opportunities to find and evolve to better performing plans. See Managing the SPM Evolve Advisor Task for more information. Automatic Indexing: Disabled by default. To take advantage of automatic indexing you can enable automatic indexing. When enabled automatic indexing automates the index management tasks in an Oracle database. Automatic indexing automatically creates, rebuilds, and drops indexes in a database based on the changes in application workload, thus improving database performance. See Managing Auto Indexes for more information. Enjoy the autonomous world of Database 19c.

If you have been autonomous data warehouse for a while now you will know that when you create a new data warehouse your instance will be based on Database 18c. But you also probably spotted that...

Autonomous

Autonomous Wednesday at OpenWorld 2019 - List of Must See Sessions

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Wednesday and make sure you drink lots of water! If you want this agenda on your phone (iPhone or Android) then checkout our smartphone web app by clicking here !function(){ var d=document.documentElement;d.className=d.className.replace(/no-js/,'js'); if(document.location.href.indexOf('betamode=') > -1) document.write(''); }(); AGENDA - WEDNESDAY 09:00AM Moscone South - Room 207/208 SOLUTION KEYNOTE: A New Vision for Oracle Analytics T.K. Anand, Senior Vice President, Analytics, Oracle 09:00 AM - 10:15 AM SOLUTION KEYNOTE: A New Vision for Oracle Analytics 09:00 AM - 10:15 AM Moscone South - Room 207/208 In this session learn about the bright new future for Oracle Analytics, where customers and partners benefit from augmented analytics working together with Oracle Autonomous Data Warehouse, automating the delivery of personalized insights to fuel innovation without limits. SPEAKERS:T.K. Anand, Senior Vice President, Analytics, Oracle Moscone South - Room 152B Managing One of the Largest IoT Systems in the World with Autonomous Technologies Manuel Martin Marquez, Senior Project Leader, Cern Organisation Européenne Pour La Recherche Nucléaire Sebastien MASSON, Oracle DBA, CERN 09:00 AM - 09:45 AM Managing One of the Largest IoT Systems in the World with Autonomous Technologies 09:00 AM - 09:45 AM Moscone South - Room 152B CERN’s particle accelerator control systems produce more than 2.5 TB of data per day from more than 2 million heterogeneous signals. This IoT system and data is used by scientists and engineers to monitor magnetic field strengths, temperatures, and beam intensities among many other parameters to determine if the equipment is operating correctly. These critical data management and analytics tasks represent important challenges for the organization, and key technologies including big data, machine learning/AI, IoT, and autonomous data warehouses, coupled cloud-based models, can radically optimize the operations. Attend this session to learn from CERN’s experience with IoT systems, Oracle’s cloud, and autonomous solutions. SPEAKERS:Manuel Martin Marquez, Senior Project Leader, Cern Organisation Européenne Pour La Recherche Nucléaire Sebastien MASSON, Oracle DBA, CERN Moscone South - Room 213 Strategy and Roadmap for Oracle Data Integrator and Oracle Enterprise Data Quality Jayant Mahto, Senior Software Development Manager, Oracle 09:00 AM - 09:45 AM Strategy and Roadmap for Oracle Data Integrator and Oracle Enterprise Data Quality 09:00 AM - 09:45 AM Moscone South - Room 213 This session provides a detailed look into Oracle Data Integrator and Oracle Enterprise Data Quality, Oracle’s strategic products for data integration and data quality. See product overviews, highlights from recent customer implementations, and future roadmap plans, including how the products will work with ADW. SPEAKERS:Jayant Mahto, Senior Software Development Manager, Oracle Moscone South - Room 203 The Hidden Data Economy and Autonomous Data Management Paul Sonderegger, Senior Data Strategist, Oracle 09:00 AM - 09:45 AM The Hidden Data Economy and Autonomous Data Management 09:00 AM - 09:45 AM Moscone South - Room 203 Inside every company is a hidden data economy. But because there are no market prices for data inside a single firm, most executives don’t think of it this way. They should. Seeing enterprise data creation, use, and management in terms of supply, demand, and transaction costs will enable companies to compete more effectively on data. In this session learn to see data economy hiding in your company and see Oracle’s vision for helping you get the most out of it. SPEAKERS:Paul Sonderegger, Senior Data Strategist, Oracle Moscone West - Room 3021 Hands-on Lab: Oracle Machine Learning Mark Hornick, Senior Director Data Science and Big Data, ORACLE Marcos Arancibia Coddou, Product Manager, Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Data Science and Big Data, Oracle 09:00 AM - 10:00 AM Hands-on Lab: Oracle Machine Learning 09:00 AM - 10:00 AM Moscone West - Room 3021 In this introductory hands-on-lab, try out the new Oracle Machine Learning Zeppelin-based notebooks that come with Oracle Autonomous Database. Oracle Machine Learning extends Oracle’s offerings in the cloud with its collaborative notebook environment that helps data scientist teams build, share, document, and automate data analysis methodologies that run 100% in Oracle Autonomous Database. Interactively work with your data, and build, evaluate, and apply machine learning models. Import, export, edit, run, and share Oracle Machine Learning notebooks with other data scientists and colleagues. Share and further explore your insights and predictions using the Oracle Analytics Cloud. SPEAKERS:Mark Hornick, Senior Director Data Science and Big Data, ORACLE Marcos Arancibia Coddou, Product Manager, Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Data Science and Big Data, Oracle 10:00AM Moscone South - Room 214 Oracle Essbase 19c: Roadmap on the Cloud Raghuram Venkatasubramanian, Product Manager, Oracle Ashish Jain, Product Manager, Oracle 10:00 AM - 10:45 AM Oracle Essbase 19c: Roadmap on the Cloud 10:00 AM - 10:45 AM Moscone South - Room 214 In this session learn about new analytic platform on Oracle Autonomous Data Warehouse and the role of Oracle Essbase as an engine for data analysis at the speed of thought. Learn how you can perform analysis on data that is stored in Oracle Autonomous Data Warehouse without having to move it into Oracle Essbase. Learn about zero footprint Oracle Essbase and how it provides an architecture that is efficient both in terms of performance and resource utilization. The session also explores new innovations in the future Oracle Essbase roadmap. SPEAKERS:Raghuram Venkatasubramanian, Product Manager, Oracle Ashish Jain, Product Manager, Oracle Moscone West - Room 3000 The Autonomous Trifecta: How a University Leveraged Three Autonomous Technologies Erik Benner, VP Enterprise Transformation, Mythics, Inc. Carla Steinmetz, Senior Principal Consultant, Mythics, Inc. 10:00 AM - 10:45 AM The Autonomous Trifecta: How a University Leveraged Three Autonomous Technologies 10:00 AM - 10:45 AM Moscone West - Room 3000 The challenges facing organizations are often more complex than what one technology can solve. Most solutions require more than just a fast database, or an intelligent analytics tool. True solutions need to store data, move data, and report on data—ideally with all of the components being accelerated with machine learning. In this session learn how Adler University migrated to the cloud with Oracle Autonomous Data Warehouse, Oracle Data Integration, and Oracle Analytics. Learn how the university enhanced the IT systems that support its mission of graduating socially responsible practitioners, engaging communities, and advancing social justice. SPEAKERS:Erik Benner, VP Enterprise Transformation, Mythics, Inc. Carla Steinmetz, Senior Principal Consultant, Mythics, Inc. Moscone South - Room 152C Graph Databases and Analytics: How to Use Them Melli Annamalai, Senior Principal Product Manager, Oracle Hans Viehmann, Product Manager EMEA, Oracle 10:00 AM - 10:45 AM Graph Databases and Analytics: How to Use Them 10:00 AM - 10:45 AM Moscone South - Room 152C Graph databases and graph analysis are powerful new tools that employ advanced algorithms to explore and discover relationships in social networks, IoT, big data, data warehouses, and complex transaction data for applications such as fraud detection in banking, customer 360, public safety, and manufacturing. Using a data model designed to represent linked and connected data, graphs simplify the detection of anomalies, the identification of communities, the understanding of who or what is the most connected, and where there are common or unnatural patterns in data. In this session learn about Oracle’s graph database and analytic technologies for Oracle Cloud, Oracle Database, and big data including new visualization tools, PGX analytics, and query language. SPEAKERS:Melli Annamalai, Senior Principal Product Manager, Oracle Hans Viehmann, Product Manager EMEA, Oracle Moscone South - Room 213 Data Architect's Dilemma: Many Specialty Databases or One Multimodel Database? Tirthankar Lahiri, Senior Vice President, Oracle Juan Loaiza, Executive Vice President, Oracle 10:00 AM - 10:45 AM Data Architect's Dilemma: Many Specialty Databases or One Multimodel Database? 10:00 AM - 10:45 AM Moscone South - Room 213 The most fundamental choice for an enterprise data architect to make is between using a single multimodel database or different specialized databases for each type of data and workload. The decision has profound effects on the architecture, cost, agility, and stability of the enterprise. This session discusses the benefits and tradeoffs of each of these alternatives and also provides an alternative solution that combines the best of the multimodal architecture with a powerful multimodel database. Join this session to find out what is the best choice for your enterprise. SPEAKERS:Tirthankar Lahiri, Senior Vice President, Oracle Juan Loaiza, Executive Vice President, Oracle 10:30AM Moscone West - Room 3021 Hands-on Lab: Oracle Big Data SQL Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle 10:30 AM - 11:30 AM Hands-on Lab: Oracle Big Data SQL 10:30 AM - 11:30 AM Moscone West - Room 3021 Modern data architectures encompass streaming data (e.g. Kafka), Hadoop, object stores, and relational data. Many organizations have significant experience with Oracle Databases, both from a deployment and skill set perspective. This hands-on lab on walks through how to leverage that investment. Learn how to extend Oracle Database to query across data lakes (Hadoop and object stores) and streaming data while leveraging Oracle Database security policies. SPEAKERS:Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle Moscone West - Room 3023 Hands-on Lab: Oracle Multitenant John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle 10:30 AM - 11:30 AM Hands-on Lab: Oracle Multitenant 10:30 AM - 11:30 AM Moscone West - Room 3023 This is your opportunity to get up close and personal with Oracle Multitenant. In this session learn about a very broad range of Oracle Multitenant functionality in considerable depth. Warning: This lab has been filled to capacity quickly at every Oracle OpenWorld that it has been offered. It is strongly recommended that you sign up early. Even if you're only able to get on the waitlist, it's always worth showing up just in case there's a no-show and you can grab an available seat. SPEAKERS:John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle Moscone West - Room 3019 HANDS-ON LAB: RESTful Services with Oracle REST Data Services and Oracle Autonomous Database Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle 10:30 AM - 11:30 AM HANDS-ON LAB: RESTful Services with Oracle REST Data Services and Oracle Autonomous Database 10:30 AM - 11:30 AM Moscone West - Room 3019 In this session learn to develop and deploy a RESTful service using Oracle SQL Developer, Oracle REST Data Services, and Oracle Autonomous Database. Then connect these services as data sources to different Oracle JavaScript Extension Toolkit visualization components to quickly build rich HTML5 applications using a free and open source JavaScript framework. SPEAKERS:Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle 10:45AM The Exchange - Ask Tom Theater Scaling Open Source R and Python for the Enterprise Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle 10:45 AM - 11:05 AM Scaling Open Source R and Python for the Enterprise 10:45 AM - 11:05 AM The Exchange - Ask Tom Theater Open source environments such as R and Python offer tremendous value to data scientists and developers. Scalability and performance on large data sets, however, is not their forte. Memory constraints and single-threaded execution can significantly limit their value for enterprise use. With Oracle Advanced Analytics’ R and Python interfaces to Oracle Database, users can take their R and Python to the next level, deploying for enterprise use on large data sets with ease of deployment. In this session learn the key functional areas of Oracle R Enterprise and Oracle Machine Learning for Python and see how to get the best combination of open source and Oracle Database. SPEAKERS:Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle Moscone South - Room 156B Demystifying Graph Analytics for the Non-expert Peter Jeffcock, Big Data and Data Science, Cloud Business Group, Oracle Sherry Tiao, Oracle 11:15 AM - 12:00 PM Demystifying Graph Analytics for the Non-expert 11:15 AM - 12:00 PM Moscone South - Room 156B This session is aimed at the non-expert: somebody who wants to know how it works so they can ask the technical experts to apply it in new ways to generate new kinds of value for the business. Look behind the curtain to see how graph analytics works. Learn how it enables use cases, from giving directions in your car, to telling the tax authorities if your business partner’s first cousin is conspiring to cheat on payments. SPEAKERS:Peter Jeffcock, Big Data and Data Science, Cloud Business Group, Oracle Sherry Tiao, Oracle Moscone South - Room 214 Oracle Autonomous Data Warehouse: How to Connect Your Tools and Applications What exactly do you need to know to connect your existing on-premises and cloud tools to Oracle Autonomous Data Warehouse? Come to this session and learn about the connection architecture of Autonomous Data Warehouse. For example, learn how to set up and configure Java Database Connectivity connections, how to configure SQLNet, and which Oracle Database Cloud driver you need. Learn how to use Oracle wallets and Java key store files, and what to do if you have a client that is behind a firewall and your network configuration requires an HTTP proxy. All types of connection configurations are explored and explained. 11:15 AM - 12:00 PM "Oracle Autonomous Data Warehouse: How to Connect Your Tools and Applications 11:15 AM - 12:00 PM Moscone South - Room 214 George Lumpkin, Vice President, Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle SPEAKERS:"What exactly do you need to know to connect your existing on-premises and cloud tools to Oracle Autonomous Data Warehouse? Come to this session and learn about the connection architecture of Autonomous Data Warehouse. For example, learn how to set up and configure Java Database Connectivity connections, how to configure SQLNet, and which Oracle Database Cloud driver you need. Learn how to use Oracle wallets and Java key store files, and what to do if you have a client that is behind a firewall and your network configuration requires an HTTP proxy. All types of connection configurations are explored and explained. Moscone South - Room 152B 11 Months with Oracle Autonomous Transaction Processing Eric Grancher, Head of Database Services group, IT department, CERN 11:15 AM - 12:00 PM 11 Months with Oracle Autonomous Transaction Processing 11:15 AM - 12:00 PM Moscone South - Room 152B Oracle Autonomous Transaction Processing and Oracle Autonomous Data Warehouse represent a new way to deploy applications, with the platform providing a performant environment with advanced and automated features. In this session hear one company’s experience with the solution over the past 11 months, from setting up the environment to application deployment, including how to work with it on a daily basis, coordinating with maintenance windows, and more. SPEAKERS:Eric Grancher, Head of Database Services group, IT department, CERN Moscone South - Room 155B Oracle Data Integration Cloud: Database Migration Service Deep Dive Alex Kotopoulis, Director, Product Management – Data Integration Cloud, Oracle Chai Pydimukkala, Senior Director of Product Management – Data Integration Cloud, Oracle 11:15 AM - 12:00 PM Oracle Data Integration Cloud: Database Migration Service Deep Dive 11:15 AM - 12:00 PM Moscone South - Room 155B Oracle Database Migration Service is a new Oracle Cloud service that provides an easy-to-use experience to migrate databases into Oracle Autonomous Transaction Processing, Oracle Autonomous Data Warehouse, or databases on Oracle Cloud Infrastructure. Join this Oracle Product Management–led session to see how Oracle Database Migration Service makes life easier for DBAs by offering an automated means to address the complexity of enterprise databases and data sets. Use cases include offline migrations for batch database migrations, online migrations requiring minimized downtime, and schema conversions for heterogeneous database migrations. The session also shows how it provides the added value of monitoring, auditability, and data validation to deliver real-time progress. SPEAKERS:Alex Kotopoulis, Director, Product Management – Data Integration Cloud, Oracle Chai Pydimukkala, Senior Director of Product Management – Data Integration Cloud, Oracle 11:30AM Moscone South - Room 301 Cloud Native Data Management Gerald Venzl, Master Product Manager, Oracle Maria Colgan, Master Product Manager, Oracle 11:30 AM - 12:15 PM Cloud Native Data Management 11:30 AM - 12:15 PM Moscone South - Room 301 The rise of the cloud has brought many changes to the way applications are built. Containers, serverless, and microservices are now commonplace in a modern cloud native architecture. However, when it comes to data persistence, there are still many decisions and trade-offs to be made. But what if you didn’t have to worry about how to structure data? What if you could store data in any format, independent of having to know if a cloud service could handle it? What if performance, web-scale, or security concerns no longer held you back when you were writing or deploying apps? Sound like a fantasy? This session shows you how you can combine the power of microservices with the agility of cloud native data management to make this fantasy a reality. SPEAKERS:Gerald Venzl, Master Product Manager, Oracle Maria Colgan, Master Product Manager, Oracle 12:00PM Moscone West - Room 3019 HANDS-ON LAB: Low-Code Development with Oracle Application Express and Oracle Autonomous Database David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle 12:00 PM - 01:00 PM HANDS-ON LAB: Low-Code Development with Oracle Application Express and Oracle Autonomous Database 12:00 PM - 01:00 PM Moscone West - Room 3019 Oracle Application Express is a low-code development platform that enables you to build stunning, scalable, secure apps with world-class features that can be deployed anywhere. In this lab start by initiating your free trial for Oracle Autonomous Database and then convert a spreadsheet into a multiuser, web-based, responsive Oracle Application Express application in minutes—no prior experience with Oracle Application Express is needed. Learn how you can use Oracle Application Express to solve many of your business problems that are going unsolved today. SPEAKERS:David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle Moscone West - Room 3021 Hands-on Lab: Oracle Autonomous Data Warehouse Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle 12:00 PM - 01:00 PM Hands-on Lab: Oracle Autonomous Data Warehouse 12:00 PM - 01:00 PM Moscone West - Room 3021 In this hands-on lab discover how you can access, visualize, and analyze lots of different types of data using a completely self-service, agile, and fast service running in the Oracle Cloud: oracle Autonomous Data Warehouse. See how quickly and easily you can discover new insights by blending, extending, and visualizing a variety of data sources to create data-driven briefings on both desktop and mobile browsers—all without the help of IT. It has never been so easy to create visually sophisticated reports that really communicate your discoveries, all in the cloud, all self-service, powered by Oracle Autonomous Data Warehouse. Oracle’s perfect quick-start service for fast data loading, sophisticated reporting, and analysis is for everybody. SPEAKERS:Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle 12:30PM Moscone South - Room 152C Oracle Autonomous Data Warehouse and Oracle Analytics Cloud Boost Agile BI Reiner Zimmermann, Senior Director, DW & Big Data Product Management, Oracle Christian Maar, CEO, 11880 Solutions AG Pawarit Ruengsuksilp, Business Development Officer, Forth Corporation PCL 12:30 PM - 01:15 PM Oracle Autonomous Data Warehouse and Oracle Analytics Cloud Boost Agile BI 12:30 PM - 01:15 PM Moscone South - Room 152C 11880.com is a midsize media company in Germany that used to do traditional BI using Oracle Database technology and Oracle BI. In this session, learn how the switch to Oracle Autonomous Data Warehouse and Oracle Analytics Cloud enabled 11880.com to give more responsibility directly to business users, who are now able to do real self-service BI. They can start their own database, load data, and analyze it the way they want and need it without the need to ask IT and wait for days or weeks to have a system set up and running. SPEAKERS:Reiner Zimmermann, Senior Director, DW & Big Data Product Management, Oracle Christian Maar, CEO, 11880 Solutions AG Pawarit Ruengsuksilp, Business Development Officer, Forth Corporation PCL Moscone South - Room 214 JSON in Oracle Database: Common Use Cases and Best Practices Beda Hammerschmidt, Consulting Member of Technical Staff, Oracle 12:30 PM - 01:15 PM JSON in Oracle Database: Common Use Cases and Best Practices 12:30 PM - 01:15 PM Moscone South - Room 214 JSON is a popular data format in modern applications (web/mobile, microservices, etc.) that brings about increased demand from customers to store, process, and generate JSON data using Oracle Database. JSON supports a wide range of requirements including real-time payment processing systems, social media analytics, and JSON reporting. This session offers common use cases, explains why customers pick JSON over traditional relational or XML data models, and provides tips to optimize performance. Discover the Simple Oracle Document Access (Soda) API that simplifies the interaction with JSON documents in Oracle Database and see how a self-tuning application can be built over Oracle Documents Cloud. SPEAKERS:Beda Hammerschmidt, Consulting Member of Technical Staff, Oracle 02:00PM Moscone North - Hall F Main Keynote Oracle Eexcutives 2:00 p.m. – 3:00 p.m. Main Keynote 2:00 p.m. – 3:00 p.m. Moscone North - Hall F Main OpenWorld Keynote SPEAKERS:Oracle Eexcutives 03:45PM Moscone South - Room 214 Top 10 SQL Features for Developers/DBAs in the Latest Generation of Oracle Database Keith Laker, Senior Principal Product Manager, Oracle 03:45 PM - 04:30 PM Top 10 SQL Features for Developers/DBAs in the Latest Generation of Oracle Database 03:45 PM - 04:30 PM Moscone South - Room 214 SQL is at the heart of every enterprise data warehouse running on Oracle Database, so it is critical that your SQL code is optimized and makes use of the latest features. The latest generation of Oracle Database includes a lot of important new features for data warehouse and application developers and DBAs. This session covers the top 10 most important features, including new and faster count distinct processing, improved support for processing extremely long lists of values, and easier management and optimization of data warehouse and operational queries. SPEAKERS:Keith Laker, Senior Principal Product Manager, Oracle Moscone West - Room 3023 Hands-on Lab: Oracle Database In-Memory Andy Rivenes, Product Manager, Oracle 03:45 PM - 04:45 PM Hands-on Lab: Oracle Database In-Memory 03:45 PM - 04:45 PM Moscone West - Room 3023 Oracle Database In-Memory introduces an in-memory columnar format and a new set of SQL execution optimizations including SIMD processing, column elimination, storage indexes, and in-memory aggregation, all of which are designed specifically for the new columnar format. This lab provides a step-by-step guide on how to get started with Oracle Database In-Memory, how to identify which of the optimizations are being used, and how your SQL statements benefit from them. The lab uses Oracle Database and also highlights the new features available in the latest release. Experience firsthand just how easy it is to start taking advantage of this technology and its performance improvements. SPEAKERS:Andy Rivenes, Product Manager, Oracle 04:00PM Moscone South - Room 306 Six Technologies, One Name: Flashback—Not Just for DBAs Connor Mcdonald, Database Advocate, Oracle 04:00 PM - 04:45 PM Six Technologies, One Name: Flashback—Not Just for DBAs 04:00 PM - 04:45 PM Moscone South - Room 306 There is a remarkable human condition where you can be both cold and sweaty at the same time. It comes about three seconds after you press the Commit button and you realize that you probably needed to have a WHERE clause on that “delete all rows from the SALES table” SQL statement. But Oracle Flashback is not just for those “Oh no!” moments. It also enables benefits for developers, ranging from data consistency and continuous integration to data auditing. Tucked away in Oracle Database, Enterprise Edition, are six independent and powerful technologies that might just save your career and open up a myriad of other benefits as well. Learn more in this session. SPEAKERS:Connor Mcdonald, Database Advocate, Oracle 04:15PM Moscone South - Room 213 The Changing Role of the DBA Maria Colgan, Master Product Manager, Oracle Jenny Tsai-Smith, Vice President, Oracle 04:15 PM - 05:00 PM The Changing Role of the DBA 04:15 PM - 05:00 PM Moscone South - Room 213 The advent of the cloud and the introduction of Oracle Autonomous Database presents opportunities for every organization, but what's the future role for the DBA? In this session explore how the role of the DBA will continue to evolve, and get advice on key skills required to be a successful DBA in the world of the cloud. SPEAKERS:Maria Colgan, Master Product Manager, Oracle Jenny Tsai-Smith, Vice President, Oracle 04:45PM Moscone South - Room 214 Remove Silos and Query the Data Warehouse, Data Lake, and Streams with Oracle SQL Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle 04:45 PM - 05:30 PM Remove Silos and Query the Data Warehouse, Data Lake, and Streams with Oracle SQL 04:45 PM - 05:30 PM Moscone South - Room 214 The latest data architectures take the approach of using the right tool for the right job. Data lakes have become the repository for capturing and analyzing raw data. Data warehouses continue to manage enterprise data, with Oracle Autonomous Data Warehouse greatly simplifying optimized warehouse deployments. Kafka is key capability capturing for real-time streams. This architecture makes perfect sense until you need answers to questions that require correlations across these sources. And you need business users—using their familiar tools and applications—to be able to find the answers themselves. This session outlines how to break down these data silos to query across these sources with security and without moving mountains of data. SPEAKERS:Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Moscone South - Room 152B Creating a Multitenant Sandbox with Oracle Cloud Infrastructure Andrew Westwood, VP, Principal Engineer - Innovation, Bank of the West 04:45 PM - 05:30 PM Creating a Multitenant Sandbox with Oracle Cloud Infrastructure 04:45 PM - 05:30 PM Moscone South - Room 152B Bank of the West needed a multitenanted sandbox database environment in the cloud to work with multiple potential partners and with internal customers simultaneously and independently of each other. The environment needed to provide rapid deployment, data segregation and versioning, data isolation that ensured each partner could access only the information it specifically needed, and self-service that enabled each team to query and build their own database objects and create their own restful services on the sandbox data. In this session learn why Bank of the West chose Oracle Cloud Infrastructure, Oracle Application Container Cloud, and Oracle Application Express. SPEAKERS:Andrew Westwood, VP, Principal Engineer - Innovation, Bank of the West 06:30PM " Chase Center Mission Bay Blocks 29-32 San Francisco, CA Oracle CloudFest.19 "John Mayer. 6:30 p.m.–11 p.m. "Oracle CloudFest.19 6:30 p.m.–11 p.m. Chase Center Mission Bay Blocks 29-32 San Francisco, CA You’ve been energized by the fresh ideas and brilliant minds you’ve engaged with at Oracle OpenWorld. Now cap off the event with an evening of inspiration and celebration with John Mayer. John is many things—a guitar virtuoso, an Instagram live host, a storyteller—and a seven-time Grammy Award-winning performer. Here’s your chance to savor his distinctive and dynamic body of work that’s touched millions worldwide. * Included in full conference pass. Can be purchased for an additional $375 with a Discover pass SPEAKERS:"John Mayer.

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Wednesday and make sure you...

Autonomous

Autonomous Data Warehouse + Oracle Analytics Cloud Hands-on Lab at #OOW19

If you are at Moscone Center for this week's OpenWorld conference then don't miss your chance to get some free hands-on time with Oracle Analytics Cloud querying Autonomous Data Warehouse. We have three sessions left to go this week: Tuesday, September 17, 03:45 PM - 04:45 PM | Moscone West - Room 3021 Wednesday, September 18, 12:00 PM - 01:00 PM | Moscone West - Room 3021 Thursday, September 19, 10:30 AM - 11:30 AM | Moscone West - Room 3021   Each session is being led by Philippe Lions Oracle's own analytics guru and master of data visualization. Philippe and his team built a special version of the their standard ADW+OAC workshop which walks you through connecting OAC to ADW, getting immediate valuable insight out of your ADW data, deepening analysis by mashing-up additional data and leveraging OAC interactive visualization features.  At a higher level the aim is to take a data set which you have never seen before and quickly and easily discover insights into brand and category performance over time, highlighting major sales weaknesses within a specific category. All in just under 45 minutes! Here is Philippe explaining the workshop flow.. Your browser does not support the video tag. What everyone ends up with is the report shown below which identifies the sales weaknesses in the mobile phone category, specifically android phones. All this is done using the powerful data interrogation features of Oracle Analytics Cloud. Make sure you sign up for an ADW hands-on lab running tomorrow, Wednesday and Thursday and learn from the our experts in data visualization and analytics. If you want to try this workshop at home then everything you need is here: https://www.oracle.com/solutions/business-analytics/data-visualization/tutorials.html  

If you are at Moscone Center for this week's OpenWorld conference then don't miss your chance to get some free hands-on time with Oracle Analytics Cloud querying Autonomous Data Warehouse. We have...

Autonomous

Need to Query Autonomous Data Warehouse directly in Slack?

We have one of THE coolest demos ever in the demogrounds area at OpenWorld (head straight to the developer tools area and look for the Chatbot and Digital Assistant demo booths. My colleagues in the Oracle Analytics team have combined Oracle Analytics Cloud with the new Digital Assistant tool to create an amazing experience for users who want to get data out using what I would call slightly unusual channels!  First step is to create a connection to ADW or ATP using the pre-built connection wizards - as shown below there are dozens of different connection wizards to help you get started... Then once the connection is setup and the data sets/projects built you just swap over to your favorite front end collaboration tool such as Slack or Microsoft Teams and just start writing your natural language query which is translated into SQL and the results are returned directly to your collaboration tool where depending what the APIs support you will get a tabular report, a summary analysis and a graphical report.  The Oracle Analytics Cloud experts on the demo booth kindly walked me through their awesome demo: Your browser does not support the video tag. Need to do the same on your mobile phone, then this has you covered as well (apologies had my thumb over the lens for a while in this video - complete amateur) Your browser does not support the video tag. So if you are at Moscone this week and want to see quite possibly the coolest demo at the show then head to the demogrounds and make for the developer tools area which is clearly marked with a huge overhead banner. If you get stuck then swing by the ADW booth which is right next to the escalators as you come down to the lower area in Moscone South. Enjoy OpenWorld 2019 folks!    

We have one of THE coolest demos ever in the demogrounds area at OpenWorld (head straight to the developer tools area and look for the Chatbot and Digital Assistant demo booths. My colleagues in...

Autonomous

Autonomous Tuesday at OpenWorld 2019 - List of Must See Sessions

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Tuesday and make sure you drink lots of water! If you want this agenda on your phone (iPhone or Android) then checkout our smartphone web app by clicking here !function(){ var d=document.documentElement;d.className=d.className.replace(/no-js/,'js'); if(document.location.href.indexOf('betamode=') > -1) document.write(''); }(); AGENDA - TUESDAY 08:45AM Moscone South - Room 204 Using Graph Analytics for New Insights Melliyal Annamalai, Senior Principal Product Manager, Oracle   08:45 AM - 10:45 AM Using Graph Analytics for New Insights 08:45 AM - 10:45 AM Moscone South - Room 204 Graph is an emerging data model for analyzing data. Graphs enable navigation of large and complex data warehouses and intuitive detection of complex relationships for new insights into your data. Powerful algorithms for Graph models such as ranking, centrality, community identification, and path-finding routines support fraud detection, recommendation engines, social network analysis, and more. In this session, learn how to load a graph; insert nodes and edges with Graph APIs; and traverse a graph to find connections and do high-performance Graph analysis with PGQL, a SQL-like graph query language. Also learn how to use visualization tools to work with Graph data. SPEAKERS: Melliyal Annamalai, Senior Principal Product Manager, Oracle     Moscone South - Room 301 Fraud Detection with Oracle Autonomous Data Warehouse and Oracle Machine Learning Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle   08:45 AM - 10:45 AM Fraud Detection with Oracle Autonomous Data Warehouse and Oracle Machine Learning 08:45 AM - 10:45 AM Moscone South - Room 301 Oracle Machine Learning is packaged with Oracle Autonomous Data Warehouse. Together, they can transform your cloud into a powerful anomaly- and fraud-detection cloud solution. Oracle Autonomous Data Warehouse’s Oracle Machine Learning extensive library of in-database machine learning algorithms can help you discover anomalous records and events that stand out—in a multi-peculiar way. Using unsupervised learning techniques (SQL functions), Oracle Autonomous Data Warehouse and Oracle Machine Learning SQL notebooks enable companies to build cloud solutions to detect anomalies, noncompliance, and fraud (taxes, expense reports, people, claims, transactions, and more). In this session, see example scripts, Oracle Machine Learning notebooks, and best practices and hear customer examples. SPEAKERS: Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle   11:15AM Moscone South - Room 215/216 Oracle Database: What's New and What's Coming Next Dominic Giles, Master Product Manager, Oracle   11:15 AM - 12:00 PM Oracle Database: What's New and What's Coming Next 11:15 AM - 12:00 PM Moscone South - Room 215/216 In this informative session learn about recent Oracle Database news and developments, and take a sneak peek into what's coming next from the Oracle Database development team. SPEAKERS: Dominic Giles, Master Product Manager, Oracle     Moscone South - Room 214 Oracle Partitioning: What Everyone Should Know Hermann Baer, Senior Director Product Management, Oracle   11:15 AM - 12:00 PM Oracle Partitioning: What Everyone Should Know 11:15 AM - 12:00 PM Moscone South - Room 214 Oracle Partitioning has proven itself over the course of decades and ensures that tens of thousands of systems run successfully day in and out. But do you really know Oracle Partitioning? Whether you are using it already or are planning to leverage it, this is the time for you to check your knowledge. Attend this session to learn about all the things you already should know about Oracle Partitioning, including the latest innovations of this most widely used functionality in the Oracle Database. SPEAKERS: Hermann Baer, Senior Director Product Management, Oracle     Moscone South - Room 156B Drop Tank: A Cloud Journey Case Study Timothy Miller, CTO, Drop Tank LLC Shehzad Ahmad, Oracle   11:15 AM - 12:00 PM Drop Tank: A Cloud Journey Case Study 11:15 AM - 12:00 PM Moscone South - Room 156B Drop Tank, based out of Burr Ridge, Illinois, was started by a couple of seasoned execs from major oil and fuel brands. Seeing the need for POS connectivity in an otherwise segmented industry, they designed and created proprietary hardware devices. Fast forward a couple years, and Drop Tank is now a full-fledged loyalty program provider for the fuel industry, allowing for an end-to-end solution. Attend this session to learn more. SPEAKERS: Timothy Miller, CTO, Drop Tank LLC Shehzad Ahmad, Oracle     Moscone West - Room 3022C HANDS-ON LAB: Migrate Databases into Oracle Cloud with Oracle Database Migration Service Alex Kotopoulis, Director, Product Management – Data Integration Cloud, Oracle Chai Pydimukkala, Senior Director of Product Management – Data Integration, Oracle Julien Testut, Senior Principal Product Manager – Data Integration Cloud, Oracle Sachin Thatte, Senior Director, Software Development, Data Integration Cloud, Oracle David Allan, Architect - Data Integration Cloud, Oracle Shubha Sundar, Software Development Director, Data Integration Cloud, Oracle   11:15 AM - 12:15 PM HANDS-ON LAB: Migrate Databases into Oracle Cloud with Oracle Database Migration Service 11:15 AM - 12:15 PM Moscone West - Room 3022C Join this hands-on lab to gain experience with Oracle Database Migration Service, which provides an easy-to-use experience to assist in migrating databases into Oracle Autonomous Transaction Processing, Oracle Autonomous Data Warehouse, and databases on Oracle Cloud Infrastructure. The service provides an automated means to address the complexity of enterprise databases and data sets. Use cases include offline migrations for batch migration of databases, online migrations needing minimized database downtime, and schema conversions for heterogeneous migrations from non-Oracle to Oracle databases. Learn how Oracle Database Migration Service delivers monitoring, auditability, and data validation to ensure real-time progress and activity monitoring of processes SPEAKERS: Alex Kotopoulis, Director, Product Management – Data Integration Cloud, Oracle Chai Pydimukkala, Senior Director of Product Management – Data Integration, Oracle Julien Testut, Senior Principal Product Manager – Data Integration Cloud, Oracle Sachin Thatte, Senior Director, Software Development, Data Integration Cloud, Oracle David Allan, Architect - Data Integration Cloud, Oracle Shubha Sundar, Software Development Director, Data Integration Cloud, Oracle     Moscone South - Room 211 Security Architecture for Oracle Database Cloud Tammy Bednar, Sr. Director of Product Management, Database Cloud Services, Oracle   11:15 AM - 12:00 PM Security Architecture for Oracle Database Cloud 11:15 AM - 12:00 PM Moscone South - Room 211 Oracle enables enterprises to maximize the number of mission-critical workloads they can migrate to the cloud while continuing to maintain the desired security posture and reducing the overhead of building and operating data center infrastructure. By design, Oracle provides the security of cloud infrastructure and operations (cloud operator access controls, infrastructure security patching, and so on), and you are responsible for securely configuring your cloud resources. This provides unparalleled control and transparency with applications running on Oracle Cloud. Security in the cloud is a shared responsibility between you and Oracle. Join this session to gain a better understanding of the Oracle Cloud security architecture. SPEAKERS: Tammy Bednar, Sr. Director of Product Management, Database Cloud Services, Oracle     Moscone West - Room 3023 Hands-on Lab: Oracle Database In-Memory Andy Rivenes, Product Manager, Oracle   11:15 AM - 12:15 PM Hands-on Lab: Oracle Database In-Memory 11:15 AM - 12:15 PM Moscone West - Room 3023 Oracle Database In-Memory introduces an in-memory columnar format and a new set of SQL execution optimizations including SIMD processing, column elimination, storage indexes, and in-memory aggregation, all of which are designed specifically for the new columnar format. This lab provides a step-by-step guide on how to get started with Oracle Database In-Memory, how to identify which of the optimizations are being used, and how your SQL statements benefit from them. The lab uses Oracle Database and also highlights the new features available in the latest release. Experience firsthand just how easy it is to start taking advantage of this technology and its performance improvements. SPEAKERS: Andy Rivenes, Product Manager, Oracle     Moscone West - Room 3021 Hands-on Lab: Oracle Machine Learning Mark Hornick, Senior Director Oracle Data Science and Big Data Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Oracle Data Science and Big Data, Oracle   11:15 AM - 12:15 PM Hands-on Lab: Oracle Machine Learning 11:15 AM - 12:15 PM Moscone West - Room 3021 In this introductory hands-on-lab, try out the new Oracle Machine Learning Zeppelin-based notebooks that come with Oracle Autonomous Database. Oracle Machine Learning extends Oracle’s offerings in the cloud with its collaborative notebook environment that helps data scientist teams build, share, document, and automate data analysis methodologies that run 100% in Oracle Autonomous Database. Interactively work with your data, and build, evaluate, and apply machine learning models. Import, export, edit, run, and share Oracle Machine Learning notebooks with other data scientists and colleagues. Share and further explore your insights and predictions using the Oracle Analytics Cloud. SPEAKERS: Mark Hornick, Senior Director Oracle Data Science and Big Data Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Oracle Data Science and Big Data, Oracle   12:00PM The Exchange - Ask Tom Theater Ask TOM: SQL Pattern Matching Chris Saxon, Developer Advocate, Oracle   12:00 PM - 12:20 PM Ask TOM: SQL Pattern Matching 12:00 PM - 12:20 PM The Exchange - Ask Tom Theater SQL is a powerful language for accessing data. Using analytic functions, you can gain lots of insights from your data. But there are many problems that are hard or outright impossible to solve using them. Introduced in Oracle Database 12c, the row pattern matching clause, match_recognize, fills this gap. With this it's easy-to-write efficient SQL to answer many previously tricky questions. This session introduces the match_recognize clause. See worked examples showing how it works and how it's easier to write and understand than traditional SQL solutions. This session is for developers, DBAs, and data analysts who need to do advanced data analysis. SPEAKERS: Chris Saxon, Developer Advocate, Oracle   12:30AM Moscone West - Room 2002 The DBA's Next Great Job Rich Niemiec, Chief Innovation Officer, Viscosity North America   12:30 PM - 01:15 PM The DBA's Next Great Job 12:30 PM - 01:15 PM " Moscone West - Room 2002 What's the next role for the DBA as the autonomous database and Oracle enhancements free up time? This session explores the DBA role (managing more databases with autonomous database) and the integration of AI and machine learning. Topics covered include what's next for the DBA with the autonomous database, important skills for the future such as machine learning and AI, how the merging of tech fields cause both pain and opportunity, and future jobs and the world ahead where Oracle will continue to lead. SPEAKERS: "Rich Niemiec, Chief Innovation Officer, Viscosity North America   12:45PM Moscone West - Room 3021 Hands-on Lab: Oracle Essbase Eric Smadja, Oracle Ashish Jain, Product Manager, Oracle Mike Larimer, Oracle   12:45 PM - 01:45 PM Hands-on Lab: Oracle Essbase 12:45 PM - 01:45 PM Moscone West - Room 3021 The good thing about the new hybrid block storage option (BSO) is that many of the concepts that we learned over the years about tuning a BSO cube still have merit. Knowing what a block is, and why it is important, is just as valuable today in hybrid BSO as it was 25 years ago when BSO was introduced. In this learn best practices for performance optimization, how to manage data blocks, how dimension ordering still impacts things such as calculation order, the reasons why the layout of a report can impact query performance, how logs will help the Oracle Essbase developer debug calculation flow and query performance, and how new functionality within Smart View will help developers understand and modify the solve order. SPEAKERS: Eric Smadja, Oracle Ashish Jain, Product Manager, Oracle Mike Larimer, Oracle   01:30PM The Exchange - Ask Tom Theater Extracting Real Value from Data Lakes with Machine Learning Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle   01:30 PM - 01:50 PM Extracting Real Value from Data Lakes with Machine Learning 01:30 PM - 01:50 PM The Exchange - Ask Tom Theater In this session learn how to interface with and analyze large data lakes using the tools provided by the Oracle Big Data platform (both on-premises and in the cloud), which includes the entire ecosystem of big data components, plus several tools and interfaces ranging from data loading to machine learning and data visualization. Using notebooks for machine learning makes the environment easy and intuitive, and because the platform is open users can choose the language they feel most comfortable with, including R, Python, SQL, Java, etc. Customer stories of how to achieve success on the Oracle Big Data platform are shared, and the proven architecture of the solutions and techniques should help anyone starting a big data project today. SPEAKERS: Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle   01:45PM Moscone South - Room 214 Choosing and Using the Best Performance Tools for Every Situation Mike Hallas, Architect, Oracle Juergen Mueller, Senior Director, Oracle John Zimmerman, Oracle   01:45 PM - 02:30 PM Choosing and Using the Best Performance Tools for Every Situation 01:45 PM - 02:30 PM Moscone South - Room 214 How do you investigate the performance of Oracle Database? The database is highly instrumented with counters and timers to record important activity, and this instrumentation is enabled by default. A wealth of performance data is stored in the Automatic Workload Repository, and this data underpins various performance tools and reports. With so many options available, how do you choose and use the tools with the best opportunity for improvement? Join this session to learn how real-world performance engineers apply different tools and techniques, and see examples based on real-world situations. SPEAKERS: Mike Hallas, Architect, Oracle Juergen Mueller, Senior Director, Oracle John Zimmerman, Oracle     Moscone South - Room 213 Oracle Multitenant: Best Practices for Isolation Can Tuzla, Senior Product Manager, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle   01:45 PM - 02:30 PM Oracle Multitenant: Best Practices for Isolation 01:45 PM - 02:30 PM Moscone South - Room 213 The dramatic efficiencies of deploying databases in cloud configurations are worthless if they come at the expense of isolation. Attend this session to learn about the technologies that deliver the isolation behind Oracle Autonomous Data Warehouse. SPEAKERS: Can Tuzla, Senior Product Manager, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle     Moscone South - Room 211 Oracle Autonomous Data Warehouse: Customer Panel Holger Friedrich, CTO, sumIT AG Reiner Zimmermann, Senior Director, DW & Big Data Product Management, Oracle Joerg Otto, Head of IT, IDS GmbH - Analysis and Reporting Services Manuel Martin Marquez, Senior Project Leader, Cern Organisation Européenne Pour La Recherche Nucléaire James Anthony, CTO, Data Intensity Ltd   01:45 PM - 02:30 PM Oracle Autonomous Data Warehouse: Customer Panel 01:45 PM - 02:30 PM Moscone South - Room 211 In this panel session, hear customers discuss their experiences using Oracle Autonomous Data Warehouse. Topics include both the business and use cases for Oracle Autonomous Data Warehouse and why this service is the perfect fit for their organization. The panel includes customers from Oracle's Global Leaders program, including Droptank, QLX, Hertz, 11880.com, Unior, Deutsche Bank, Caixa Bank, and many more. SPEAKERS: Holger Friedrich, CTO, sumIT AG Reiner Zimmermann, Senior Director, DW & Big Data Product Management, Oracle Joerg Otto, Head of IT, IDS GmbH - Analysis and Reporting Services Manuel Martin Marquez, Senior Project Leader, Cern Organisation Européenne Pour La Recherche Nucléaire James Anthony, CTO, Data Intensity Ltd     Moscone South - Room 215/216 Oracle Autonomous Database: Looking Under the Hood Yasin Baskan, Director Product Manager, Oracle   01:45 PM - 02:30 PM Oracle Autonomous Database: Looking Under the Hood 01:45 PM - 02:30 PM Moscone South - Room 215/216 This session takes you under the hood of Oracle Autonomous Database so you can really understand what makes it tick. Learn about key database features and how these building-block features put the “autonomous” into Autonomous Database. See how to manage and monitor an Autonomous Database using all the most popular tools including Oracle Cloud Infrastructure’s management console, Oracle SQL Developer, Oracle Application Express, REST, and more. Take a deep-dive into what you really need to know to make your DBA career thrive in an autonomous-driven world. SPEAKERS: Yasin Baskan, Director Product Manager, Oracle     Moscone South - Room 152C How Oracle Autonomous Data Warehouse/Oracle Analytics Cloud Can Help FinTech Marketing Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle Pawarit Ruengsuksilp, Business Development Officer, Forth Corporation PCL   01:45 PM - 02:30 PM How Oracle Autonomous Data Warehouse/Oracle Analytics Cloud Can Help FinTech Marketing 01:45 PM - 02:30 PM Moscone South - Room 152C Forth Smart has a unique FinTech business based in Thailand. It has more than 120,000 vending machines, and customers can top-up their mobile credit with cash, as well as sending money to bank accounts, ewallets, and more. It uses Oracle Database and Oracle Analytics Cloud for segmenting its customer base, improving operational efficiency, and quantifying the impact of marketing campaigns. This has allowed Forth Smart to identify cross-selling opportunities and utilize its marketing budget. In this session learn about Forth Smart's journey to introduce Oracle Analytics platforms. Hear about its recent trials to introduce technologies, such as Oracle Autonomous Data Warehouse, to help it stay ahead of competitors. SPEAKERS: Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle Pawarit Ruengsuksilp, Business Development Officer, Forth Corporation PCL   03:15PM Moscone South - Room 207/208 Solution Keynote: Oracle Cloud Infrastructure Clay Magouyrk, Senior Vice President, Software Development, Oracle   03:15 PM – 05:00 PM Solution Keynote: Oracle Cloud Infrastructure 03:15 PM – 05:00 PM Moscone South - Room 207/208 In this session join Oracle Cloud Infrastructure senior leadership for the latest news and updates about Oracle's new class of cloud. SPEAKERS: Clay Magouyrk, Senior Vice President, Software Development, Oracle     Moscone South - Room 214 Oracle Machine Learning: Overview of New Features and Roadmap Mark Hornick, Senior Director, Data Science and Machine Learning, Oracle Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle   03:15 PM – 04:00 PM Oracle Machine Learning: Overview of New Features and Roadmap 03:15 PM – 04:00 PM Moscone South - Room 214 Oracle extended its database into one that discovers insights and makes predictions. With more than 30 algorithms as parallelized SQL functions, Oracle Machine Learning and Oracle Advanced Analytics eliminate data movement, preserve security, and leverage database parallelism and scalability. Oracle Advanced Analytics 19c delivers unequaled performance and a library of in-database machine learning algorithms. In Oracle Autonomous Database, Oracle Machine Learning provides collaborative notebooks for data scientists. Oracle Machine Learning for Python, Oracle R Enterprise, Oracle R Advanced Analytics for Hadoop, cognitive analytics for images and text, and deploying models as microservices round out Oracle’s ML portfolio. In this session hear the latest developments and learn what’s next. SPEAKERS: Mark Hornick, Senior Director, Data Science and Machine Learning, Oracle Marcos Arancibia Coddou, Product Manager, Oracle Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle   03:45PM Moscone West - Room 3022C Introduction to Oracle Data Integration Service, a Fully Managed Serverless ETL Service Julien Testut, Senior Principal Product Manager – Data Integration Cloud, Oracle Sachin Thatte, Senior Director, Software Development, Data Integration Cloud, Oracle Denis Gray, Senior Director of Product Management – Data Integration Cloud, Oracle David Allan, Architect - Data Integration Cloud, Oracle Shubha Sundar, Software Development Director, Data Integration Cloud, Oracle Abhiram Gujjewar, Director Product Management, OCI, Oracle SOMENATH DAS, Senior Director, Software Development - Data Integration Cloud, Oracle   03:45 PM - 04:45 PM Introduction to Oracle Data Integration Service, a Fully Managed Serverless ETL Service 03:45 PM - 04:45 PM Moscone West - Room 3022C Cloud implementations need data integration to be successful. To maximize value from data, you must decide how to transform and aggregate data as it is moved from source to target. Join this hands-on lab to have firsthand experience with Oracle Data Integration Service, a key component of Oracle Cloud Infrastructure that provides fully managed data integration and extract/transform/load (ETL) capabilities. Learn how to implement processes that will prepare, transform, and load data into data warehouses or data lakes and hear how Oracle Data Integration Service integrates with Oracle Autonomous Databases and Oracle Cloud Infrastructure. SPEAKERS: Julien Testut, Senior Principal Product Manager – Data Integration Cloud, Oracle Sachin Thatte, Senior Director, Software Development, Data Integration Cloud, Oracle Denis Gray, Senior Director of Product Management – Data Integration Cloud, Oracle David Allan, Architect - Data Integration Cloud, Oracle Shubha Sundar, Software Development Director, Data Integration Cloud, Oracle Abhiram Gujjewar, Director Product Management, OCI, Oracle SOMENATH DAS, Senior Director, Software Development - Data Integration Cloud, Oracle     Moscone West - Room 3019 hands-on lab: Low-Code Development with Oracle Application Express and Oracle Autonomous Database David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle   03:45 PM - 04:45 PM hands-on lab: Low-Code Development with Oracle Application Express and Oracle Autonomous Database 03:45 PM - 04:45 PM Moscone West - Room 3019 Oracle Application Express is a low-code development platform that enables you to build stunning, scalable, secure apps with world-class features that can be deployed anywhere. In this lab start by initiating your free trial for Oracle Autonomous Database and then convert a spreadsheet into a multiuser, web-based, responsive Oracle Application Express application in minutes—no prior experience with Oracle Application Express is needed. Learn how you can use Oracle Application Express to solve many of your business problems that are going unsolved today. SPEAKERS: David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle   Moscone West - Room 3021 Hands-on Lab: Oracle Autonomous Data Warehouse Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle   03:45 PM - 04:45 PM Hands-on Lab: Oracle Autonomous Data Warehouse 03:45 PM - 04:45 PM Moscone West - Room 3021 In this hands-on lab discover how you can access, visualize, and analyze lots of different types of data using a completely self-service, agile, and fast service running in the Oracle Cloud: oracle Autonomous Data Warehouse. See how quickly and easily you can discover new insights by blending, extending, and visualizing a variety of data sources to create data-driven briefings on both desktop and mobile browsers—all without the help of IT. It has never been so easy to create visually sophisticated reports that really communicate your discoveries, all in the cloud, all self-service, powered by Oracle Autonomous Data Warehouse. Oracle’s perfect quick-start service for fast data loading, sophisticated reporting, and analysis is for everybody. SPEAKERS: Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle   04:15PM Moscone South - Room 214 JSON Document Store in Oracle Autonomous Database: A Developer Perspective Roger Ford, Product Manager, Oracle   04:15 PM - 05:00 PM JSON Document Store in Oracle Autonomous Database: A Developer Perspective 04:15 PM - 05:00 PM Moscone South - Room 214 JSON Document Store is a service that works alongside Oracle Autonomous Data Warehouse and Oracle Autonomous Transaction Processing. In this session see how JSON Document Store makes it easy for developers to create straightforward document-centric applications utilizing the power of Oracle Database without needing to write SQL code or to plan schemas in advance. Learn about the full development lifecycle, from JSON source to an end-user application running on Oracle Cloud. SPEAKERS: Roger Ford, Product Manager, Oracle   05:00PM The Exchange - Ask Tom Theater Ask TOM: SQL Pattern Matching Chris Saxon, Developer Advocate, Oracle   05:00 PM - 05:20 PM Ask TOM: SQL Pattern Matching 05:00 PM - 05:20 PM The Exchange - Ask Tom Theater SQL is a powerful language for accessing data. Using analytic functions, you can gain lots of insights from your data. But there are many problems that are hard or outright impossible to solve using them. Introduced in Oracle Database 12c, the row pattern matching clause, match_recognize, fills this gap. With this it's easy-to-write efficient SQL to answer many previously tricky questions. This session introduces the match_recognize clause. See worked examples showing how it works and how it's easier to write and understand than traditional SQL solutions. This session is for developers, DBAs, and data analysts who need to do advanced data analysis. SPEAKERS: Chris Saxon, Developer Advocate, Oracle   05:15PM Moscone South - Room 211 What’s New in Oracle Optimizer Nigel Bayliss, Senior Principal Product Manager, Oracle   05:15 PM - 06:00 PM What’s New in Oracle Optimizer 05:15 PM - 06:00 PM Moscone South - Room 211 This session covers all the latest features in Oracle Optimizer and provides an overview of the most important differences you can expect to see as you upgrade from earlier releases. Discover all the new, enhanced, and fully automated features relating to Oracle Optimizer. For example, learn how Oracle Database delivers better SQL execution plans with real-time statistics, protects you from query performance regression with SQL quarantine and automatic SQL plan management, and self-tunes your workload with automatic indexing. SPEAKERS: Nigel Bayliss, Senior Principal Product Manager, Oracle     Moscone South - Room 214 Oracle Advanced Compression: Essential Concepts, Tips, and Tricks for Enterprise Data Gregg Christman, Product Manager, Oracle   05:15 PM - 06:00 PM Oracle Advanced Compression: Essential Concepts, Tips, and Tricks for Enterprise Data 05:15 PM - 06:00 PM Moscone South - Room 214 Oracle Database 19c provides innovative row and columnar compression technologies that improve database performance while reducing storage costs. This enables organizations to utilize optimal compression based on usage. Automatic policies enable dynamic, lifecycle-aware compression including lower compression for faster access to hot/active data and higher compression for less frequently accessed inactive/archival data. In this session learn about the essential concepts, as well as compression best practices, and tips and tricks to ensure your organization is achieving the best possible compression savings, and database performance, for your enterprise data. SPEAKERS: Gregg Christman, Product Manager, Oracle     Moscone South - Room 152C Billboards to Dashboards: OUTFRONT Media Uses Oracle Analytics Cloud to Analyze Marketing Tim Vlamis, Vice President & Analytics Strategist, VLAMIS SOFTWARE SOLUTIONS INC Dan Vlamis, CEO - President, VLAMIS SOFTWARE SOLUTIONS INC Derek Hayden, VP - Data Strategy & Analytics, OUTFRONT Media   05:15 PM - 06:00 PM Billboards to Dashboards: OUTFRONT Media Uses Oracle Analytics Cloud to Analyze Marketing 05:15 PM - 06:00 PM Moscone South - Room 152C As one of the premier outdoor and out-of-home media organizations in the world, OUTFRONT Media (Formerly CBS Outdoor) serves America’s largest companies with complex marketing and media planning needs. In this session learn how the OUTFRONT’s analytics team migrated from Oracle Business Intelligence Cloud Service to Oracle Analytics Cloud with an Oracle Autonomous Data Warehouse backend. Hear what lessons were learned and what best practices to follow for Oracle Autonomous Data Warehouse and Oracle Analytics Cloud integration and see how sales analytics and location analysis are leading the organization to new analytics insights. Learn how a small team produced gigantic results in a short time. SPEAKERS: Tim Vlamis, Vice President & Analytics Strategist, VLAMIS SOFTWARE SOLUTIONS INC Dan Vlamis, CEO - President, VLAMIS SOFTWARE SOLUTIONS INC Derek Hayden, VP - Data Strategy & Analytics, OUTFRONT Media     Moscone South - Room 212 Oracle Autonomous Transaction Processing Dedicated Deployment: The End User’s Experience Robert Greene, Senior Director, Product Management, Oracle Jim Czuprynski, Enterprise Data Architect, Viscosity NA   05:15 PM - 06:00 PM Oracle Autonomous Transaction Processing Dedicated Deployment: The End User’s Experience 05:15 PM - 06:00 PM Moscone South - Room 212 Oracle Autonomous Transaction Processing dedicated deployment is the latest update to Oracle Autonomous Database. In this session see the service in action and hear a preview customer describe their experience using a dedicated deployment. Walk through the process of setting up and using HammerDB to run some quick performance workloads and then put the service through its paces to test additional functionality in the areas of private IP, fleet administrator resource isolation, service setup, bulk data loading, patching cooperation controls, transparent application continuity, and more. Hear directly from one of the earliest end users about their real-world experience using Autonomous Database. SPEAKERS: Robert Greene, Senior Director, Product Management, Oracle Jim Czuprynski, Enterprise Data Architect, Viscosity NA     Moscone West - Room 3019 Hands-on lab: RESTful Services with Oracle REST Data Services and Oracle Autonomous Database Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle   05:15 PM - 06:15 PM Hands-on lab: RESTful Services with Oracle REST Data Services and Oracle Autonomous Database 05:15 PM - 06:15 PM Moscone West - Room 3019 In this session learn to develop and deploy a RESTful service using Oracle SQL Developer, Oracle REST Data Services, and Oracle Autonomous Database. Then connect these services as data sources to different Oracle JavaScript Extension Toolkit visualization components to quickly build rich HTML5 applications using a free and open source JavaScript framework. SPEAKERS: Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle     Moscone West - Room 3023 Hands-on Lab: Oracle Multitenant John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle   05:15 PM - 06:15 PM Hands-on Lab: Oracle Multitenant 05:15 PM - 06:15 PM Moscone West - Room 3023 This is your opportunity to get up close and personal with Oracle Multitenant. In this session learn about a very broad range of Oracle Multitenant functionality in considerable depth. Warning: This lab has been filled to capacity quickly at every Oracle OpenWorld that it has been offered. It is strongly recommended that you sign up early. Even if you're only able to get on the waitlist, it's always worth showing up just in case there's a no-show and you can grab an available seat. SPEAKERS: John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle     Moscone West - Room 3021 Hands-on Lab: Oracle Big Data SQL Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle   05:15 PM - 06:15 PM Hands-on Lab: Oracle Big Data SQL 05:15 PM - 06:15 PM Moscone West - Room 3021 Modern data architectures encompass streaming data (e.g. Kafka), Hadoop, object stores, and relational data. Many organizations have significant experience with Oracle Databases, both from a deployment and skill set perspective. This hands-on lab on walks through how to leverage that investment. Learn how to extend Oracle Database to query across data lakes (Hadoop and object stores) and streaming data while leveraging Oracle Database security policies. SPEAKERS: Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle   06:00PM Moscone South - Room 309 Building Apps with an Autonomous Multi-model Database in the Cloud Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle   06:00 PM - 06:45 PM Building Apps with an Autonomous Multi-model Database in the Cloud 06:00 PM - 06:45 PM Moscone South - Room 309 Oracle Autonomous Database provides full support for relational data and non-relational data such as JSON, XML, text, and spatial. It brings new capabilities, delivers dramatically faster performance, moves more application logic and analytics into the database, provides cloud-ready JSON and REST services to simplify application development, and enables analysis on dramatically larger datasets—making it ideally suited for the most advanced multi-model cloud-based applications. Learn more in this session. SPEAKERS: Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle   Oracle Autonomous Database Product Management Team Designed by: Keith Laker Senior Principal Product Manager | Analytic SQL and Autonomous Database Oracle Database Product Management Scotscroft, Towers Business Park, Wilmslow Road, Didsbury. M20 2RY.  

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Tuesday and make sure you...

Autonomous

Where to find us in the demogrounds at #oow19....

If you are OpenWorld tomorrow then I expect that you have lots of questions that need answering. Well, the demogrounds is this place for that! Our development teams will be at the demogrounds all week ready to answer all your questions. The demo area is located in the base of Moscone South and it will formally open at 10:00 am tomorrow (Monday) morning. The full list of opening times is: Monday 10:00am Tuesday 10:30am Wednesday 10:00am The full agenda for #OOW19 is here: https://www.oracle.com/openworld/agenda.html. So where can you find us? And by "us" I mean the analytic SQL, parallel execution, PL/SQL and Autonomous Data Warehouse development teams? Well here some information to guide you to the right area: Got questions about Analytics SQL, parallel execution and PL/SQL? So where do you go to answers to questions about Analytics SQL, parallel execution and PL/SQL? You need to find demo booth ADB-014. Which probably isn't that helpful really! So here is a short video showing you how to find the demo booth Analytics SQL, parallel execution and PL/SQL once you have got yourself to Moscone South and come down the moving stairway/escalator...and then follow in my footsteps and we will be ready and waiting for you! Your browser does not support the video tag.   Got questions about Autonomous Data Warehouse and Analytics Cloud? So where do you go to answers to questions about Autonomous Data Warehouse and Oracle Analytics Cloud? Well, these booths are in a different area so you need to take a slightly different route which is outlined here: Your browser does not support the video tag.   Hopefully that's all clear now! Enjoy OpenWorld and looking forward to answering your questions.      

If you are OpenWorld tomorrow then I expect that you have lots of questions that need answering. Well, the demogrounds is this place for that! Our development teams will be at the demogrounds all week...

Autonomous

Autonomous Monday at OpenWorld 2019 - List of Must See Sessions

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Monday and make sure you drink lots of water! If you want this agenda on your phone (iPhone or Android) then checkout our smartphone web app by clicking here !function(){ var d=document.documentElement;d.className=d.className.replace(/no-js/,'js'); if(document.location.href.indexOf('betamode=') > -1) document.write(''); }(); AGENDA - MONDAY 08:30AM Moscone West - Room 3019 RESTful Services with Oracle REST Data Services and Oracle Autonomous Database Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle 08:30 AM - 09:30 AM RESTful Services with Oracle REST Data Services and Oracle Autonomous Database 08:30 AM - 09:30 AM Moscone West - Room 3019 In this session learn to develop and deploy a RESTful service using Oracle SQL Developer, Oracle REST Data Services, and Oracle Autonomous Database. Then connect these services as data sources to different Oracle JavaScript Extension Toolkit visualization components to quickly build rich HTML5 applications using a free and open source JavaScript framework. SPEAKERS:Jeff Smith, Senior Principal Product Manager, Oracle Ashley Chen, Senior Product Manager, Oracle Colm Divilly, Consulting Member of Technical Staff, Oracle Elizabeth Saunders, Principal Technical Staff, Oracle 09:00AM Moscone South - Room 201 Oracle Database 19c for Developers Chris Saxon, Developer Advocate, Oracle 09:00 AM - 09:45 AM Oracle Database 19c for Developers 09:00 AM - 09:45 AM Moscone South - Room 201 Another year brings another new release: Oracle Database 19c. As always, this brings a host of goodies to help developers. This session gives an overview of the best features for developers in recent releases. It covers JSON support, SQL enhancements, and PL/SQL improvements. It also shows you how to use these features to write faster, more robust, more secure applications with Oracle Database. If you’re a developer or a DBA who regularly writes SQL or PL/SQL and wants to keep up to date, this session is for you. SPEAKERS:Chris Saxon, Developer Advocate, Oracle Moscone South - Room 156B A Unified Platform for All Data Peter Jeffcock, Big Data and Data Science, Cloud Business Group, Oracle Aali Masood, Senior Director, Big Data Go-To-Market, Oracle 09:00 AM - 09:45 AM A Unified Platform for All Data 09:00 AM - 09:45 AM Moscone South - Room 156B Seamless access to all your data, no matter where it’s stored, should be a given. But the proliferation of new data storage technologies has led to new kinds of data silo. In this session learn how to build an information architecture that will stand the test of time and make all data available to the business, whether it’s in Oracle Database, object storage, Hadoop, or elsewhere. SPEAKERS:Peter Jeffcock, Big Data and Data Science, Cloud Business Group, Oracle Aali Masood, Senior Director, Big Data Go-To-Market, Oracle Moscone South - Room 210 Oracle Data Integration Cloud: The Future of Data Integration Now Chai Pydimukkala, Senior Director of Product Management – Data Integration Cloud, Oracle Arun Patnaik, Vice President, Architect - Data Integration Cloud, Oracle 09:00 AM - 09:45 AM Oracle Data Integration Cloud: The Future of Data Integration Now 09:00 AM - 09:45 AM Moscone South - Room 210 The rapid adoption of enterprise cloud–based solutions brings new data integration challenges as data moves from ground-to-cloud and cloud-to-cloud. Join this session led by Oracle Product Management to hear about the Oracle Data Integration Cloud strategy, vision, and roadmap to solve data needs now and into the future. Explore Oracle’s next-generation data integration cloud services and learn about the evolution of data integration in the cloud with a practical path forward for customers using Oracle Data Integration Platform Cloud, Oracle Data Integrator, Oracle GoldenGate, Oracle Enterprise Metadata Management, and more. Learn the crucial role of data integration in successful cloud implementations. SPEAKERS:Chai Pydimukkala, Senior Director of Product Management – Data Integration Cloud, Oracle Arun Patnaik, Vice President, Architect - Data Integration Cloud, Oracle Moscone West - Room 3016 All Analytics, All Data: No Nonsense Shyam Varan Nath, Director IoT & Cloud, BIWA User Group Dan Vlamis, CEO - President, VLAMIS SOFTWARE SOLUTIONS INC Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle 09:00 AM - 09:45 AM All Analytics, All Data: No Nonsense 09:00 AM - 09:45 AM Moscone West - Room 3016 n this session learn from people who have implemented real-world solutions using Oracle's business analytics, machine learning, and spatial and graph technologies. Several Oracle BI, Data Warehouse, and Analytics (BIWA) User Community customers explain how they use Oracle's analytics and machine learning, cloud, autonomous, and on-premises applications to extract more information from their data. Come join the community and learn from people that have been there and done that. SPEAKERS:Shyam Varan Nath, Director IoT & Cloud, BIWA User Group Dan Vlamis, CEO - President, VLAMIS SOFTWARE SOLUTIONS INC Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle 10:00AM YBCA Theater SOLUTION KEYNOTE: Autonomous Data Management Andrew Mendelsohn, Executive Vice President Database Server Technologies, Oracle 10:00 AM - 11:15 AM SOLUTION KEYNOTE: Autonomous Data Management 10:00 AM - 11:15 AM YBCA Theater Oracle is proven to be the database of choice for managing customers operational and analytical workloads on-premises and in the cloud, with truly unique and innovate autonomous database services. In his annual Oracle OpenWorld address, Oracle Executive Vice President Andy Mendelsohn discusses what's new and what's coming next from the Database Development team. SPEAKERS:Andrew Mendelsohn, Executive Vice President Database Server Technologies, Oracle Moscone South - Room 210 Oracle Data Integration Cloud: Data Catalog Service Deep Dive Denis Gray, Senior Director of Product Management – Data Integration Cloud, Oracle Abhiram Gujjewar, Director Product Management, OCI, Oracle 10:00 AM - 10:45 AM Oracle Data Integration Cloud: Data Catalog Service Deep Dive 10:00 AM - 10:45 AM Moscone South - Room 210 The new Oracle Data Catalog enables transparency, traceability, and trust in enterprise data assets in Oracle Cloud and beyond. Join this Oracle Product Management–led session to learn how to get control of your enterprise’s data, metadata, and data movement. The session includes a walkthrough of enterprise metadata search, business glossary, data lineage, automatic metadata harvesting, and native metadata integration with Oracle Big Data and Oracle Autonomous Database. Learn the importance of a metadata catalog and overall governance for big data, data lakes, and data warehouses as well as metadata integration for PaaS services. See how Oracle Data Catalog enables collaboration across enterprise data personas. SPEAKERS:Denis Gray, Senior Director of Product Management – Data Integration Cloud, Oracle Abhiram Gujjewar, Director Product Management, OCI, Oracle Moscone West - Room 3021 Hands-on Lab: Oracle Big Data SQL Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle 10:00 AM - 11:00 AM Hands-on Lab: Oracle Big Data SQL 10:00 AM - 11:00 AM Moscone West - Room 3021 Modern data architectures encompass streaming data (e.g. Kafka), Hadoop, object stores, and relational data. Many organizations have significant experience with Oracle Databases, both from a deployment and skill set perspective. This hands-on lab on walks through how to leverage that investment. Learn how to extend Oracle Database to query across data lakes (Hadoop and object stores) and streaming data while leveraging Oracle Database security policies. SPEAKERS:Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Eric Vinck, Principal Sales Consultant, EMEA Oracle Solution Center, Oracle, Oracle Moscone West - Room 3023 Hands-on Lab: Oracle Multitenant John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle 10:00 AM - 11:00 AM  Hands-on Lab: Oracle Multitenant 10:00 AM - 11:00 AM  Moscone West - Room 3023 This is your opportunity to get up close and personal with Oracle Multitenant. In this session learn about a very broad range of Oracle Multitenant functionality in considerable depth. Warning: This lab has been filled to capacity quickly at every Oracle OpenWorld that it has been offered. It is strongly recommended that you sign up early. Even if you're only able to get on the waitlist, it's always worth showing up just in case there's a no-show and you can grab an available seat. SPEAKERS:John Mchugh, Senior Principal Product Manager, Oracle Thomas Baby, Architect, Oracle Patrick Wheeler, Senior Director, Product Management, Oracle Moscone West - Room 3019 Low-Code Development with Oracle Application Express and Oracle Autonomous Database David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle 10:00 AM - 11:00 AM Low-Code Development with Oracle Application Express and Oracle Autonomous Database 10:00 AM - 11:00 AM Moscone West - Room 3019 Oracle Application Express is a low-code development platform that enables you to build stunning, scalable, secure apps with world-class features that can be deployed anywhere. In this lab start by initiating your free trial for Oracle Autonomous Database and then convert a spreadsheet into a multiuser, web-based, responsive Oracle Application Express application in minutes—no prior experience with Oracle Application Express is needed. Learn how you can use Oracle Application Express to solve many of your business problems that are going unsolved today. SPEAKERS:David Peake, Senior Principal Product Manager, Oracle Marc Sewtz, Senior Software Development Manager, Oracle 11:30AM Moscone West - Room 3023 Hands-on Lab: Oracle Database In-Memory Andy Rivenes, Product Manager, Oracle 11:30 AM - 12:30 PM Hands-on Lab: Oracle Database In-Memory 11:30 AM - 12:30 PM Moscone West - Room 3023 Oracle Database In-Memory introduces an in-memory columnar format and a new set of SQL execution optimizations including SIMD processing, column elimination, storage indexes, and in-memory aggregation, all of which are designed specifically for the new columnar format. This lab provides a step-by-step guide on how to get started with Oracle Database In-Memory, how to identify which of the optimizations are being used, and how your SQL statements benefit from them. The lab uses Oracle Database and also highlights the new features available in the latest release. Experience firsthand just how easy it is to start taking advantage of this technology and its performance improvements. SPEAKERS:Andy Rivenes, Product Manager, Oracle Moscone West - Room 3021 Hands-on Lab: Oracle Autonomous Data Warehouse Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle 11:30 AM - 12:30 PM Hands-on Lab: Oracle Autonomous Data Warehouse 11:30 AM - 12:30 PM Moscone West - Room 3021 In this hands-on lab discover how you can access, visualize, and analyze lots of different types of data using a completely self-service, agile, and fast service running in the Oracle Cloud: oracle Autonomous Data Warehouse. See how quickly and easily you can discover new insights by blending, extending, and visualizing a variety of data sources to create data-driven briefings on both desktop and mobile browsers—all without the help of IT. It has never been so easy to create visually sophisticated reports that really communicate your discoveries, all in the cloud, all self-service, powered by Oracle Autonomous Data Warehouse. Oracle’s perfect quick-start service for fast data loading, sophisticated reporting, and analysis is for everybody. SPEAKERS:Hermann Baer, Senior Director Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle Nilay Panchal, Product Manager, Oracle Yasin Baskan, Senior Principal Product Manager, Oracle 12:15PM The Exchange - Ask Tom Theater Using Machine Learning and Oracle Autonomous Database to Target Your Best Customers Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle 12:15 PM - 12:35 PM Using Machine Learning and Oracle Autonomous Database to Target Your Best Customers 12:15 PM - 12:35 PM The Exchange - Ask Tom Theater Oracle Machine Learning extends Oracle’s offerings in the cloud with its collaborative notebook environment that helps data scientist teams build, share, document, and automate data analysis methodologies that run 100% in Oracle Autonomous Database. In this session learn how to interactively work with your data, and build, evaluate, and apply machine learning models. Import, export, edit, run, and share Oracle Machine Learning notebooks with other data scientists and colleagues, all on Oracle Autonomous Database. SPEAKERS:Charlie Berger, Sr. Director Product Management, Machine Learning, AI and Cognitive Analytics, Oracle Moscone West - Room 3024C Node.js SODA APIs on Oracle Autonomous Database - BYOL Dan Mcghan, Developer Advocate, Oracle 12:30 PM - 02:30 PM Node.js SODA APIs on Oracle Autonomous Database - BYOL 12:30 PM - 02:30 PM " Moscone West - Room 3024C PLEASE NOTE: YOU MUST BRING YOUR OWN LAPTOP (BYOL) TO PARTICIPATE IN THIS HANDS-ON LAB. Oracle Database has many features for different type of data, including spatial and graph, XML, text, and SecureFiles. One of the latest incarnations is Simple Oracle Document Access (SODA), which provides a set of NoSQL-style APIs that enable you to create collections of documents (often JSON), retrieve them, and query them—all without any knowledge of SQL. In this hands-on lab, create an Oracle Autonomous Database instance and learn how to connect to it securely. Then use SODA APIs to complete a Node.js-based REST API powering the front end of a “todo” tracking application. Finally learn how to use the latest JSON functions added to the SQL engine to explore and project JSON data relationally. The lab includes an introduction to more-advanced usage and APIs. SPEAKERS:"Dan Mcghan, Developer Advocate, Oracle 01:00PM Moscone West - Room 3021 Hands-on Lab: Oracle Essbase Eric Smadja, Oracle Ashish Jain, Product Manager, Oracle Mike Larimer, Oracle 01:00 PM - 02:00 PM Hands-on Lab: Oracle Essbase 01:00 PM - 02:00 PM Moscone West - Room 3021 The good thing about the new hybrid block storage option (BSO) is that many of the concepts that we learned over the years about tuning a BSO cube still have merit. Knowing what a block is, and why it is important, is just as valuable today in hybrid BSO as it was 25 years ago when BSO was introduced. In this learn best practices for performance optimization, how to manage data blocks, how dimension ordering still impacts things such as calculation order, the reasons why the layout of a report can impact query performance, how logs will help the Oracle Essbase developer debug calculation flow and query performance, and how new functionality within Smart View will help developers understand and modify the solve order. SPEAKERS:Eric Smadja, Oracle Ashish Jain, Product Manager, Oracle Mike Larimer, Oracle 01:45PM Moscone South - Room 213 Oracle Multitenant: Seven Sources of Savings Patrick Wheeler, Senior Director, Product Management, Oracle 01:45 PM - 02:30 PM Oracle Multitenant: Seven Sources of Savings 01:45 PM - 02:30 PM Moscone South - Room 213 You've heard that the efficiencies of Oracle Multitenant help reduce capital expenses and operating expenses. How much can you expect to save? Attend this session to understand the many ways Oracle Multitenant can help reduce costs so that you can arrive at an answer to this important question. SPEAKERS:Patrick Wheeler, Senior Director, Product Management, Oracle Moscone South - Room 215/216 Rethink Database IT with Autonomous Database Dedicated Robert Greene, Senior Director, Product Management, Oracle Juan Loaiza, Executive Vice President, Oracle 01:45 PM - 02:30 PM Rethink Database IT with Autonomous Database Dedicated 01:45 PM - 02:30 PM Moscone South - Room 215/216 Larry Ellison recently launched a new dedicated deployment option for Oracle Autonomous Transaction Processing. But what does this mean to your organization and how you achieve your key data management goals? This session provides a clear understanding of how dedicated deployment works and illustrates how it can simplify your approach to data management and accelerate your transition to the cloud. SPEAKERS:Robert Greene, Senior Director, Product Management, Oracle Juan Loaiza, Executive Vice President, Oracle Moscone South - Room 211 Oracle Autonomous Data Warehouse: Update, Strategy, and Roadmap George Lumpkin, Vice President, Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle 01:45 PM - 02:30 PM Oracle Autonomous Data Warehouse: Update, Strategy, and Roadmap 01:45 PM - 02:30 PM Moscone South - Room 211 The session provides an overview of Oracle’s product strategy for Oracle Autonomous Data Warehouse. Learn about the capabilities and associated business benefits of recently released features. There is a focus on themes for future releases of the product roadmap as well as the key capabilities being planned. This is a must-attend session for Oracle DBAs, database developers, and cloud architects. SPEAKERS:George Lumpkin, Vice President, Product Management, Oracle Keith Laker, Senior Principal Product Manager, Oracle 01:00PM Moscone West - Room 3019 Building Microservices Using Oracle Autonomous Database Jean De Lavarene, Software Development Director, Oracle Kuassi Mensah, Director, Product Management, Oracle Simon Law, Product Manager, Oracle Pablo Silberkasten, Software Development Manager, Oracle 01:00 PM - 02:00 PM Building Microservices Using Oracle Autonomous Database 01:00 PM - 02:00 PM Moscone West - Room 3019 In this hands-on lab build a Java microservice connecting to Oracle Autonomous Database using the reactive zstreams Ingestor Java library, Helidon SE, Oracle’s JDBC driver, and the connection pool library (UCP). See how to containerize the service with Docker and perform container orchestration using Kubernetes. SPEAKERS:Jean De Lavarene, Software Development Director, Oracle Kuassi Mensah, Director, Product Management, Oracle Simon Law, Product Manager, Oracle Pablo Silberkasten, Software Development Manager, Oracle 02:30PM Moscone South - Room 203 Ten Amazing SQL Features Connor Mcdonald, Database Advocate, Oracle 02:30 PM - 03:15 PM Ten Amazing SQL Features 02:30 PM - 03:15 PM Moscone South - Room 203 Sick and tired of writing thousands of lines of middle-tier code and still having performance problems? Let’s become fans once again of the database by being reintroduced to just how powerful the SQL language really is! Coding is great fun, but we do it to explore complex algorithms, build beautiful applications, and deliver fantastic solutions for our customers, not just to do boring data processing. By expanding our knowledge of SQL facilities, we can let all the boring work be handled via SQL rather than a lot of middle-tier code—and get performance benefits as an added bonus. This session highlights some SQL techniques for solving problems that would otherwise require a lot of complex coding. SPEAKERS:Connor Mcdonald, Database Advocate, Oracle Moscone West - Room 3021 Hands-on Lab: Oracle Machine Learning Mark Hornick, Senior Director Data Science and Big Data, Oracle Marcos Arancibia Coddou, Product Manager, Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Data Science and Big Data, Oracle 02:30 PM - 03:30 PM Hands-on Lab: Oracle Machine Learning 02:30 PM - 03:30 PM Moscone West - Room 3021 In this introductory hands-on-lab, try out the new Oracle Machine Learning Zeppelin-based notebooks that come with Oracle Autonomous Database. Oracle Machine Learning extends Oracle’s offerings in the cloud with its collaborative notebook environment that helps data scientist teams build, share, document, and automate data analysis methodologies that run 100% in Oracle Autonomous Database. Interactively work with your data, and build, evaluate, and apply machine learning models. Import, export, edit, run, and share Oracle Machine Learning notebooks with other data scientists and colleagues. Share and further explore your insights and predictions using the Oracle Analytics Cloud. SPEAKERS:Mark Hornick, Senior Director Data Science and Big Data, Oracle Marcos Arancibia Coddou, Product Manager, Data Science and Big Data, Oracle Charlie Berger, Sr. Director Product Management, Data Science and Big Data, Oracle 02:45PM Moscone South - Room 213 Choosing the Right Database Cloud Service for Your Application Tammy Bednar, Sr. Director of Product Management, Database Cloud Services, Oracle 02:45 PM - 03:30 PM Choosing the Right Database Cloud Service for Your Application 02:45 PM - 03:30 PM Moscone South - Room 213 Oracle Cloud provides automated, customer-managed Oracle Database services in flexible configurations to meet your needs, large or small, with the performance of dedicated hardware. On-demand and highly available Oracle Database, on high-performance bare metal servers or Exadata, is 100% compatible with on-premises Oracle workloads and applications: seamlessly move between the two platforms. Oracle Database Cloud allows you to start at the cost and capability level suitable to your use case and then gives you the flexibility to adapt as your requirements change over time. Join this session to learn about Oracle Database Cloud services. SPEAKERS:Tammy Bednar, Sr. Director of Product Management, Database Cloud Services, Oracle Moscone South - Room 211 Roadmap for Oracle Big Data Appliance and Oracle Big Data Cloud Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Alexey Filanovskiy, Senior Product Manager, Oracle 02:45 PM - 03:30 PM Roadmap for Oracle Big Data Appliance and Oracle Big Data Cloud 02:45 PM - 03:30 PM Moscone South - Room 211 Driven by megatrends like AI and the cloud, big data platforms are evolving. But does this mean it’s time to throw away everything being used right now? This session discusses the impact megatrends have on big data platforms and how you can ensure that you properly invest for a solid future. Topics covered include how platforms such as Oracle Big Data Appliance cater to these trends, as well as the roadmap for Oracle Big Data Cloud on Oracle Cloud Infrastructure. Learn how the combination of these products enable customers and partners to build out a platform for big data and AI, both on-premises and in Oracle Cloud, leveraging the synergy between the platforms and the trends to solve actual business problems. SPEAKERS:Martin Gubar, Director of Product Management, Big Data and Autonomous Database, Oracle Alexey Filanovskiy, Senior Product Manager, Oracle Oracle Autonomous Database Product Management Team Designed by: Keith Laker Senior Principal Product Manager | Analytic SQL and Autonomous Database Oracle Database Product Management Scotscroft, Towers Business Park, Wilmslow Road, Didsbury. M20 2RY.

Here you go folks...OpenWorld is so big this year so to help you get the most from Monday I have put together a cheat sheet listing all the best sessions. Enjoy your Monday and make sure you...

Autonomous Database Now Supports Accessing the Object Storage with OCI Native Authentication

Loading and querying external data from Oracle Cloud Object Storage are amongst some of the common operations performed in Autonomous Database. Accessing Object Storage requires users to have credentials that they can create via the CREATE_CREDENTIAL procedure of DBMS_CLOUD package. Object Storage credentials are based on the user's OCI username and an Oracle-generated token string (also known as 'auth token'). While auth token based credentials are still supported, DBMS_CLOUD now supports creating OCI native credentials as well! In this blog post, we are going to cover how to create a native credential and use it in an operation that requires Object Storage authentication. Let’s start… As you may already be familiar, the syntax to create a credential with a username and an auth token in ADB is as follows: DBMS_CLOUD.CREATE_CREDENTIAL ( credential_name IN VARCHAR2, username IN VARCHAR2, password IN VARCHAR2 DEFAULT NULL); CREATE_CREDENTIAL procedure is now overloaded to provide native authentication with the following syntax: DBMS_CLOUD.CREATE_CREDENTIAL ( credential_name IN VARCHAR2, user_ocid IN VARCHAR2, tenancy_ocid IN VARCHAR2, private_key IN VARCHAR2, fingerprint IN VARCHAR2); In native authentication, the username and password parameters are replaced with the user_ocid, tenancy_ocid, private_key, and fingerprint parameters.  user_ocid and tenancy_ocid are pretty self-explanatory and they correspond to user’s and tenancy’s OCIDs respectively (Check out “Where to Get the Tenancy's OCID and User's OCID” for more details). The private_key parameter specifies the generated private key in PEM format. When it comes to the private_key parameter, there are couple important details worth mentioning. Currently, a private key that is created with a passphrase is not supported. Therefore, you need to make sure you generate a key with no passphrase (Check out “How to Generate an API Signing Key” for more details on how to create a private key with no passphrase). Additionally, the private key that you provide for this parameter should only contain the key itself without any header or footer (e.g. ‘-----BEGIN RSA PRIVATE KEY-----', ‘-----END RSA PRIVATE KEY-----’). Lastly, the fingerprint parameter specifies the fingerprint that can be obtained either after uploading the public key to the console (See “How to Upload the Public Key”) or via the OpenSSL commands (See “How to Get the Key's Fingerprint”). Once you gather all the necessary info and generate your private key, your CREATE_CREDENTIAL procedure should look similar to this: BEGIN DBMS_CLOUD.CREATE_CREDENTIAL ( credential_name => 'OCI_NATIVE_CRED', user_ocid => 'ocid1.user.oc1..aaaaaaaatfn77fe3fxux3o5lego7glqjejrzjsqsrs64f4jsjrhbsk5qzndq', tenancy_ocid => 'ocid1.tenancy.oc1..aaaaaaaapwkfqz3upqklvmelbm3j77nn3y7uqmlsod75rea5zmtmbl574ve6a', private_key => 'MIIEogIBAAKCAQEAsbNPOYEkxM5h0DF+qXmie6ddo95BhlSMSIxRRSO1JEMPeSta0C7WEg7g8SOSzhIroCkgOqDzkcyXnk4BlOdn5Wm/BYpdAtTXk0sln2DH/GCH7l9P8xC9cvFtacXkQPMAXIBDv/zwG1kZQ7Hvl7Vet2UwwuhCsesFgZZrAHkv4cqqE3uF5p/qHfzZHoevdq4EAV6dZK4Iv9upACgQH5zf9IvGt2PgQnuEFrOm0ctzW0v9JVRjKnaAYgAbqa23j8tKapgPuREkfSZv2UMgF7Z7ojYMJEuzGseNULsXn6N8qcvr4fhuKtOD4t6vbIonMPIm7Z/a6tPaISUFv5ASYzYEUwIDAQABAoIBACaHnIv5ZoGNxkOgF7ijeQmatoELdeWse2ZXll+JaINeTwKU1fIB1cTAmSFv9yrbYb4ubKCJuYZJeC6I92rT6gEiNpr670Pn5n43cwblszcTryWOYQVxAcLkejbPA7jZd6CW5xm/vEgRv5qgADVCzDCzrij0t1Fghicc+EJ4BFvOetnzEuSidnFoO7K3tHGbPgA+DPN5qrO/8NmrBebqezGkOuOVkOA64mp467DQUhpAvsy23RjBQ9iTuRktDB4g9cOdOVFouTZTnevN6JmDxufu9Lov2yvVMkUC2YKd+RrTAE8cvRrn1A7XKkH+323hNC59726jT57JvZ+ricRixSECgYEA508e/alxHUIAU9J/uq98nJY/6+GpI9OCZDkEdBexNpKeDq2dfAo9pEjFKYjH8ERj9quA7vhHEwFL33wk2D24XdZl6vq0tZADNSzOtTrtSqHykvzcnc7nXv2fBWAPIN59s9/oEKIOdkMis9fps1mFPFiN8ro4ydUWuR7B2nM2FWkCgYEAxKs/zOIbrzVLhEVgSH2NJVjQs24S8W+99uLQK2Y06R59L0Sa90QHNCDjB1MaKLanAahP30l0am0SB450kEiUD6BtuNHH8EIxGL4vX/SYeE/AF6tw3DqcOYbLPpN4CxIITF0PLCRoHKxARMZLCJBTMGpxdmTNGyQAPWXNSrYEKFsCgYBp0sHr7TxJ1WtO7gvvvd91yCugYBJAyMBr18YY0soJnJRhRL67A/hlk8FYGjLW0oMlVBtduQrTQBGVQjedEsepbrAcC+zm7+b3yfMb6MStE2BmLPdF32XtCH1bOTJSqFe8FmEWUv3ozxguTUam/fq9vAndFaNre2i08sRfi7wfmQKBgBrzcNHN5odTIV8l9rTYZ8BHdIoyOmxVqM2tdWONJREROYyBtU7PRsFxBEubqskLhsVmYFO0CD0RZ1gbwIOJPqkJjh+2t9SH7Zx7a5iV7QZJS5WeFLMUEv+YbYAjnXK+dOnPQtkhOblQwCEY3Hsblj7Xz7o=', fingerprint => '4f:0c:d6:b7:f2:43:3c:08:df:62:e3:b2:27:2e:3c:7a'); END; / PL/SQL procedure successfully completed. We should now be able to see our new credential in the dba_credentials table: SELECT owner, credential_name FROM dba_credentials WHERE credential_name LIKE '%NATIVE%'; OWNER CREDENTIAL_NAME ----- --------------- ADMIN OCI_NATIVE_CRED Let’s go ahead and create an external table using our new credential: BEGIN DBMS_CLOUD.CREATE_EXTERNAL_TABLE( table_name =>'CHANNELS_EXT', credential_name =>'OCI_NATIVE_CRED', file_uri_list =>'https://objectstorage.us-phoenix-1.oraclecloud.com/n/adb/b/bucket_testpdb/o/channels.txt', format => json_object('delimiter' value ','), column_list => 'CHANNEL_ID NUMBER, CHANNEL_DESC VARCHAR2(20), CHANNEL_CLASS VARCHAR2(20), CHANNEL_CLASS_ID NUMBER, CHANNEL_TOTAL VARCHAR2(13), CHANNEL_TOTAL_ID NUMBER'); END; / PL/SQL procedure successfully completed. SELECT count(*) FROM channels_ext; COUNT(*) -------- 5 To summarize, in addition to the auth token based authentication, you can now also have OCI native authentication and CREATE_CREDENTIAL procedure is overloaded to accommodate both options as we demonstrated above.

Loading and querying external data from Oracle Cloud Object Storage are amongst some of the common operations performed in Autonomous Database. Accessing Object Storage requires users to have...

Big Data

Oracle Big Data SQL 4.0 – Query Server

One of the popular new Big Data SQL features is its Query Server.  You can think of Query Server as an Oracle Database 18c query engine that uses the Hive metastore to capture table definitions.  Data isn’t stored in Query Server;  it allows you to access data in Hadoop, NoSQL, Kafka and Object Stores (Oracle Object Store and Amazon S3) using Oracle SQL.  Installation and Configuration Architecturally, here’s what a Big Data SQL deployment with Query Server looks like: There are two parts to the Big Data SQL deployment: Query Server is deployed to an edge node of the cluster (eliminating resource contention with services running on the data nodes) “Cells” are deployed to data nodes. These cells are responsible for scanning, filtering and processing of data and returning summarized results to Query Server Query Server setup is handled by Jaguar – the Big Data SQL install utility.  As part of the installation, update the installer configuration file – bds-config.json – to simply specify that you want to use Query Server and the host that it should be deployed to (that host should be a “gateway” server).  Also, include the Hive databases that should synchronize with Query Server (here we're specifying all): {    “edgedb”: {        "node": “your-host.com",        "enabled": "true"        "sync_hive_db_list": "*"   } } Jaguar will automatically detect the Hive source and the Hadoop cluster security configuration information and configure Query Server appropriately.  Hive metadata will be synchronized with Query Server (either full metadata replacement or incremental updates) using the PL/SQL API (dbms_bdsqs.sync_hive_databases) or thru the cluster management framework (see picture of Cloudera Manager below): For secure clusters, you will log into Query Server using Kerberos – just like you would access other Hadoop cluster services.  Similar to Hive metadata, Kerberos principals can be synchronized thru your cluster admin tool (Cloudera Manager or Ambari), Jaguar (jaguar sync_principals) or PL/SQL (DBMS_BDSQS_ADMIN.ADD_KERBEROS_PRINCIPALS and DBMS_BDSQS_ADMIN.DROP_KERBEROS_PRINCIPALS). Query Your Data Once your Query Server is deployed, query your data using Oracle SQL.  There is a bdsql user that is automatically created and data is accessible thru the bdsqlusr PDB. sqlplus bdsql@bdsqlusr You will see schemas defined for all your hive databases – and external tables within those schemas that map to your Hive tables. The full Oracle SQL language is available to you (queries – not inserts/updates/deletes).  Authorization will leverage the underlying privileges set up on the Hadoop cluster; there are no authorization rules to replicate. You can create new external tables using the Big Data SQL drivers: ORACLE_HIVE – to leverage tables using hive metadata (note, you probably don’t need to do this b/c the external tables are already available) ORACLE_HDFS – to create tables over HDFS data for which there is no hive metadata ORACLE_BIGDATA – to create tables over object store sources Query Server provides a limited use Oracle Database license.  This allows you to create external tables over sources – but not internal tables.  Although there is nothing physically stopping the creation of tables – you will find that any internal table created will be deleted when the Query Server restarts. This beauty of query server is that you get to use the powerful Oracle SQL language and mature Oracle Database optimizer.  It means that your existing query applications will be able to use Query Server as they would any other Oracle Database.  No need to change your queries to support a less rich query engine.  Correlate near real-time Kafka data with information captured in your data lake.  Apply advanced SQL functions like Pattern Matching and time series analyses to gain insights from all your data – and watch your insights and productivity soar :-).

One of the popular new Big Data SQL features is its Query Server.  You can think of Query Server as an Oracle Database 18c query engine that uses the Hive metastore to capture table definitions.  Data...

Big Data

Oracle Big Data SQL 4.0 - Great New Performance Feature

Big Data SQL 4.0 introduces a data processing enhancement that can have a dramatic impact on query performance:  distributed aggregation using in-memory capabilities. Big Data SQL has always done a great job of filtering data on the Hadoop cluster.  It does this using the following optimizations:  1) column projection, 2) partition pruning, 3) storage indexes, 4) predicate pushdown. Column projection is the first optimization.  If your table has 200 columns – and you are only selecting one – then only a single column’s data will be transferred from the Big Data SQL Cell on the Hadoop cluster to the Oracle Database.  This optimization is applied to all file types – CSV, Parquet, ORC, Avro, etc. The image below shows the other parts of the data elimination steps.  Let’s say you are querying 100TB data set. Partition Pruning:  Hive partitions data by a table’s column(s).  If you have two years of data and your table is partitioned by day – and the query is only selecting 2 months – then in this example, 90% of the data will be “pruned” – or not scanned Storage Index:  SIs are a fine-grained data elimination technique.  Statistics are collected for each file’s data blocks based on query usage patterns – and these statistics are used to determine whether or not it’s possible that data for the given query is contained within that block.  If the data does not exist in that block, then the block is not scanned (remember, a block can represent a significant amount of data - oftentimes 128MB). This information is automatically maintained and stored in a lightweight, in-memory structure. Predicate Pushdown:  Certain file types – like Parquet and ORC – are really database files.  Big Data SQL is able to push predicates into those files and only retrieve the data that meets the query criteria Once those scan elimination techniques are applied, Big Data SQL Cells will process and filter the remaining data - returning the results to the database. In-Memory Aggregation In-memory aggregation has the potential to dramatically speed up queries.  Prior to Big Data SQL 4.0, Oracle Database performed the aggregation over the filtered data sets that were returned by Big Data SQL Cells.  With in-memory aggregation, summary computations are run across the Hadoop cluster data nodes.  The massive compute power of the cluster is used to perform aggregations. Below, detailed activity is captured at the customer location level; the query is asking for a summary of activity by region and month. When the query is executed, processing is distributed to each data node on the Hadoop cluster.  Data elimination techniques and filtering is applied – and then each node will aggregate the data up to region/month.  This aggregated data is then returned to the database tier from each cell - and the database then completes the aggregation and applies other functions. Big Data SQL is using an extension to the in-memory aggregation functionality offered by Oracle Database.  Check out the documentation for details on the capabilities and where you can expect a good performance gain. The results can be rather dramatic, as illustrated by the chart found below: This test compares running the same queries with aggregation offload disabled and then enabled.  It shows 1) a simple, single table “count(*)” query, 2) a query against a single table that performs a group by and 3) a query that joins a dimension table to a fact table.  The second and third examples also show increasing the number of columns accessed by the query.  In this simple test, performance improved from 13x to 36x :-). Lots of great new capabilities in Big Data SQL 4.0.  This one may be my favorite :-).

Big Data SQL 4.0 introduces a data processing enhancement that can have a dramatic impact on query performance:  distributed aggregation using in-memory capabilities. Big Data SQL has always done a...

Autonomous

SQL Developer Web comes to Autonomous Data Warehouse - oh YES!

If you login into your cloud console and create a new autonomous data warehouse, or if you have an existing data warehouse instance, then there is great news - you can now launch SQL Developer Web direct from the service console. There is no need to download and install the full desktop version of SQL Developer anymore.  If you want a quick overview of this feature then there is great video by Jeff Smith (Oracle Product Manager for SQL Developer) on YouTube: https://www.youtube.com/watch?v=asHlUW-Laxk. In the video Jeff gives an overview and a short demonstration of this new UI.  ADMIN-only Access First off, straight out the box, only the ADMIN user can access SQL Developer web - which makes perfect sense when you think about it! Therefore, the ADMIN user is always going to be the first person to connect to SQL Dev Web and then they enable access for other users/schemas as required.  A typical autonomous workflow will look something like this: Create a new ADW instance Open Service Console Connect to SQL Dev Web as ADMIN user Enable each schema/user via the ords_admin.enable_schema package Send schema-specific URL to each developer     Connecting as the ADMIN user From the Administration tab on the service console you will see that we added two new buttons - one to access APEX (more information here) and one to access SQL Developer Web.: As this is on the Administration tab then the link for SQL Developer Web, not surprisingly, provides a special admin-only URL which, one you are logged in as the admin user, brings you to the home screen:   and the admin user has some additional features enabled for monitoring their autonomous data warehouse via the hamburger menu in the top left corner     The Dashboard view displays general status information about the data warehouse:. Database Status: Displays the overall status of the database. Alerts: Displays the number of Error alerts in the alert log.  Database Storage: Displays how much storage is being used by the database. Sessions: Displays the status of open sessions in the database. Physical IO Panel: Displays the rates of physical reads and writes of database data. Waits: Displays how many wait events are occurring in the database for various reasons Quick Links: Provides buttons to open the Worksheet, Data Modelel. It also provides a button to open the Oracle Application Express sign-in page for the current database.   Home Page View This page has some cool features - there is a timeline that tracks when objects got added to the database: and there is an associated quick glance view that shows the status of those objects so you know that if it's a table whether it's been automatically analyzed and the stats are up to date:   Enabling the users/schemas To allow a developer to access their schema and login the ADMIN user has to run a small PL/SQL script to enable the schema and that process is outlined here: https://docs.oracle.com/en/database/oracle/sql-developer-web/19.1/sdweb/about-sdw.html#GUID-A79032C3-86DC-4547-8D39-85674334B4FE. Once that's done the ADMIN user can provide the developer with their personal URL to access SQL Developer Web. Essentially, this developer URL is the same as the URL the ADMIN user gets from the service console, but with the /admin/ segment of the URL replaced by /schema-alias/ specified during the "enable-user-access" step. The doc lays this out very nicely.   Guided Demo   Overall adding SQL Dev Web to Autonomous Data Warehouse is going to make life so much easier for DBAs and developers. For most tasks SQL Developer Web can now be the go-to interface for doing most in-database tasks which means you don't have to download and install a desktop tool (which in most corporate environments creates all sorts of problems due to locked-down Windows and Mac desktops). Where to get more information When it comes to SQL Developer there is only one URL you need and it belongs the Jeff Smith who is the product manager for SQL Developer: https://www.thatjeffsmith.com/. Jeff's site contains everything you could ever want to know about using SQL Developer Desktop and SQL Developer Web. There are overview videos, tutorial videos, feature videos, tips & tricks etc etc. Have fun with SQL Developer Web and Autonomous Data Warehouse      

If you login into your cloud console and create a new autonomous data warehouse, or if you have an existing data warehouse instance, then there is great news - you can now launch SQL Developer...

Autonomous

APEX comes to Autonomous Data Warehouse - oh YES!

A big "Autonomous Welcome" to all our APEX developers because your favorite low-code development environment is now built in Autonomous Data Warehouse. And before you ask - YES, if you have existing autonomous data warehouse instances you will find an APEX launch button got added to the Admin tab on your service console (see the screen capture below).  APEX comes to ADW. YES! APEX is now bundled with Autonomous Data Warehouse (even existing data warehouse instances have been updated). What does this mean? It means that you now have free access to  Oracle’s premiere low-code development platform: Application Express (APEX). This provides a low-code development environment that enables customers and partners to build stunning, scalable, secure apps with world-class features fully supported by Autonomous Database.   As an application developer you can now benefit from a simple but very powerful development platform powered by an autonomous database. It’s the perfect combination of low-code development meets zero management database. You can focus on building rich, sophisticated applications with APEX and the database will take care of itself. There are plenty of great use cases for APEX combined with Autonomous Database and from a data warehouse perspective 2 key ones stand out: 1) A replacement for a data mart built around spreadsheets We all done it at some point in our careers - used spreadsheets to build business critical applications and reporting systems. We all know this approach is simply a disaster waiting to happen! Yet almost every organization utilizes spreadsheets to share and report on data. Why? Because spreadsheets are so easy to create - anyone can put together a spreadsheet once they have the data. Once created they often send it out to colleagues who then tweak the data and pass it on to other colleagues, and so forth. This inevitably leads to numerous copies with different data and very a totally flawed business processes. A far better solution is to have a single source of truth stored in a fully secured database with a browser-based app that everyone can use to maintain the data. Fortunately, you now have one! Using Autonomous Data Warehouse and APEX any user can go from a spreadsheet to web app in a few clicks.  APEX provides a very powerful but easy to use wizard that in just a few clicks can transform your spreadsheet into a fully-populated table in Oracle Autonomous Data Warehouse, complete with a fully functioning app with a report and form for maintaining the data. One of the key benefits of switching to APEX is that your data becomes completely secure. The Autonomous Data Warehouse automatically encrypts data at rest and in transit, you can apply data masking profiles on any sensitive data that you share with others and Oracle takes care of making sure you have all the very latest security patches applied. Lastly, all your data is automatically backed. 2) Sharing external data with partners and customers. Many data warehouses make it almost impossible to share data with partners. This can make it very hard to improve your business processes. Providing an app to enable your customers to interact with you and see the same data sets can greatly improve customer satisfaction and lead to repeat business. However, you don't want to expose your internal systems on the Internet, and you have concerns about security, denial of service attacks, and web site uptime. By combining Autonomous Data Warehouse with APEX you can now safely develop public facing apps. Getting Started with APEX!  Getting started with APEX is really easy. Below you will see that I have put together a quick animation which guides you through the process of logging in to your APEX workspace from Autonomous Data Warehouse: What see you above is the process of logging in to APEX for the first time. In this situation you connect as the ADMIN user to the reserved workspace called “INTERNAL”. Once you login you will be required to create a new workspace and assign a user to that workspace to get things setup. In the above screenshots a new workspace called GKL is created for the user GKL. Then at that point everything becomes fully focused on APEX and your  Autonomous Data Warehouse just fades into the background, taking care of itself. It could not be simpler! Learn More about APEX If you are completely new to APEX then I would recommend jumping over to the dedicated Application Express website - apex.oracle.com. On this site you will find the APEX PM team has put together a great 4-step process to get you up-and-running with APEX: https://apex.oracle.com/en/learn/getting-started/ - quick note: obviously, you can skip step 1 which covers how to request and environment on our public APEX service because you have your dedicated environment within your very own Autonomous Data Warehouse. Enjoy your new, autonomous APEX-enabled environment!      

A big "Autonomous Welcome" to all our APEX developers because your favorite low-code development environment is now built in Autonomous Data Warehouse. And before you ask - YES, if you have existing...

Autonomous

There is a minor tweak to our UI - DEDICATED

You may have spotted from all the recent online news headlines and social media activity that we launched a new service for transactional workloads - ATP Dedicated. It allows an organization to rethink how they deliver Database IT, enabling a customizable private database cloud in the public cloud. Obviously this does not affect you if you are using Autonomous Data Warehouse but it does have a subtle impact because our UI has had to change slightly. You will notice in the top left corner of the main console page we now have three types of services: Autonomous Database Autonomous Container Database Autonomous Exadata Infrastructure From a data warehouse perspective you are only interested in the first one in that list: Autonomous Database. In the main table that list all your instances you can see there is a new column headed “Dedicated Infrastructure”. For ADW, this will always show “No” as you can see below.     If you create a new ADW you will notice that the pop-up form has now been replaced by a full width page to make it easier to focus on the fields you need to complete. The new auto-scaling feature is still below the CPU Core Count box (for more information about auto scaling with ADW see this blog post). …and that’s about it for this modest little tweak to our UI. So nothing major, just a subtle change visible when you click on the "Transaction Processing" box. Moving on...  

You may have spotted from all the recent online news headlines and social media activity that we launched a new service for transactional workloads - ATP Dedicated. It allows an organization to...

Autonomous

How to Create a Database Link from an Autonomous Data Warehouse to a Database Cloud Service Instance

Autonomous Data Warehouse (ADW) now supports outgoing database links to any database that is accessible from an ADW instance including Database Cloud Service (DBCS) and other ADW/ATP instances. To use database links with ADW, the target database must be configured to use TCP/IP with SSL (TCPS) authentication. Since both ADW and ATP use TCPS authentication by default, setting up a database link between these services is pretty easy and takes only a few steps. We covered the ADB-to-ADB linking process in the first of this two part series of blog posts about using database links, see Making Database Links from ADW to other Databases. That post explained the simplest use case to configure and use. On the other hand, enabling TCPS authentication in a database that doesn't have it configured (e.g. in DBCS) requires some additional steps that need to be followed carefully. In this blog post, I will try to demonstrate how to create a database link from an ADW instance to a DBCS instance including the steps to enable TCPS authentication. Here is an outline of the steps that we are going to follow: Enable TCPS Authentication in DBCS Connect to DBCS Instance from Client via TCPS Create a DB Link from ADW to DBCS Create a DB Link from DBCS to ADW (Optional) Enable TCPS Authentication in DBCS A DBCS instance uses TCP/IP protocol by default. Configuring TCPS in DBCS involves several steps that need to be performed manually. Since we are going to modify the default listener to use TCPS and it's configured under the grid user, we will be using both oracle and grid users. Here are the steps needed to enable TCPS in DBCS: Create wallets with self signed certificates for server and client Exchange certificates between server and client wallets (Export/import certificates) Add wallet location in the server and the client network files Add TCPS endpoint to the database listener Create wallets with self signed certificates for server and client As part of enabling TCPS authentication, we need to create individual wallets for the server and the client. Each of these wallets has to have their own certificates that they will exchange with one another. For the sake of this example, I will be using a self signed certificate. The client wallet and certificate can be created in the client side; however, I'll be creating my client wallet and certificate in the server and moving them to my local system later on. See Configuring Secure Sockets Layer Authentication for more information. Let's start... Set up wallet directories with the root user [root@dbcs0604 u01]$ mkdir -p /u01/server/wallet [root@dbcs0604 u01]$ mkdir -p /u01/client/wallet [root@dbcs0604 u01]$ mkdir /u01/certificate [root@dbcs0604 /]# chown -R oracle:oinstall /u01/server [root@dbcs0604 /]# chown -R oracle:oinstall /u01/client [root@dbcs0604 /]# chown -R oracle:oinstall /u01/certificate Create a server wallet with the oracle user [oracle@dbcs0604 ~]$ cd /u01/server/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet create -wallet ./ -pwd Oracle123456 -auto_login Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a server certificate with the oracle user [oracle@dbcs0604 wallet]$ orapki wallet add -wallet ./ -pwd Oracle123456 -dn "CN=dbcs" -keysize 1024 -self_signed -validity 3650 -sign_alg sha256 Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a client wallet with the oracle user [oracle@dbcs0604 wallet]$ cd /u01/client/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet create -wallet ./ -pwd Oracle123456 -auto_login Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Create a client certificate with the oracle user [oracle@dbcs0604 wallet]$ orapki wallet add -wallet ./ -pwd Oracle123456 -dn "CN=ctuzla-mac" -keysize 1024 -self_signed -validity 3650 -sign_alg sha256 Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Exchange certificates between server and client wallets (Export/import certificates) Export the server certificate with the oracle user [oracle@dbcs0604 wallet]$ cd /u01/server/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet export -wallet ./ -pwd Oracle123456 -dn "CN=dbcs" -cert /tmp/server.crt Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Export the client certificate with the oracle user [oracle@dbcs0604 wallet]$ cd /u01/client/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet export -wallet ./ -pwd Oracle123456 -dn "CN=ctuzla-mac" -cert /tmp/client.crt Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Import the client certificate into the server wallet with the oracle user [oracle@dbcs0604 wallet]$ cd /u01/server/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet add -wallet ./ -pwd Oracle123456 -trusted_cert -cert /tmp/client.crt Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Import the server certificate into the client wallet with the oracle user [oracle@dbcs0604 wallet]$ cd /u01/client/wallet/ [oracle@dbcs0604 wallet]$ orapki wallet add -wallet ./ -pwd Oracle123456 -trusted_cert -cert /tmp/server.crt Oracle PKI Tool Release 18.0.0.0.0 - Production Version 18.1.0.0.0 Copyright (c) 2004, 2017, Oracle and/or its affiliates. All rights reserved. Operation is successfully completed. Change permissions for the server wallet with the oracle user We need to set the permissions for the server wallet so that it can be accessed when we restart the listener after enabling TCPS endpoint. [oracle@dbcs0604 wallet]$ cd /u01/server/wallet [oracle@dbcs0604 wallet]$ chmod 640 cwallet.sso Add wallet location in the server and the client network files Creating server and client wallets with self signed certificates and exchanging certificates were the initial steps towards the TCPS configuration. We now need to modify both the server and client network files so that they point to their corresponding wallet location and they are ready to use the TCPS protocol. Here's how those files look in my case: Server-side $ORACLE_HOME/network/admin/sqlnet.ora under the grid user # sqlnet.ora Network Configuration File: /u01/app/18.0.0.0/grid/network/admin/sqlnet.ora # Generated by Oracle configuration tools. NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT) wallet_location = (SOURCE= (METHOD=File) (METHOD_DATA= (DIRECTORY=/u01/server/wallet))) SSL_SERVER_DN_MATCH=(ON) Server-side $ORACLE_HOME/network/admin/listener.ora under the grid user wallet_location = (SOURCE= (METHOD=File) (METHOD_DATA= (DIRECTORY=/u01/server/wallet))) LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET # line added by Agent Server-side $ORACLE_HOME/network/admin/tnsnames.ora under the oracle user # tnsnames.ora Network Configuration File: /u01/app/oracle/product/18.0.0.0/dbhome_1/network/admin/tnsnames.ora # Generated by Oracle configuration tools. LISTENER_CDB1 = (ADDRESS = (PROTOCOL = TCPS)(HOST = dbcs0604)(PORT = 1521)) CDB1_IAD1W9 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = dbcs0604)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = cdb1_iad1w9.sub05282047220.vcnctuzla.oraclevcn.com) ) (SECURITY= (SSL_SERVER_CERT_DN="CN=dbcs")) ) PDB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = dbcs0604)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = pdb1.sub05282047220.vcnctuzla.oraclevcn.com) ) (SECURITY= (SSL_SERVER_CERT_DN="CN=dbcs")) ) Add TCPS endpoint to the database listener Now that we are done with configuring our wallets and network files, we can move onto the next step, which is configuring the TCPS endpoint for the database listener. Since our listener is configured under grid, we will be using srvctl command to modify and restart it. Here are the steps: [grid@dbcs0604 ~]$ srvctl modify listener -p "TCPS:1521/TCP:1522" [grid@dbcs0604 ~]$ srvctl stop listener [grid@dbcs0604 ~]$ srvctl start listener [grid@dbcs0604 ~]$ srvctl stop database -database cdb1_iad1w9 [grid@dbcs0604 ~]$ srvctl start database -database cdb1_iad1w9 [grid@dbcs0604 ~]$ lsnrctl status LSNRCTL for Linux: Version 18.0.0.0.0 - Production on 05-JUN-2019 16:07:24 Copyright (c) 1991, 2018, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 18.0.0.0.0 - Production Start Date 05-JUN-2019 16:05:50 Uptime 0 days 0 hr. 1 min. 34 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/18.0.0.0/grid/network/admin/listener.ora Listener Log File /u01/app/grid/diag/tnslsnr/dbcs0604/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.0.0.4)(PORT=1521)))   (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.0.0.4)(PORT=1522))) Services Summary... Service "867e3020a52702dee053050011acf8c0.sub05282047220.vcnctuzla.oraclevcn.com" has 1 instance(s). Instance "cdb1", status READY, has 2 handler(s) for this service... Service "8a8e0ea41ac27e2de0530400000a486a.sub05282047220.vcnctuzla.oraclevcn.com" has 1 instance(s). Instance "cdb1", status READY, has 2 handler(s) for this service... Service "cdb1XDB.sub05282047220.vcnctuzla.oraclevcn.com" has 1 instance(s). Instance "cdb1", status READY, has 1 handler(s) for this service... Service "cdb1_iad1w9.sub05282047220.vcnctuzla.oraclevcn.com" has 1 instance(s). Instance "cdb1", status READY, has 2 handler(s) for this service... Service "pdb1.sub05282047220.vcnctuzla.oraclevcn.com" has 1 instance(s). Instance "cdb1", status READY, has 2 handler(s) for this service... The command completed successfully Please note that in the first step we added the TCPS endpoint to the port 1521 and TCP endpoint to the port 1522 of the default listener. It's also possible to keep the port 1521 as is and add TCPS endpoint to a different port (e.g. 1523). Connect to DBCS Instance from Client via TCPS We should have TCPS authentication configured now. Before we move onto testing, let's take a look at the client-side network files (Please note the public IP address of the DBCS instance in tnsnames.ora): Client-side tnsnames.ora CDB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = 132.145.151.208)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = cdb1_iad1w9.sub05282047220.vcnctuzla.oraclevcn.com) ) (SECURITY= (SSL_SERVER_CERT_DN="CN=dbcs")) ) PDB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = 132.145.151.208)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = pdb1.sub05282047220.vcnctuzla.oraclevcn.com) ) (SECURITY= (SSL_SERVER_CERT_DN="CN=dbcs")) ) Client-side sqlnet.ora WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /Users/cantuzla/Desktop/wallet) ) ) SSL_SERVER_DN_MATCH=(ON) In order to connect to the DBCS instance from the client, you need to add an ingress rule for the port that you want to use (e.g. 1521) in the security list of your virtual cloud network (VCN) in OCI as shown below: We can now try to establish a client connection to PDB1 in our DBCS instance (CDB1): ctuzla-mac:~ cantuzla$ cd Desktop/InstantClient/instantclient_18_1/ ctuzla-mac:instantclient_18_1 cantuzla$ ./sqlplus /nolog SQL*Plus: Release 18.0.0.0.0 Production on Wed Jun 5 09:39:56 2019 Version 18.1.0.0.0 Copyright (c) 1982, 2018, Oracle. All rights reserved. SQL> connect c##dbcs/DBcs123_#@PDB1 Connected. SQL> select * from dual; D - X Create a DB Link from ADW to DBCS We now have a working TCPS authentication in our DBCS instance. Here are the steps from the documentation that we will follow to create a database link from ADW to DBCS: Copy your target database wallet (the client wallet cwallet.sso that we created in /u01/client/wallet) for the target database to Object Store. Create credentials to access your Object Store where you store the cwallet.sso. See CREATE_CREDENTIAL Procedure for details. Create a directory to store the target database wallet: SQL> create directory wallet_dir as 'walletdir'; Directory WALLET_DIR created. Upload the target database wallet to the wallet_dir directory on ADW using DBMS_CLOUD.GET_OBJECT: SQL> BEGIN DBMS_CLOUD.GET_OBJECT( credential_name => 'OBJ_STORE_CRED', object_uri => 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/adwctraining8/b/target-wallet/o/cwallet.sso', directory_name => 'WALLET_DIR'); END; / PL/SQL procedure successfully completed. On ADW create credentials to access the target database. The username and password you specify with DBMS_CLOUD.CREATE_CREDENTIAL are the credentials for the target database that you use to create the database link. Make sure the username consists of all uppercase letters. For this example, I will be using the C##DBCS common user that I created in my DBCS instance: SQL> BEGIN DBMS_CLOUD.CREATE_CREDENTIAL( credential_name => 'DBCS_LINK_CRED', username => 'C##DBCS', password => 'DBcs123_#'); END; / PL/SQL procedure successfully completed. Create the database link to the target database using DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK: SQL> BEGIN DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK( db_link_name => 'DBCSLINK', hostname => '132.145.151.208', port => '1521', service_name => 'pdb1.sub05282047220.vcnctuzla.oraclevcn.com', ssl_server_cert_dn => 'CN=dbcs', credential_name => 'DBCS_LINK_CRED',   directory_name => 'WALLET_DIR'); END; / PL/SQL procedure successfully completed. Use the database link you created to access the data on the target database: SQL> select * from dual@DBCSLINK; D - X Create a DB Link from DBCS to ADW (Optional) Although the previous section concludes the purpose of this blog post, here's something extra for those who are interested. In just couple additional steps, we can also create a DB link from DBCS to ADW: Download your ADW wallet. Upload the wallet to your DBCS instance using sftp or an FTP client. Unzip the wallet: [oracle@dbcs0604 ~]$ cd /u01/targetwallet [oracle@dbcs0604 targetwallet]$ unzip Wallet_adwtuzla.zip Archive: Wallet_adwtuzla.zip inflating: cwallet.sso inflating: tnsnames.ora inflating: truststore.jks inflating: ojdbc.properties inflating: sqlnet.ora inflating: ewallet.p12 inflating: keystore.jks Set GLOBAL_NAMES parameter to FALSE. This step is very important. If you skip this, your DB link will not work. SQL> alter system set global_names=FALSE; System altered. SQL> sho parameter global NAME TYPE VALUE ---------------------- ----------- ----------- allow_global_dblinks boolean FALSE global_names boolean FALSE global_txn_processes integer 1 Create a DB link as follows (notice the my_wallet_directory clause pointing to where we unzipped the ADW wallet): create database link ADWLINK connect to ADMIN identified by ************ using '(description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com)) (connect_data=(service_name=ctwoqpkdfcuwpsd_adwtuzla_high.adwc.oraclecloud.com)) (security=(my_wallet_directory=/u01/targetwallet)(ssl_server_cert_dn="CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US")))'; Database link created. Use the database link you created to access the data on the target database (your ADW instance in this case): SQL> select * from dual@ADWLINK; D - X That's it! In this blog post, we covered how to enable TCPS authentication in DBCS and create an outgoing database link from ADW to our DBCS instance. As a bonus content, we also explored how to create a DB link in the opposite direction, that is from DBCS to ADW. Even though we focused on the DBCS configuration, these steps can be applied when setting up a database link between ADW and any other Oracle database.

Autonomous Data Warehouse (ADW) now supports outgoing database links to any database that is accessible from an ADW instance including Database Cloud Service (DBCS) and other ADW/ATP instances. To use...

Autonomous

Making Database Links from ADW to other Databases

Autonomous Database now fully supports database links. What does this mean? It means that from within your Autonomous Data Warehouse you can make a connection to any other database (on-premise or in the cloud) including other Autonomous Data Warehouse instances and/or Autonomous Transaction Processing instances. Before I dive into an example, let’s take a small step backwards and get a basic understanding of what a database links. Firstly, what is a database link? What Are Database Links? A database link is a pointer that defines a one-way communication path from, in this case an Autonomous Data Warehouse instance to another database. The link is one-way in the sense that a client connected to Autonomous Data Warehouse A can use a link stored in Autonomous Data Warehouse A to access information (schema objects such as tables, views etc) in remote database B, however, users connected to database B cannot use the same link to access data in Autonomous Data Warehouse A. If local users on database B want to access data on Autonomous Data Warehouse A, then they must define their own link to Autonomous Data Warehouse A.     There is more information about database links in the Administrator's Guide. Why Are Database Links Useful? In a lot of situations it can be really useful to have access to the very latest data without having to wait for the next run of the ETL processing. Being able to reach through directly into other databases using a DBLINK can be the fastest way to get an upto-the-minute view of what’s happening with sales orders, or expense claims, or trading positions etc. Another use case is to actually make use of dblinks within the actual ETL processing by pulling data from remote databases into staging tables for further processing. This makes the ETL process impose a minimal processing overhead on the remote databases since all that is being typically executed is a basic SQL SELECT statement. There are additional security benefits as well. For example if you consider an example where employees submit expense reports to Accounts Payable (A/P) application and that information needs to be viewed within a financial data mart. The data mart users should be able to connect to the AP database and run queries to retrieve the desired information. The mart users do not need to be A/P application users to do their analysis or run their ETL jobs; they should only be able to access AP information in a controlled, secured way. Setting Up A Database Link in ADW There are not many steps involved in creating a new database link since all the hard work happens under the covers. The first step is to check that you can actually access the target database- i.e. you have a username and password along with all the connection information. To use database links with Autonomous Data Warehouse the target database must be configured to use TCP/IP with SSL (TCPS) authentication. Fortunately if you want to connect to another Autonomous Data Warehouse or Autonomous Transaction Processing instance then everything already in place because ADB’s use TCP/IP with SSL (TCPS) authentication by default. For other cloud and on-premise databases you will most likely have to configure them to use TCP/IP with SSL (TCPS) authentication. I will try and cover this topic in a separate blog post. Word of caution here…don’t forget to check your Network ACLs settings if you are connecting to another ATP or ADW instance since your attempt to connect might get blocked! There is more information about setting up Network ACLs here. Scenario 1 - Connecting an Autonomous Data Warehouse to your Autonomous Transaction Processing instance Let’s assume that I have an ATP instance running a web store application that contains information about sales orders, distribution channels, customers, products etc. I want to access some of that data in real-time from within my sales data mart. The first step is get hold of the secure connection information for my ATP instance - essentially I need the cwallet.sso file that is part of the client credential file. If I click on the “APDEMO” link above I can access the information about that autonomous database and in the list of “management” buttons is the facility to download the client credentials file…    this gets me a zip file containing a series of files two of which are needed to create a database link: cwallet.sso contains all the security credentials and tnsnames.ora contains all the connection information that I am going to need. Uploading the wallet file… Next I goto to my Object Storage page and create a new bucket to store my wallet file. In this case I have just called it “wallet”. Probably in reality you will name your buckets to identify the target database such as “atpdemo_wallet” simply because every wallet for each database will have exactly the same name - cwallet.sso - so you will need to have a way to identify the target database each wallet is associated with and avoid over-writing each wallet.   within my bucket and I click on the blue “Upload” button to find the cwallet.sso file and move it to my Object Storage bucket:     once my wallet file is in my bucket I then need to setup my autonomous data warehouse to use that file when it makes a connection to my ATP instance.   This is where we step out of the cloud GUI and switch to a client tool like SQL Developer. I have already defined my SQL Developer connection to my Autonomous Data Warehouse which means I can start building my new database link. Step 1 - Moving the wallet file To allow Autonomous Data Warehouse to access the wallet file for my ATP target database wallet I need to put it in a special location -  the data_pump_dir directory. This is done by using DBMS_CLOUD.GET_OBJECT as follows: BEGIN DBMS_CLOUD.GET_OBJECT( credential_name => 'DEF_CRED_NAME', object_uri => 'https://objectstorage.us-phoenix-1.oraclecloud.com/n/adwc/b/adwc_user/o/cwallet.sso', directory_name => 'DATA_PUMP_DIR'); END; / If you execute the above command all you will get back in the console is a message something like this “PL/SQL procedure successfully completed”. So to find out if the file actually got moved you can use the following query to query the data_pump_dir directory  SELECT *  FROM table(dbms_cloud.list_files('DATA_PUMP_DIR')) WHERE object_name LIKE '%.sso' which hopefully returns the following result within SQL Developer that confirms my wallet file is now available to my Autonomous Data Warehouse:     Step 2 - Setting up authentication When my database link process connects to my target ATP instance it obviously needs a valid username and password on my target ATP instance. However, if I can use an account in mu Autonomous Data Warehouse if it matches the account in my ATP instance. Chances are you will want to use a specific account on the target database so a credential is required. This can be setup relatively quickly using the following command: BEGIN DBMS_CLOUD.CREATE_CREDENTIAL( credential_name => ‘ATP_DB_LINK_CRED', username => ’scott', password => ’tiger' ); END; / Step 3 - Defining the new database link  For this step I am going to need access to the tnsnames.ora file to extract specific pieces of information about my ATP instance. Don’t forget that for each autonomous instances there is a range of connections that are identified by resource group ids such as “low”, “medium”, “high”, “tp_urgent” etc. When defining your database link make sure you select the correct information from your tnsnames file. You will need to find the following identifiers: hostname port service name  ssl_server_cert_dn In the example below I am using the “low” resource group connection: BEGIN DBMS_CLOUD_ADMIN.CREATE_DATABASE_LINK( db_link_name => 'SHLINK', hostname => 'adb.us-phoenix-1.oraclecloud.com', port => '1522', service_name => 'example_low.adwc.example.oraclecloud.com', ssl_server_cert_dn => ‘CN=adwc.example.oraclecloud.com,OU=Oracle BMCS PHOENIX,O=Oracle Corporation,L=Redwood City,ST=California,C=US’, credential_name => 'ATP_DB_LINK_CRED'); END; /   I could configure the database link to authenticate using the current user within my Autonomous Data Warehouse (assuming that I had a corresponding account in my Autonomous Transaction Processing instance). That’s all there is to it! Everything is now in place which means I can directly query my transactional data from my data warehouse. For example if I want to see the table of distribution channels for my tp_app_orders then I can simply query the channels table as follows: SELECT  channel_id,  channel_desc,  channel_class,  channel_class_id,  channel_total,  channel_total_id  FROM channels@SHLINK;  Will now return the following:   and if I query my tp_app_orders table I can see the live data in my Autonomous Transaction Processing instance: All Done! That's it. It's now possible to connect your Autonomous Data Warehouse to any other database running on-premise or in the cloud, including other Autonomous Database instances.  This makes it even quicker and easier to pull data from existing systems into your staging tables or even just query data directly from your source applications to get the most up to date view.  In this post you will have noticed that I have created a new database link between an Autonomous Data Warehouse and an Autonomous Transaction Processing instance. Whilst this is a great use case I suspect that many of you will want to connect your Autonomous Data Warehouse to an on-premise database. Well, as I mentioned at the start of this post there are some specific requirements related to using database links with Autonomous Data Warehouse where the target instance is not an autonomous database and we will deal with those in the next post: How to Create a Database Link from an Autonomous Data Warehouse to a Database Cloud Service Instance. For more information about using dblinks with ADW click here.  

Autonomous Database now fully supports database links. What does this mean? It means that from within your Autonomous Data Warehouse you can make a connection to any other database (on-premise or...

Autonomous

Autonomous Data Warehouse - Now with Spatial Intelligence

We are pleased to announce that Oracle Autonomous Data Warehouse now comes with spatial intelligence! If you are completely new to Oracle Autonomous Data Warehouse (where have you been for the last 18 months?) then here is a quick recap of the key features: What is  Oracle Autonomous Data Warehouse Oracle Autonomous Data Warehouse provides a self-driving, self-securing, self-repairing cloud service that eliminate the overhead and human errors associated with traditional database administration. Oracle Autonomous Data Warehouse takes care of configuration, tuning, backup, patching, encryption, scaling, and more. Additional information can be found at https://www.oracle.com/database/autonomous-database.html. Special Thanks... This post has been prepared by David Lapp who is part of the Oracle Spatial and Graph product management team.He is extremely well known within our spatial and graph community. If you want to follow David's posts on the Spatial and Graph blog then use this link and the spatial and graph blog is here. Spatial Features The core set of Spatial features have been enabled on Oracle Autonomous Data Warehouse.  Highlights of the enabled features are; native storage and indexing of point/line/polygon geometries, spatial analysis and processing, such as proximity, containment, combining geometries, distance/area calculations, geofencing to monitor objects entering and exiting areas of interest, and linear referencing to analyze events and activities located along linear networks such as roads and utilities. For details on enabled Spatial features, please see the Oracle Autonomous Data Warehouse documentation.   Loading Your Spatial Data into ADW In Oracle Autonomous Data Warehouse, data loading is typically performed using either Oracle Data Pump or Oracle/3rd party data integration tools. There are a few different ways to load and configure your spatial data sets: Load existing spatial data Load GeoJSON, WKT, or WKB and convert to Spatial using SQL.  Load coordinates and convert to Spatial using SQL.  Obviously the files containing your spatial data sets can be located in your on-premise data center or maybe your desktop computer, but for the fastest data loading performance Oracle Autonomous Data Warehouse also supports loading from files stored in Oracle Cloud Infrastructure Object Storage and other cloud file stores. Details can be found here: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/load-data.html. Configuring Your Spatial Data Routine Spatial data configuration is performed using Oracle SQL Developer GUIs or SQL commands for: Insertion of Spatial metadata Creation of Spatial index Validation of Spatial data   Example Use Case The Spatial features enabled for Oracle Autonomous Data Warehouse support the most common use cases in data warehouse contexts. Organizations such as insurance, finance, and public safety require data warehouses to perform a wide variety of analytics. These data warehouses provide the clues to answer questions such as: What are the major risk factors for a potential new insurance policy? What are the patterns associated with fraudulent bank transactions? What are the predictors of various types of crimes?  In all of these data warehouse scenarios, location is an important factor, and the Spatial features of Oracle Autonomous Data Warehouse enable building and analyzing the dimensions of geographic data. Using the insurance scenario as an example, the major steps for location analysis are: Load historical geocoded policy data including outcomes such as claims and fraud Load geospatial reference data for proximity such as businesses and transportation features Use Spatial to calculate location-based metrics For example lets find the number of restaurants within 5 miles, and the distance to the nearest restaurant:   -- Count within distance -- Use a SQL statement with SDO_WITHIN_DISTANCE   -- and DML to build the result data SELECT policy_id, count(*) as no_restaurant_5_mi  FROM policies, businesses WHERE businesses.type = 'RESTAURANT' AND SDO_WITHIN_DISTANCE(          businesses.geometry,          policies.geometry,         'distance=5 UNIT=mile') = 'TRUE' GROUP BY policy_id; POLICY_ID  NO_RESTAURANT_5_MI 81902842   5 86469385   1 36378345   3 36323540   3 36225484   2 40830185   5 40692826   1 ...   Now we can expand the above query to use the SDO_NN function to do further analysis and find the closest restaurant within the group of restaurants that are within a mile radius of a specific location. Something like the following: -- Distance to nearest -- The SDO_NN function does not perform an implicit join -- so use PL/SQL with DML to build the result data DECLARE  distance_mi NUMBER; BEGIN FOR item IN (SELECT * FROM policies)   LOOP   execute immediate    'SELECT sdo_nn_distance(1) FROM businesses '||   'WHERE businesses.type = ''RESTAURANT'' '||   'AND SDO_NN(b.ora_geometry,:1,'||   '''sdo_batch_size=10 unit=mile'', 1) = ''TRUE'' '||   'AND ROWNUM=1'  INTO distance_mi USING item.geometry;  DBMS_OUTPUT.PUT_LINE(item.policy_id||' '||distance_mi); END LOOP; END;   POLICY_ID RESTAURANT_MI 81902842 4.100 86469385 1.839 36378345 4.674 36323540 3.092 36225484 1.376 40830185 2.237 40692826 4.272 44904642 2.216 ...   Generate the desired spectrum of location-based metrics by stepping through combinations of proximity targets (i.e., restaurants, convenience stores, schools, hospitals, police stations ...) and distances (i.e., 0.25 mi, 0.5 mi, 1 mi, 3 mi, 5 mi...). Combine these location-based metrics with traditional metrics (i.e., value of property, age of policy holder, household income ...) for analytics to identify predictors of outcomes. To enable geographic aggregation, start with a geographic hierarchy with geometry at the most detailed level. For example, a geographic hierarchy where ZONE rolls up to SUB_REGION which rolls up to REGION: DESCRIBE geo_hierarchy Name                     Type ---------------------    ------------------------------------------------- ZONE                     VARCHAR2(30) GEOMETRY                 SDO_GEOMETRY SUB_REGION               VARCHAR2(30) REGION                   VARCHAR2(30) Use Spatial to calculate containment (things found within a region) within the detailed level, which by extension associates the location with all levels of the geo-hierarchy for aggregations: -- Calculate containment -- -- The SDO_ANYINTERACT function performs an implicit join -- so, use a SQL statement with DML to build the result data -- SELECT policy_id, zone FROM policies, geo_hierarchy WHERE SDO_ANYINTERACT(policies.geometry, geo_hierarchy.geometry) = 'TRUE'; POLICY_ID ZONE 81902842 A23 86469385 A21 36378345 A23 36323540 A23 36225484 B22 40830185 C05 40692826 C10 44904642 B16 ...   With these and similar operations, analytics may be performed including the calculation of additional location-based metrics and aggregation by geography.     Summary For important best practices and further details on the use of these and many other Spatial operations, please refer to the Oracle Autonomous Data Warehouse documentation: https://www.oracle.com/database/autonomous-database.html.    

We are pleased to announce that Oracle Autonomous Data Warehouse now comes with spatial intelligence! If you are completely new to Oracle Autonomous Data Warehouse (where have you been for the last 18...

Autonomous

Using Oracle Management Cloud to Monitor Autonomous Databases

How To Monitor All Your Autonomous Database Instances  The latest release of Autonomous Database (which, as you should already know, covers both Autonomous Data Warehouse and Autonomous Transaction processing) has brought integration with Oracle Management Cloud (OMC). This is great news for cloud DBAs and cloud Fleet Managers since it means you can now monitor all your Autonomous Database instances from within a single, integrated console.  So What Is Oracle Management Cloud? Oracle Management Cloud is a suite of integrated monitoring, management services that can bring together information about all your autonomous database instances so you can monitor and manage everything from a single console. In a much broader context where you need to manage a complete application ecosystem or data warehouse ecosystem then Oracle Management Cloud can help you eliminate multiple management/system information silos and infrastructure data, resolve issues faster across your complete cloud ecosystem, and run IT like a business.  What About My Service Console? Each Autonomous Database instance has its own service console for managing and monitoring that specific instance (application database, data mart, data warehouse, sandbox etc). It has everything you, as a DBA or business user, needs to understand how your database is performing and using resources. To date this has been the console that everyone has used for monitoring. But, as you can see, this service console only shows you what’s happening within a specific instance. If you have not looked at the Service Console before then checkout Section 9 in the documentation (in this case for Autonomous Data Warehouse but the same applies to ATP) Managing and Monitoring Performance of Autonomous Data Warehouse. As more and more business teams deploy more and more Autonomous Database instances for their various projects, the challenge has been for anyone tasked with monitoring all these instances: how to get a high-level overview of what’s been deployed and in use across a whole organization? That’s where Oracle Management Cloud (OMC) comes in….the great news is that OMC monitoring of Autonomous Databases is completely free!   OMC Takes Monitoring To The Next Level  The purpose of this post is to look at the newly released integration between Autonomous Database and the Oracle Database Management part of Oracle Management Cloud’s. Let’s look at how to set up Oracle Database Management to discover your Autonomous Databases, how to monitor your Autonomous Database instances and check for predefined alerts and how to use the Performance Hub page. This is what we are aiming to setup in this blog post….the Oracle Database Fleet Home page which as you can see is telling me that I have 8 autonomous database instances - 6 ADW and 2 ATP instances - and 7 of those instances are up and running and one is currently either starting up or shutting down (identified as yellow).       Getting Started… Before you get started with OMC it’s worth taking a step back and thinking about how you want to manage your OMC instance. My view is that it makes sense to create a completely new, separate cloud account which will own your OMC instance (or instances if you want to have more than one). It’s not a requirement but in my opinion it keeps things nice and simple and your DBAs and fleet managers then typically won’t need access to the individual cloud accounts being used by each team for their autonomous database projects. So the first step is probably going to be registering a new cloud account and setting up users to access your OMC instance. Once you have a cloud account setup then there is some initial user configuration that needs to be completed before you can start work with OMC instance. The setup steps are outlined in the documentation - see here. To help you I have also covered these steps in the PowerPoint presentation which is provided at the end of this blog post. Creating an OMC Instance Starting from your My Services home page, click on the big “Create Instance” button and find the “Management Cloud” service in the list of all available services…  …this will allow you to provide the details for your new OMC instance   Oracle Cloud will send you an email notification as soon as your new OMC instance is ready, but it only takes a few minutes and then you can navigate to the “Welcome” page for your new instance which looks like this:   The next step is setup “Discovery Profiles” for each your cloud accounts. This will require a lot of the IDs and information that were part of the user account setup process so you may want to look back over that stage of this process for quick refresh. As you can see below, a discovery profile can be configured to look for only autonomous data warehouse instances or only autonomous transaction processing instances or it can search for both within a specific cloud account. Of course if you have multiple cloud accounts (maybe each department or project team has their own cloud account) then you will need to create discovery profiles for each cloud account. This gives you the ultimate flexibility in terms of setting up OMC in a way that best suits how you and your team want to work. There is more detailed information available in the OMC documentation, see here. The discovery process starts as soon as you click on the “Start” button in the top right-hand corner of the page. It doesn’t take long but the console provides feedback on the status of the discovery job. Once the job or jobs have completed you can navigate to the “Oracle Database” option in the left-hand menu which will bring you to this page - Oracle Database Fleet Home.       You can customise the graphs and information displayed on this page. For example the heat map in the middle of the page can display metrics for: DB Time, execution rate, network I/O, space used or transaction rate. You can switch between different types of views: listing the data as a table rather than a graph:   and because there is so much available data there is a Filter menu that allows you to focus on instances that are up or down, ADW instances vs. ATP instances, database version, and you can even focus in on a specific data center or group of data centers. Once you have setup your filters you can bookmark that view by saving the filters…   In the section headed “Fleet Members”, clicking on one of the instances listed in name column will drill into the performance analysis for that instance. This takes all the information from the Fleet Home page and brings the focus down to that specific instance. For example, selecting my demo instance which is the last row in the table above brings me to this page…   You will notice that this contains a mixture of information from the OCI console page and service console page for my demo instance so it provides me with a great overview of how many CPUs are allocated, database version, amount of storage allocated, database activity and a list of any SQL statements. A sort of mash-up of my OCI console page and service console page. If you then go to the Performance Hub page, we can start to investigate what’s currently happening within my demo instance…   as with the previous screens I can customize the metrics displayed on the graph although this time there is a huge library of metrics to choose from:   and OMC allows me to drill-in to my currently running SQL statement (highlighted in blue in the above screenshot) to look at the resource usage..   and I can get right down to the SQL execution plan…   Take Monitoring To The Next Level With OMC As you can see Oracle Management Cloud takes monitoring of your autonomous database instances to whole new level. Now you can get a single, integrated view for managing all your autonomous database instances, across all your different cloud accounts and across all your data centres. For cloud DBAs and cloud Fleet Managers this is definitely the way to go and more importantly OMC is free for Autonomous Database customers. If you are already using OMC to monitor other parts of your Oracle Cloud deployments (mobile apps, GoldenGate, data integration tools, IaaS, SaaS) then monitoring Autonomous Database instances can now be included in your day-to-day use of OMC. Which means you can use a single console to manage a complete application ecosystem and/or a complete data warehouse ecosystem. For cloud DBAs and cloud Fleet Managers life just got a whole lot easier! Happy monitoring with Oracle Management Cloud.   Where To Get More Information: Step-by-Step setup guide in PDF format is here. Autonomous Data Warehouse documentation is here. Autonomous Transaction Processing documentation is here. OMC documentation for Autonomous Database is here. 

How To Monitor All Your Autonomous Database Instances  The latest release of Autonomous Database (which, as you should already know, covers both Autonomous Data Warehouse and Autonomous Transaction proc...

Autonomous

Loading data into Autonomous Data Warehouse using Datapump

Oracle introduced Autonomous Data warehouse over a year ago, and one of the most common questions that customers ask me is how they can move their data/schema's to ADW (Autonomous Data Warehouse) with minimal efforts. My answer to that is to use datapump, also known as expdp/impdp. ADW doesn't support traditional import and export, so you have to use datapump. Oracle suggests using schema and parallel as a parameter while using datapump. Use the parallel depending upon the number of OCPU that you have for your ADW instance. Oracle also suggests to exclude index, cluster, indextype, materialized_view, materialized_view_log, materialized_zonemap, db_link. This is done in order to save space and speed up the data load process. At present you can only use data_pump_dir as an option for directory. This is the default directory created in ADW.  Also you don't have to worry about the performance of the database since ADW uses technologies like storage indexes, Machine learning, etc to achieve the optimal performance. You can use the file stored on Oracle Object Storage, Amazon S3 storage and Azure Blob Storage as your dumpfile location. I will be using Oracle Object storage in this article.  We will be using the steps below to load data: 1) Export the schema of your current database using expdp 2) Upload the .dmp file to object storage 3) Create Authentication Token  4) Login into ADW using SQLPLUS 5) Create credentials in Autonomous Data Warehouse to connect to Object Storage 6) Run import using Datapump 7) Verify the import Instead of writing more, let's show you how easy it is to do it.    Step 1 : Export the schema of your current database using expdp Use the expdp on your current database to run export. Copy that dump file put it in a location from where you can upload it to object storage.  Step 2 : Upload the .dmp file to object storage.  In order to upload the .dmp file on object storage log in into your cloud console and click object storage:  Once in object storage, select the compartment that you want to use and create a bucket. I am going to use compartment  "Liftandshift" and create bucket "LiftandShiftADW".   Next click on the bucket and click upload to upload the .dmp file. At this point either you can use CLI (Command line Interface) or GUI (Graphic User interface) to upload the .dmp file. If your .dmp file is larger that 2Gib then you have to use CLI. I am going to use GUI since i have a small schema for the demonstration purpose.  Select the .dmp file that you want to upload to object storage and then click upload object. Once you're done, your .dmp file will show up under objects in your Bucket Details Section Step 3 : Create Authentication Token Authentication Token will help us access Object Storage from Autonomous DB.  Under Governance and Administration Section, Click on Identity tab and go to users     Click on authorized user id and then click on Auth Token under resources on the left side to generate the Auth token. Click Generate Token, give it a description, and then click Generate token again and it will create the token for you. Remember to save the token. Once the token is created and saved, you won't be able to retrieve it again.  You can click on the copy button and copy the token to a notepad. Remember to save the token because you will not be able to see the token again. Once done, you can hit the close button on the screen.  Step 4 : Login into ADW using SQLPLUS Go to ADW homepage and click on the ADW database you have created.  Once in the database page click on DB Connection. Click on the Download button to download the wallet. Once the zip file is downloaded, hit the close button.   Download the latest version of instant-client from Oracle website : https://www.oracle.com/technetwork/database/database-technologies/instant-client/downloads/index.html Unzip all the files in one location. I used the location "C:\instantclient\instantclient_18_3" on my system. Once unzipped you will be able to use sqlplus.exe and impdp.exe at that location. Also move the compressed wallet file to that location and unzip the file.  Next update the entries in the sqlnet.ora file and point it to the location of your wallet. I have changed mine to "C:\instantclient\instantclient_18_3" as shown below. Test the connectivity using sqlplus.exe and make sure you are able to connect using the user-id admin. Step 5: Create credentials in Autonomous Data Warehouse to connect to Object Storage Use the below script to create credentials in ADW, and use the Authentication token created earlier as the password. BEGIN   DBMS_CLOUD.CREATE_CREDENTIAL(     credential_name => 'DEF_CRED_NAME',     username => 'oracleidentitycloudservice/ankur.saini@oracle.com',     password => '<password>'                                   <------------ (Use Authentication token Value here instead of the password)   ); END; / Step 6 : Run import using Datapump Since my ADW instance is built using 1 OCPU, I won't be using parallel as an option. I used the script below to run the import: ./impdp.exe admin/<Password>@liftandshift_high directory=data_pump_dir credential=def_cred_name dumpfile= https://swiftobjectstorage.us-ashburn-1.oraclecloud.com/v1/orasenatdoracledigital05/AnkurObject/hrapps.dmp exclude=index, cluster, indextype, materialized_view, materialized_view_log, materialized_zonemap, db_link   Step 7: Verify the import Login into the database using sqlplus or sqldeveloper and verify the import.  You can see how easy it is to move the data to ADW, and that there is not a huge learning curve. Now you can be more productive and focus on your business. Reference: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/load-data.html#GUID-297FE3E6-A823-4F98-AD50-959ED96E6969    

Oracle introduced Autonomous Data warehouse over a year ago, and one of the most common questions that customers ask me is how they can move their data/schema's to ADW (Autonomous Data Warehouse) with...

Autonomous

So you have your CSV, TSV and JSON data lying in your Oracle Cloud Object Store. How do you get it over into your Autonomous Database?

You have finally gotten your ducks in a row to future proof your data storage, and uploaded all your necessary production data into Oracle Cloud's object store. Splendid! Now, how do you get this data into your Autonomous Database? Here I provide some practical examples of how to copy over your data from the OCI object store to your Autonomous Data Warehouse (ADW). You may use a similar method to copy data into your Autonomous Transaction Processing (ATP) instance too. We will dive into the meanings of some of the widely used parameters, which will help you and your teams derive quick business value by creating your Data Warehouse in a jiffy! An extremely useful feature of the fully managed ADW service is the ability to copy data lying in your external object store quickly and easily. The DBMS_CLOUD.COPY_DATA API procedure enables this behavior of copying (or loading) your data into your database from data files lying in your object store, enabling your ADW instance to run queries and analyses on said data. A few pre-cursor requirements to get us running these analyses: Make sure you have a running ADW instance with a little storage space, a credentials wallet and a working connection to your instance. If you haven’t done this already you can follow this simple Lab 1 tutorial. Use this link to download the data files for the following examples. You will need to unzip and upload these files to your Object Store. Once again, if you don’t know how to do this, follow Lab 3 Step 4 in this tutorial, which uploads files to a bucket in the Oracle Cloud Object Store, the most streamlined option. You may also use AWS or Azure object stores if required, you may refer to the documentation for more information on this. You will provide the URLs of the files lying in your object store to the API. If you already created your object store bucket’s URL in the lab you may use that, else to create this, use the URL below and replace the placeholders <region_name>, <tenancy_name> and <bucket_name> with your object store bucket’s region, tenancy and bucket names. The easiest way to find this information is to look at your object’s details in the object store, by opening the right-hand menu and clicking “Object details” (see screenshot below).  https://objectstorage.<region_name>..oraclecloud.com/n/<tenancy_name>/b/<bucket_name>/o/ Note: You may also use a SWIFT URL for your file here if you have one.   Have the latest version of SQL Developer installed (ADW requires at least v18.3 and above).   Comma Separated Value (CSV) Files   CSV files are one of the most common file formats out there. We will begin by using a plain and simple CSV format file for Charlotte’s (NC) Weather History dataset, which we will use as the data for our first ADW table. Open this Weather History ‘.csv’ file in a text editor to have a look at the data. Notice each field is separated by a comma, and each row ends by going to the next line. (ie. Which implies a newline ‘\n’ character). Also note that the first line is not data, but metadata (column names).   Let us now write a script to create a table, with the appropriate column names, in our ADW instance, and copy over this data file lying in our object store into it. We will specify the format of the file as CSV. The format parameter in the DBMS_CLOUD.COPY_DATA procedure takes a JSON object, which can be provided in two possible formats. format => '{"format_option" : “format_value” }' format => json_object('format_option' value 'format_value')) The second format option has been used in the script below. set define on define base_URL = <paste Object Store or SWIFT URL created above here> create table WEATHER_REPORT_CSV (REPORT_DATE VARCHAR2(20),     ACTUAL_MEAN_TEMP NUMBER,     ACTUAL_MIN_TEMP NUMBER,     ACTUAL_MAX_TEMP NUMBER,     AVERAGE_MIN_TEMP NUMBER,     AVERAGE_MAX_TEMP NUMBER,     AVERAGE_PRECIPITATION NUMBER(5,2)); begin    DBMS_CLOUD.COPY_DATA(   table_name =>'WEATHER_REPORT_CSV',   credential_name =>'OBJ_STORE_CRED',     file_uri_list =>   '&base_URL/Charlotte_NC_Weather_History.csv',   format =>     json_object('type' value 'csv',      'skipheaders' value '1',     'dateformat' value 'mm/dd/yy')); end; / Let us breakdown and understand this script. We are first creating the WEATHER_REPORT_CSV table with the appropriate named columns for our destination table. We are then invoking the “COPY_DATA” procedure in the DBMS_CLOUD API  package and providing it the table name we created in our Data Warehouse, our user credentials (we created this in the pre-requisites), the object store file list that contains our data, and a format JSON object that describes the format of our file to the API.  The format parameter is a constructed JSON object with format options ‘type’ and ‘skipheaders’. The type specifies the file format as CSV, while skipheaders tells the API how many rows are metadata headers which should be skipped. In our file, that is 1 row of headers. The 'dateformat' parameter specifies the format of the date column in the file we are reading from; We will look at this parameter in more detailed examples below. Great! If this was successful, we have our first data warehouse table containing data from an object store file. If you do see errors during this copy_data process, follow Lab 3 Step 12 to troubleshoot them with the help of the necessary log file. If required, you can also drop this table with the “DROP TABLE” command. On running this copy data without errors, you now have a working data warehouse table. You may now query and join the WEATHER_REPORT_CSV with other tables in your Data Warehouse instance with the regular SQL or PL/SQL you know and love. As an example, let us find the days in our dataset during which it was pleasant in Charlotte. SELECT * FROM WEATHER_REPORT_CSV where actual_mean_temp > 69 and        actual_mean_temp < 74;   Tab Separated Value (TSV) Files   Another popular file format involves tab delimiters or TSV files. In the files you downloaded look for the Charlotte Weather History ‘.gz’ file. Unzip, open and have look at the ".tsv" file in it in a text editor as before. You will notice each row in this file is ended by a pipe ‘|’ character instead of a newline character, and the fields are separated by tabspaces. Oftentimes applications you might work with will output data in less intelligible formats such as this one, and so below is a slightly more advanced example of how to pass such data into DBMS_CLOUD. Let’s run the following script: create table WEATHER_REPORT_TSV (REPORT_DATE VARCHAR2(20),     ACTUAL_MEAN_TEMP NUMBER,     ACTUAL_MIN_TEMP NUMBER,     ACTUAL_MAX_TEMP NUMBER,     AVERAGE_MIN_TEMP NUMBER,     AVERAGE_MAX_TEMP NUMBER,     AVERAGE_PRECIPITATION NUMBER(5,2));   begin   DBMS_CLOUD.COPY_DATA(     table_name =>'WEATHER_REPORT_TSV',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'&base_URL/Charlotte_NC_Weather_History.gz',     format => json_object(                           'removequotes' value 'true',                           'dateformat' value 'mm/dd/yy',                           'delimiter' value '\t',                            'recorddelimiter' value '''|''',                            'skipheaders' value '1'                           )  ); end; /   SELECT * FROM WEATHER_REPORT_TSV where actual_mean_temp > 69 and        actual_mean_temp < 74; Let us understand the new parameters here: 'ignoremissingcolumns' value 'true': Notice there is no data for the last column “AVERAGE_PRECIPITATION”. This parameter allows the copy data script to skip over columns from the column list, that have no data in the data file. 'removequotes' value 'true': The first column ‘date’ has data surrounded by double quotes. For this data to be converted to an Oracle date type, these quotes need to be removed. Note that when using the type parameter for CSV files as we did in the first example, this removequotes option is true by default. 'dateformat' value 'mm/dd/yy': If we expect a date column to be converted and stored into an Oracle date column (after removing the double quotes of course), we should provide the date column’s format. If we don’t provide a format, the date column will look for the database's default date format. You can see the dateformat documentation here. 'delimiter' value '\t': Fields in this file are tab delimited, so the delimiter we specify is the special character. 'recorddelimiter' value '''|''': Each record or row in our file is delimited by a pipe ‘|’ symbol, and so we specify this parameter which separates out each row. Note that unlike the delimiter parameter, the recorddelimiter must be enclosed in single quotes as shown here. A nuance here is that the last row in your dataset doesn’t need the record delimiter when it is the default newline character, however it does for other character record delimiters to indicate the end of that row. Also note that since ADW is LINUX/UNIX based, source data files with newline as record delimiters, that have been created on Windows, must use “\r\n” as the format option. Both these nuances will likely have updated functionality in future releases. 'rejectlimit' value '1': We need this parameter here to fix an interesting problem. Unlike with the newline character, if we don’t specify a pipe record delimiter here at the very end of the file, we get an error because the API doesn’t recognize where the last row’s, last column ends. If we do specify the pipe record delimiter however, the API expects a new line because the record has been delimited, and we get a null error for the last non-existent row. To fix situations like this, where we know we might have one or more problem rows, we use the reject limit parameter to allow some number of rows to be rejected. If we use ‘unlimited’ as our reject limit, then any number of rows may be rejected. The default reject limit is 0. 'compression' value 'gzip': Notice the .tsv file is zipped into a gzip “.gz” file, which we have used in the URL. We use this parameter so the file will be unzipped appropriately before the table is created. As before, once this is successful, the table structure has been created following which the data is loaded into the table from the data file in the object store. We then proceed to query the table in your Data Warehouse.   Field Lists - For more Granular parsing options:   A more advanced feature of the DBMS_CLOUD.COPY_DATA is the Field_List parameter, which borrows it’s feature set from the Field_List parameter of the Oracle Loader access driver. This parameter allows you to specify more granular information about the fields being loaded. For example, let’s use “Charlotte_NC_Weather_History_Double_Dates.csv” from the list of files in our object store. This file is similar to our first CSV example, except it has a copy of the date column in a different date format. Now, if we were to specify a date format in the format parameter, it would apply to universally to all date columns. With the field_list parameter, we can specify two different date formats for the two date columns. We do need to list all the columns and their types when including the field_list; Not mentioning any type parameters simply uses default Varchar2 values. create table WEATHER_REPORT_DOUBLE_DATE (REPORT_DATE VARCHAR2(20),     REPORT_DATE_COPY DATE,     ACTUAL_MEAN_TEMP NUMBER,     ACTUAL_MIN_TEMP NUMBER,     ACTUAL_MAX_TEMP NUMBER,     AVERAGE_MIN_TEMP NUMBER,     AVERAGE_MAX_TEMP NUMBER,     AVERAGE_PRECIPITATION NUMBER(5,2));   begin  DBMS_CLOUD.COPY_DATA(     table_name =>'WEATHER_REPORT_DOUBLE_DATE',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'&base_URL/Charlotte_NC_Weather_History_Double_Dates.csv',     format => json_object('type' value 'csv',  'skipheaders' value '1'),     field_list => 'REPORT_DATE DATE ''mm/dd/yy'',                    REPORT_DATE_COPY DATE ''yyyy-mm-dd'',                    ACTUAL_MEAN_TEMP,                   ACTUAL_MIN_TEMP,                   ACTUAL_MAX_TEMP,                   AVERAGE_MIN_TEMP,                   AVERAGE_MAX_TEMP,                   AVERAGE_PRECIPITATION'  ); end; /   SELECT * FROM WEATHER_REPORT_DOUBLE_DATE where actual_mean_temp > 69 and actual_mean_temp < 74; It's important to recognize that the date format parameters are to provide the API with the information to read the data file. The output format from your query will be your Database default (based on your NLS Parameters). This can also be formatted in your query using TO_CHAR.   JSON Files   You may be familiar with JSON files for unstructured and semi-structured data. The "PurchaseOrders.txt" file contains JSON Purhcase Order data, which when parsed and formatted looks like the following. Using JSON data in an ADW instance can be as simple as putting each JSON document into a table row as a BLOB, and using the powerful, native JSON features that the Oracle Database provides to parse and query it. You can also view the JSON documentation for additional features here.  Let’s try this! Copy and run the script below: CREATE TABLE JSON_DUMP_FILE_CONTENTS (json_document blob); begin  DBMS_CLOUD.COPY_DATA(    table_name =>'JSON_DUMP_FILE_CONTENTS',    credential_name =>'OBJ_STORE_CRED',    file_uri_list =>'&base_URL/PurchaseOrders.dmp',    field_list => 'json_document CHAR(5000)' ); end; / COLUMN Requestor FORMAT A30 COLUMN ShippingInstructions FORMAT A30 SELECT JSON_VALUE(json_document,'$.Requestor') as Requestor,        JSON_VALUE(json_document,'$.ShippingInstructions.Address.city') as ShippingInstructions FROM JSON_DUMP_FILE_CONTENTS where rownum < 50; The query above lists all the PO requestors and the city where their shipment is to be delivered. Here, we have simply created one column ‘json_document’ in the table ‘JSON_FILE_CONTENTS’. We do not incur the time it takes to validate these JSON document, and are instead directly querying the table using the Database’s JSON_VALUE feature. This means the check for well-formed JSON data will be on the fly, which would fail unless you properly skip over the failed data. Here, 'COPY_DATA' will not check for valid JSON data, but will simply check that the data is of the correct native datatype (less than 5000 characters long), that is the datatype of the table’s column. For better performance on large JSON data files, using this ADW table we can also make use of the Database’s JSON features to parse and insert the JSON data into a new table ‘j_purchaseorder’ ahead of time, as below. Note that this insert statement actually brings the data into your ADW instance. You benefit from doing this as it checks to make sure your JSON data is well-formed and valid ahead of time, and therefore incur less of a performance impact when you query this JSON data from your ADW instance. CREATE TABLE j_purchaseorder  (id          VARCHAR2 (32) NOT NULL,   date_loaded TIMESTAMP (6) WITH TIME ZONE,   po_document BLOB   CONSTRAINT ensure_json CHECK (po_document IS JSON));   INSERT INTO j_purchaseorder (id, date_loaded, po_document) SELECT SYS_GUID(), SYSTIMESTAMP, json_document FROM json_file_contents    WHERE json_document IS JSON;   We can now query down JSON paths using the JSON simplified syntax as with the following query:     SELECT po.po_document.Requestor,          po.po_document.ShippingInstructions.Address.city                  FROM j_purchaseorder po;   Beyond Copying Data into your Autonomous Data Warehouse Here, we've gone through simple examples of how to copy your Oracle object store data into your Autonomous Data Warehouse instance. In following posts, we will walk through more ways you might use to load your data, from on-premise or cloud based storage, as well as more detail on how you might troubleshoot any data loading errors you may encounter. See you in the next one!

You have finally gotten your ducks in a row to future proof your data storage, and uploaded all your necessary production data into Oracle Cloud's object store. Splendid! Now, how do you get this data...

Autonomous

Oracle Autonomous Databases - Accessing Apache Avro Files

Apache Avro is a common data format in big data solutions.  Now, these types of files are easily accessible to Oracle Autonomous Databases.  One of Avro's key benefits is that it enables efficient data exchange between applications and services.  Data storage is compact and efficient – and the file format itself supports schema evolution.  It does this by including the schema within each file – with an explanation of the characteristics of each field. In a previous post about Autonomous Data Warehouse and access parquet, we talked about using a utility called parquet-tools to review parquet files.  A similar tool – avro-tools – is available for avro files.  Using avro-tools, you can create avro files, extract the schema from a file, convert an avro file to json, and much more (check out the Apache Avro home for details).  A schema file is used to create the avro files.  This schema file describes the fields, data types and default values.  The schema becomes part of the generated avro file – which allows applications to read the file and understand its contents.  Autonomous Database uses this schema to automate table creation.  Similar to parquet sources, Autonomous Database will read the schema to create the columns with the appropriate Oracle Database data types.  Avro files may include complex types – like arrays, structs, maps and more; Autonomous Database supports Avro files that contain Oracle data types. Let’s take a look at an example.  Below, we have a file - movie.avro - that contains information about movies (thanks to Wikipedia for providing info about the movies).  We’ll use the avro-tools utility to extract the schema: $ avro-tools getschema movie.avro {   "type" : "record",   "name" : "Movie",   "namespace" : "oracle.avro",   "fields" : [ {     "name" : "movie_id",     "type" : "int",     "default" : 0   }, {     "name" : "title",     "type" : "string",     "default" : ""   }, {     "name" : "year",     "type" : "int",     "default" : 0   }, {     "name" : "budget",     "type" : "int",     "default" : 0   }, {     "name" : "gross",     "type" : "double",     "default" : 0   }, {     "name" : "plot_summary",     "type" : "string",     "default" : ""   } ] } The schema is in an easy to read JSON format.  Here, we have movie_id, , title, year, budget, gross and plot_summary columns. The data has been loaded into an Oracle Object Store bucket called movies.  The process for making this data available to ADW is identical to the steps for parquet – so check out that post for details.  At a high level, you will: Create a credential that is used to authorize access to the object store bucket Create an external table using dbms_cloud.create_external_table. Query the data! 1.  Create the credential begin   DBMS_CLOUD.create_credential (     credential_name => 'OBJ_STORE_CRED',     username => '<user>',     password => '<password>'   ) ; end; / 2.  Create the table begin     dbms_cloud.create_external_table (     table_name =>'movies_ext',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'https://objectstorage.ca-toronto-1.oraclecloud.com/n/<tenancy>/b/<bucket>/o/*',     format =>  '{"type":"avro",  "schema": "first"}'     ); end; / Things got a little easier when specifying the URI list.  Instead of transforming the URI into a specific format, you can use the same path that is found in the OCI object browser: Object Details -> URL Path (accessed from the OCI Console:  Oracle Cloud -> Object Storage – “movies” bucket): This URL Path was specified in the file-uri-list parameter – although a wildcard was used instead of the specific file name. Now that the table is created, we can look at its description: SQL> desc movies_ext Name         Null? Type          ------------ ----- -------------- MOVIE_ID           NUMBER(10)    TITLE              VARCHAR2(4000) YEAR               NUMBER(10)    BUDGET             NUMBER(10)    GROSS              BINARY_DOUBLE PLOT_SUMMARY       VARCHAR2(4000) And, run queries against that table: SQL> select title, year, budget, plot_summary from movies_ext where title like 'High%'; All of this is very similar to parquet – especially from a usability standpoint.  With both file formats, the metadata in the file is used to automate the creation of tables.  However, there is a significant difference when it comes to processing the data.  Parquet is a columnar format that has been optimized for queries.  Column projection and predicate pushdown is used to enhance performance by minimizing the amount of data that is scanned and subsequently transferred from the object store to the database.  The same is not true for Avro; the entire file will need to be scanned and processed.  So, if you will be querying this data frequently – consider alternative storage options and use tools like dbms_cloud.copy_data to easily load the data into Autonomous Database.

Apache Avro is a common data format in big data solutions.  Now, these types of files are easily accessible to Oracle Autonomous Databases.  One of Avro's key benefits is that it enables efficient...

Autonomous

How To Update The License Type For Your Autonomous Database

How To Update The License Type For Your Autonomous Database As of today (Wednesday April 17) you can now quickly and easily change the type licensing for your Autonomous Database from BYOL to a new cloud subscription or vice-versa. It’s as easy as 1-2-3. So assuming you already have already created an autonomous database instance, how do you go about changing your licensing? Let me show you! Step 1 - Accessing the management console First stage involves signing into your cloud account using your tenancy name and cloud account. Then you can navigate to either your “Autonomous Data Warehouse” or “Autonomous Transaction Processing” landing pad as shown below: Let’s now change the type of license for the autonomous database instance “pp1atpsep2”. If we click on the blue text of the instance name which is in the first column of the table, this will take us to the instance management console page as shown below: Notice in the above image that, on the righthand side, the console shows the current license type as set to "Bring Your Own License" which is often referred to as a BYOL-license. Step 2 - Selecting “Update License Type” from Action menu Now click on the “Actions” button in the row of menu buttons as shown below:   Step 3 - Change the “License Type” The pop-up form shows the current type of license associated with our autonomous database instance “pp1atpsep2”, which in this case is set to BYOL If you want more information about what is and is not covered within a BYOL-license then visit the BYOL FAQ page which is here. In this case we are going to flip to using a new cloud subscription, as shown below: That's it! All that's left to do is click on the blue Update button and the new licensing model will be applied to our autonomous database instance. At this point your autonomous database will switch into “Updating” mode, as shown below. However, the database is still up and accessible. There is no downtime. When the update is complete the status will return to “Available” and the console will show that the license type has changed to "License Included" as shown below. Summary Congratulations you have successfuly swapped your BYOL license to a new cloud subscription license for your autonomous database with absolutely no downtime or impact on your users. In this post I have showed you how to quickly and easily change the type of license associated with your autonomous database. An animation of the complete end-to-end process is shown below:   Featured Neon "Change" image courtesy of wikipedia  

How To Update The License Type For Your Autonomous Database As of today (Wednesday April 17) you can now quickly and easily change the type licensing for your Autonomous Database from BYOL to a new...

Hadoop Best Practices

Big Data. See How Easily You Can Do Disaster Recovery

Earlier I've written about Big Data High Availability in different aspects and I intentionally avoided the Disaster Recovery topic. High Availability answers on the question how system should process in case of failure one of the component (like Name Node or KDC) within one system (like one Hadoop Cluster), Disaster Recovery answers on the question what to do in case if entire system will fail (Hadoop cluster or even Data Center will go down). In this blog, I also would like to talk about backup and how to deal with human mistakes (it's not particularly DR topics, but quite close).  Also, I'd like to introduce few terms. From Wikipedia: Business Continuity:  Involveskeeping all essential aspects of a business functioning despite significant disruptive events.  Disaster Recovery: (DR)  A set of policies and procedures to enable the recovery or continuation of vital technology infrastructure and systems following a disaster. Step 1. Protect system from human errors. HDFS snapshots. HDFS snapshots functionality has been a while in Hadoop portfolio. This is a great way to protect system from human mistakes. There are few simple steps to enable it (full snapshot documentation you could find here).  - go to Cloudera Manager and drill down into the HDFS service: - then go to the "File Browser" and navigate to the directory, which you would like to protect by snapshots - click on the "Enable Snapshots" button: as soon as command finished, you have directory, protected by snapshots! you may take snapshots on demand: or you may create a snapshot policy, which will be periodically repeated (it's recommended). In order to make it work you have to go to a Cloudera Manager -> Backup -> Snapshot Policies:   - Click on the "Create Policy" (Note: you have to enable Snapshots for certain directory before creating policy) - and fill up the form: easy, but very powerful. It's a good time for a demo. Let's imagine, that we have directory with critical datasets on HDFS: [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - bdruser supergroup          0 2019-02-13 14:12 /tmp/snapshot_demo/dir1 drwxr-xr-x   - bdruser supergroup          0 2019-02-13 14:12 /tmp/snapshot_demo/dir2   then occasionally user deleted one of the directory: [root@destination_cluster15 ~]# hadoop fs -rm -r -skipTrash /tmp/snapshot_demo/dir1 Deleted /tmp/snapshot_demo/dir1 [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 1 items drwxr-xr-x   - bdruser supergroup          0 2019-02-13 14:12 /tmp/snapshot_demo/dir2 fortunately, it's quite easy to restore state of this dir using snapshots: - go to a Cloudera Manager -> HDFS -> File Browser - choose option "Restore from snapshot": - choose appropriate snapshot and click "Restore": - check what you have: [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:32 /tmp/snapshot_demo/dir1 drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:32 /tmp/snapshot_demo/dir2 Note: snapshot revert you to the stage where you've made it. For example, if you add some directory and then restore to a snapshot, you will not have this directory, which I've created after taking snapshot: [root@destination_cluster15 ~]# hadoop fs -mkdir /tmp/snapshot_demo/dir3 [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 3 items drwxr-xr-x   - hdfs    supergroup          0 2019-02-13 14:32 /tmp/snapshot_demo/dir1 drwxr-xr-x   - hdfs    supergroup          0 2019-02-13 14:32 /tmp/snapshot_demo/dir2 drwxr-xr-x   - bdruser supergroup          0 2019-02-13 14:35 /tmp/snapshot_demo/dir3   and restore from the early taken snapshot: after recovery done: [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:36 /tmp/snapshot_demo/dir1 drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:36 /tmp/snapshot_demo/dir2   I have only two directories.  Another one common case - when user change file permissions or file owner by accident and want to return it back: [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:36 /tmp/snapshot_demo/dir1 drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:36 /tmp/snapshot_demo/dir2 [root@destination_cluster15 ~]# hadoop fs -chown yarn:yarn /tmp/snapshot_demo/* [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - yarn yarn          0 2019-02-13 14:36 /tmp/snapshot_demo/dir1 drwxr-xr-x   - yarn yarn          0 2019-02-13 14:36 /tmp/snapshot_demo/dir2 restore from snapshot and have previous file owner: [root@destination_cluster15 ~]# hadoop fs -ls /tmp/snapshot_demo/ Found 2 items drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:38 /tmp/snapshot_demo/dir1 drwxr-xr-x   - hdfs supergroup          0 2019-02-13 14:38 /tmp/snapshot_demo/dir2 Conclusion: snapshots is very powerful tool for protect your file system from human mistakes. It stores only delta (changes), so that means that it will not consume many space in case if you don't delete data frequently. Step 2.1. Backup data. On-premise backup. NFS Storage. Backups in Hadoop world is ticklish topic. The reason is time to recovery. How long will it take to bring data back to the production system? Big Data systems tend to be not so expensive and have massive datasets, so it may be easier to have second cluster (details on how to do this coming later in this blog). But if you have some reasons to do backups, you may consider either NFS storage (in case if you want to take backup on-premise in your datacenter) or Object Store (if you want to take backup outside of your data center) in Oracle Cloud Infrastructure (OCI) as an options. In case of NFS storage (like Oracle ZFS), you have to mount your NFS storage at the same directory on every Hadoop node. Like this: Run on each BDA node: [root]#  mount nfs_storage_ip:/stage/files /tmp/src_srv Now you have share storage on every server and it means that every single Linux server has the same directory. It allows you to run distcp command (that originally was developed for coping big amount of data between HDFS filesystems). For start parallel copy, just run: $ hadoop distcp -m 50 -atomic hdfs://nnode:8020/tmp/test_load/* file:///tmp/src_srv/files/; You will create MapReduce job that will copy from one place (local file system) to HDFS with 50 mappers. Step 2.2. Backup data. Cloud. Object Storage. Object Store is key element for every cloud provider, oracle is not an exception. Documentation for Oracle Object Storage you could find here. Object Store provides some benefits, such as: - Elasticity. Customers don't have to plan ahead how many space to they need. Need some extra space? Simply load data into Object Store. There is no difference in process to copy 1GB or 1PB of data - Scalability. It's infinitely scale. At least theoretically :) - Durability and Availability. Object Store is first class citizen in all Cloud Stories, so all vendors do all their best to maintain 100% availability and durability. If some diet will go down, it shouldn't worry you. If some node with OSS software will go down, it shouldn't worry you. As user you have to put data there and read data from Object Store.  - Cost. In a Cloud Object Store is most cost efficient solution. Nothing comes for free and as downside I may highlight: - Performance in comparison with HDFS or local block devices. Whenever you read data from Object Store, you read it over the network. - Inconsistency of performance. You are not alone on object store and obviously under the hood it uses physical disks, which have own throughput. If many users will start to read and write data to/from Object Store, you may get performance which is different with what you use to have a day, week, month ago - Security. Unlike filesystems Object Store has not file grain permissions policies and customers will need to reorganize and rebuild their security standards and policies. Before running backup, you will need to configure OCI Object Store in you Hadoop system.  After you config your object storage, you may check the bucket that you've intent to copy to: [root@source_clusternode01 ~]# hadoop fs -ls oci://BDAx7Backup@oraclebigdatadb/   Now you could trigger actual copy by running either distcp: [root@source_clusternode01 ~]# hadoop distcp -Dmapred.job.queue.name=root.oci -Dmapreduce.task.timeout=6000000 -m 240 -skipcrccheck -update -bandwidth 10240 -numListstatusThreads 40 /user/hive/warehouse/parq.db/store_returns oci://BDAx7Backup@oraclebigdatadb/ or ODCP - oracle build tool. You could find more info about ODCP here.   [root@source_clusternode01 ~]# odcp --executor-cores 3 --executor-memory 9 --num-executors 100  hdfs:///user/hive/warehouse/parq.db/store_sales oci://BDAx7Backup@oraclebigdatadb/   and after copy done, you will be able to see all your data in OCI Object Store bucket:   Step 2.3. Backup data. Big Data Appliance metadata. it's easies section for me because Oracle support engineers made a huge effort writing support note, which tells customer on how to take backups of metadata. For more details please refer to: Customer RecommendedHow to Backup Critical Metadata on Oracle Big Data Appliance Prior to Upgrade V2.3.1 and Higher Releases (Doc ID 1623304.1)  Step 2.4. Backup data. MySQL Separately, I'd like to mention that MySQL backup is very important and you could get familiar with this here: How to Redirect a MySQL Database Backup on BDA Node 3 to a Different Node on the BDA or on a Client/Edge Server (Doc ID 1926159.1) Step 3. HDFS Disaster Recovery. Recommended Architecture Here I'd like to share Oracle recommended architecture for Disaster Recovery setup: We do recommend to have same Hardware and Software environment for Production and DR environments. If you want to have less powerful nodes on the DR side, you should answer to yourself on the question - what you are going to do in case of disaster? What is going to happen if you will switch all production applications to the DR side. Will it be capable to handle this workload? Also, one very straight recommendation from Oracle is to have small BDA (3-6 nodes) in order to perform tests  on it. Here is the rough separation of duties for these three clusters: Production (Prod): - Running production workload Disaster Recovery (DR): - Same (or almost the same) BDA hardware configuration - Run non-critical Ad-Hoc queries - Switch over in case of unplanned (disaster) or planned (upgrade) outages of the prod Test: Use small BDA cluster (3-6 nodes) to test different things, such as: - Upgrade - Change settings (HDFS, YARN) - Testing of new engines (add and test Hbase, Flume..) - Testing integration with other systems (AD, Database) - Test Dynamic pools … Note: for test environment you also consider Oracle cloud offering Step 3.1 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Now we are approaching most interesting part of the blog. Disaster recovery.  Cloudera offers the tool out of the box, which called Big Data Disaster Recovery (BDR), which allows to Hadoop administrators easily, using web interface create replication policies and schedule the data replication. Let me show the example how to do this replication with BDR. I have two BDA clusters source_cluster and destination_cluster. under the hood BDR uses special version of distcp, which has many performance and functional optimizations. Step 3.1.1 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Network throughput It's crucial to understand network throughput between clusters. To measure network throughput you may use any tool convenient for you. I personally prefer iperf. Note: iperf has two modes - UDP and TCP. Use TCP in order to make measurements between servers in context of BDR, because it uses TCP connections. After installation it's quite easy to run it: On probation make one machine (let's say on destination cluster) as server and run iperf in listening mode: [root@destination_cluster15 ~]# iperf -s on the source machine run client command, which will send some TCP traffic to server machine for 10 minutes with maximum bandwidth: [root@source_clusternode01 ~]# iperf -c destination_cluster15 -b 10000m -t 600   after getting this numbers you may understand what could you count on when you will run the copy command.   Step 3.1.2 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Ports. it's quite typical when different Hadoop clusters are in different data centers over firewalls. Before start running  BDR jobs, please make sure, that all necessary ports are open from both sides.   Step 3.1.3 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Kerberos Naming Recommendations. We (as well as Cloudera) generally recommend you use different KDC realms with trust between them and different realm names for each cluster. All user principals obtain their credentials from AD, MIT KDC will store service principals. More details on BDA security good practices you could find here. I'll assume that both clusters are kerberized (kerberos is almost default nowadays) and we will need to do some config around this. Detailed steps on how to setup Kerberos setting for two clusters, which use different KDCs you could find here. If you want to know how to set up trusted relationships between clusters you could refer here. I just briefly want to highlight most important steps. 1) On the source cluster go to "Administration -> Settings -> Search for ("KDC Server Host") and set up hostname for Source KDC. Do the same for "KDC Admin Server Host". It's important because when destination cluster comes to the source and ask for KDC it does not read the /etc/krb5.conf as you may think. It read KDC address from this property. 2) Both clusters are in the same domain. It's quite probable and quite often case. You may conclude this by seen follow error message: "Peer cluster has domain us.oracle.com and realm ORACLE.TEST but a mapping already exists for this domain us.oracle.com with realm US.ORACLE.COM. Please use hostname(s) instead of domain(s) for realms US.ORACLE.COM and ORACLE.TEST, so there are no conflicting domain to realm mapping." it's easy to fix by adding in /etc/krb5.conf exact names of the hosts under "domain_realm" section: [domain_realm] source_clusternode01.us.oracle.com = ORACLE.TEST source_clusternode02.us.oracle.com = ORACLE.TEST source_clusternode03.us.oracle.com = ORACLE.TEST source_clusternode04.us.oracle.com = ORACLE.TEST source_clusternode05.us.oracle.com = ORACLE.TEST source_clusternode06.us.oracle.com = ORACLE.TEST destination_cluster13.us.oracle.com = US.ORACLE.COM destination_cluster14.us.oracle.com = US.ORACLE.COM destination_cluster13.us.oracle.com = US.ORACLE.COM .us.oracle.com = US.ORACLE.COM us.oracle.com = US.ORACLE.COM   Note: here you do Host-Realm mapping, because I have two different REALMs and two different KDCs, but only one domain. In case if I'll use any host outside of the given list, I need to specify default realm for the domain (last two rows)   3) at a destination cluster add REALM for the source cluster in /etc/krb5.conf: [root@destination_cluster13 ~]# cat /etc/krb5.conf ... [realms]  US.ORACLE.COM = {   kdc = destination_cluster13.us.oracle.com:88   kdc = destination_cluster14.us.oracle.com:88   admin_server = destination_cluster13.us.oracle.com:749   default_domain = us.oracle.com  } ORACLE.TEST = { kdc = source_clusternode01.us.oracle.com admin_server = source_clusternode01.us.oracle.com default_domain = us.oracle.com } ... try to obtain credentials and explore source Cluster HDFS: [root@destination_cluster13 ~]# kinit oracle@ORACLE.TEST Password for oracle@ORACLE.TEST:  [root@destination_cluster13 ~]# klist  Ticket cache: FILE:/tmp/krb5cc_0 Default principal: oracle@ORACLE.TEST   Valid starting     Expires            Service principal 02/04/19 22:47:42  02/05/19 22:47:42  krbtgt/ORACLE.TEST@ORACLE.TEST     renew until 02/11/19 22:47:42 [root@destination_cluster13 ~]# hadoop fs -ls hdfs://source_clusternode01:8020 19/02/04 22:47:54 WARN security.UserGroupInformation: PriviledgedActionException as:oracle@ORACLE.TEST (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]   it fails, but it's not a big surprise - Cluster don't have trusted relationships. let's fix this.  Note: Big Data Disaster Recovery doesn't require trusted kerberos relationships between clusters (distcp does), but in order to make it easier to debug and to some other operation activities, I'd recommend to add it on. On the destination cluster: [root@destination_cluster13 ~]# kadmin.local  kadmin.local:  addprinc krbtgt/ORACLE.TEST@US.ORACLE.COM WARNING: no policy specified for krbtgt/ORACLE.TEST@US.ORACLE.COM; defaulting to no policy Enter password for principal "krbtgt/ORACLE.TEST@US.ORACLE.COM":  Re-enter password for principal "krbtgt/ORACLE.TEST@US.ORACLE.COM":  Principal "krbtgt/ORACLE.TEST@US.ORACLE.COM" created. on the source Cluster: [root@source_clusternode01 ~]# kadmin.local  kadmin.local:  addprinc krbtgt/ORACLE.TEST@US.ORACLE.COM WARNING: no policy specified for krbtgt/ORACLE.TEST@US.ORACLE.COM; defaulting to no policy Enter password for principal "krbtgt/ORACLE.TEST@US.ORACLE.COM":  Re-enter password for principal "krbtgt/ORACLE.TEST@US.ORACLE.COM":  Principal "krbtgt/ORACLE.TEST@US.ORACLE.COM" created.   make sure that you create same user within the same passwords on both KDCs. try to explore destination's HDFS: [root@destination_cluster13 ~]# hadoop fs -ls hdfs://source_clusternode01:8020 Found 4 items drwx------   - hbase hbase               0 2019-02-04 22:34 /hbase drwxr-xr-x   - hdfs  supergroup          0 2018-03-14 06:46 /sfmta drwxrwxrwx   - hdfs  supergroup          0 2018-10-31 15:41 /tmp drwxr-xr-x   - hdfs  supergroup          0 2019-01-07 09:30 /user Bingo! it works. Now we have to do the same on both clusters to allow reverse direction: 19/02/05 02:02:02 INFO util.KerberosName: No auth_to_local rules applied to oracle@US.ORACLE.COM 19/02/05 02:02:03 WARN security.UserGroupInformation: PriviledgedActionException as:oracle@US.ORACLE.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] same error and same fix for it. I just simply automate this by running follow commands on the both KDCs: delprinc -force krbtgt/US.ORACLE.COM@ORACLE.TEST delprinc -force krbtgt/ORACLE.TEST@US.ORACLE.COM addprinc -pw "welcome1" krbtgt/US.ORACLE.COM@ORACLE.TEST addprinc -pw "welcome1" krbtgt/ORACLE.TEST@US.ORACLE.COM   4) make sure that in /var/kerberos/krb5kdc/kdc.conf you have: default_principal_flags = +renewable, +forwardable   Step 3.1.4 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). SSL The next assumption is that your's Cloudera manager is working over encrypted channel and if you will try to do add source peer, most probably, you'll get an exception:   in order to fix this: a. Check certificate for Cloudera Manager (run this command on the destination cluster): [root@destination_cluster13 ~]# openssl s_client -connect source_clusternode05.us.oracle.com:7183 CONNECTED(00000003) depth=0 C = , ST = , L = , O = , OU = , CN = source_clusternode05.us.oracle.com verify error:num=18:self signed certificate verify return:1 depth=0 C = , ST = , L = , O = , OU = , CN = source_clusternode05.us.oracle.com verify return:1 --- Certificate chain  0 s:/C=/ST=/L=/O=/OU=/CN=source_clusternode05.us.oracle.com    i:/C=/ST=/L=/O=/OU=/CN=source_clusternode05.us.oracle.com --- Server certificate -----BEGIN CERTIFICATE----- MIIDYTCCAkmgAwIBAgIEP5N+XDANBgkqhkiG9w0BAQsFADBhMQkwBwYDVQQGEwAx CTAHBgNVBAgTADEJMAcGA1UEBxMAMQkwBwYDVQQKEwAxCTAHBgNVBAsTADEoMCYG A1UEAxMfYmRheDcyYnVyMDlub2RlMDUudXMub3JhY2xlLmNvbTAeFw0xODA3MTYw MzEwNDVaFw0zODA0MDIwMzEwNDVaMGExCTAHBgNVBAYTADEJMAcGA1UECBMAMQkw BwYDVQQHEwAxCTAHBgNVBAoTADEJMAcGA1UECxMAMSgwJgYDVQQDEx9iZGF4NzJi dXIwOW5vZGUwNS51cy5vcmFjbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAkLwi9lAsbiWPVUpQNAjtGE5Z3pJOExtJMSuvnj02FC6tq6I09iJ0 MsTu6+Keowv5CUlhfxTy1FD19ZhX3G7OEynhlnnhJ+yjprYzwRDhMHUg1LtqWib/ osHR1QfcDfLsByBKO0WsLBxCz/+OVm8ZR+KV/AeZ5UcIsvzIRZB4V5tWP9jziha4 3upQ7BpSvQhd++eFb4wgtiBsI8X70099ZI8ctFpmPjxtYHQSGRGdoZZJnHtPY4IL Vp0088p+HeLMcanxW7CSkBZFn9nHgC5Qa7kmLN4EHhjwVfPCD+luR/k8itH2JFw0 Ub+lCOjSSMpERlLL8fCnETBc2nWCHNQqzwIDAQABoyEwHzAdBgNVHQ4EFgQUkhJo 0ejCveCcbdoW4+nNX8DjdX8wDQYJKoZIhvcNAQELBQADggEBAHPBse45lW7TwSTq Lj05YwrRsKROFGcybpmIlUssFMxoojys2a6sLYrPJIZ1ucTrVNDspUZDm3WL6eHC HF7AOiX4/4bQZv4bCbKqj4rkSDmt39BV+QnuXzRDzqAxad+Me51tisaVuJhRiZkt AkOQfAo1WYvPpD6fnsNU24Tt9OZ7HMCspMZtYYV/aw9YdX614dI+mj2yniYRNR0q zsOmQNJTu4b+vO+0vgzoqtMqNVV8Jc26M5h/ggXVzQ/nf3fmP4f8I018TgYJ5rXx Kurb5CL4cg5DuZnQ4zFiTtPn3q5+3NTWx4A58GJKcJMHe/UhdcNvKLA1aPFZfkIO /RCqvkY= -----END CERTIFICATE----- b. Go to the source cluster and find a file which has this certificate (run this command on the source cluster): [root@source_clusternode05 ~]# grep -iRl "MIIDYTCCAkmgAwIBAgIEP5N+XDANBgkqhkiG9w0BAQsFADBhMQkwBwYDVQQGEwAx" /opt/cloudera/security/|grep "cert" /opt/cloudera/security/x509/node.cert /opt/cloudera/security/x509/ssl.cacerts.pem   c. Make sure that each node has different certificate by calculating the hash (run this command on the source cluster): [root@source_clusternode05 ~]# dcli -C "md5sum /opt/cloudera/security/x509/node.cert" 192.168.8.170: cc68d7f5375e3346d312961684d728c0  /opt/cloudera/security/x509/node.cert 192.168.8.171: 9259bb0102a1775b164ce56cf438ed0e  /opt/cloudera/security/x509/node.cert 192.168.8.172: 496fd4e12bdbfc7c6aab35d970429a72  /opt/cloudera/security/x509/node.cert 192.168.8.173: 8637b8cfb5db843059c7a0aeb53071ec  /opt/cloudera/security/x509/node.cert 192.168.8.174: 4aabab50c256e3ed2f96f22a81bf13ca  /opt/cloudera/security/x509/node.cert 192.168.8.175: b50c2e40d04a026fad89da42bb2b7c6a  /opt/cloudera/security/x509/node.cert [root@source_clusternode05 ~]#    d. rename this certificates (run this command on the source cluster):   [root@source_clusternode05 ~]# dcli -C cp /opt/cloudera/security/x509/node.cert /opt/cloudera/security/x509/node_'`hostname`'.cert e. Check the new names (run this command on the source cluster): [root@source_clusternode05 ~]# dcli -C "ls /opt/cloudera/security/x509/node_*.cert" 192.168.8.170: /opt/cloudera/security/x509/node_source_clusternode01.us.oracle.com.cert 192.168.8.171: /opt/cloudera/security/x509/node_source_clusternode02.us.oracle.com.cert 192.168.8.172: /opt/cloudera/security/x509/node_source_clusternode03.us.oracle.com.cert 192.168.8.173: /opt/cloudera/security/x509/node_source_clusternode04.us.oracle.com.cert 192.168.8.174: /opt/cloudera/security/x509/node_source_clusternode05.us.oracle.com.cert 192.168.8.175: /opt/cloudera/security/x509/node_source_clusternode06.us.oracle.com.cert f. Pull those certificates from source cluster to one node of the destination cluster (run this command on the destination cluster): [root@destination_cluster13 ~]# for i in {1..6}; do export NODE_NAME=source_clusternode0$i.us.oracle.com; scp root@$NODE_NAME:/opt/cloudera/security/x509/node_$NODE_NAME.cert /opt/cloudera/security/jks/node_$NODE_NAME.cert; done; g. propagate it on the all nodes of the destination cluster (run this command on the destination cluster): [root@destination_cluster13 ~]# for i in {4..5}; do scp /opt/cloudera/security/jks/node_bda*.cert root@destination_cluster1$i:/opt/cloudera/security/jks; done;   h. on the destination host option truster password and trustore location (run this command on the destination cluster): [root@destination_cluster13 ~]# bdacli getinfo cluster_https_truststore_path Enter the admin user for CM (press enter for admin):  Enter the admin password for CM:  /opt/cloudera/security/jks/cdhs49.truststore   [root@destination_cluster13 ~]# bdacli getinfo cluster_https_truststore_password Enter the admin user for CM (press enter for admin):  Enter the admin password for CM:  dl126jfwt1XOGUlNz1jsAzmrn1ojSnymjn8WaA7emPlo5BnXuSCMtWmLdFZrLwJN i. and add them environment variables on all hosts of the destination cluster (run this command on the destination cluster): [root@destination_cluster13 ~]# export TRUSTORE_PASSWORD=dl126jfwt1XOGUlNz1jsAzmrn1ojSnymjn8WaA7emPlo5BnXuSCMtWmLdFZrLwJN [root@destination_cluster13 ~]# export TRUSTORE_FILE=/opt/cloudera/security/jks/cdhs49.truststore j. now we are ready to copy add certificates to the destination clusters trustore (run this command on the destination cluster) do this on all hosts of the destination cluster: [root@destination_cluster13 ~]# for i in {1..6}; do export NODE_NAME=export NODE_NAME=source_clusternode0$i.us.oracle.com; keytool -import -noprompt  -alias $NODE_NAME -file /opt/cloudera/security/jks/node_$NODE_NAME.cert -keystore $TRUSTORE_FILE -storepass $TRUSTORE_PASSWORD; done; Certificate was added to keystore Certificate was added to keystore Certificate was added to keystore Certificate was added to keystore Certificate was added to keystore   k. to validate that we add it, run (run this command on the destination cluster): [root@destination_cluster13 ~]# keytool -list -keystore $TRUSTORE_FILE -storepass $TRUSTORE_PASSWORD Keystore type: jks Keystore provider: SUN   Your keystore contains 9 entries   destination_cluster14.us.oracle.com, May 30, 2018, trustedCertEntry,  Certificate fingerprint (SHA1): B3:F9:70:30:77:DE:92:E0:A3:20:6E:B3:96:91:74:8E:A9:DC:DF:52 source_clusternode02.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): 3F:6E:B9:34:E8:F9:0B:FF:CF:9A:4A:77:09:61:E9:07:BF:17:A0:F1 source_clusternode05.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): C5:F0:DB:93:84:FA:7D:9C:B4:C9:24:19:6F:B3:08:13:DF:B9:D4:E6 destination_cluster15.us.oracle.com, May 30, 2018, trustedCertEntry,  Certificate fingerprint (SHA1): EC:42:B8:B0:3B:25:70:EF:EF:15:DD:E6:AA:5C:81:DF:FD:A2:EB:6C source_clusternode03.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): 35:E1:07:F0:ED:D5:42:51:48:CB:91:D3:4B:9B:B0:EF:97:99:87:4F source_clusternode06.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): 16:8E:DF:71:76:C8:F0:D3:E3:DF:DA:B2:EC:D5:66:83:83:F0:7D:97 destination_cluster13.us.oracle.com, May 30, 2018, trustedCertEntry,  Certificate fingerprint (SHA1): 76:C4:8E:82:3C:16:2D:7E:C9:39:64:F4:FC:B8:24:40:CD:08:F8:A9 source_clusternode01.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): 26:89:C2:2B:E3:B8:8D:46:41:C6:C0:B6:52:D2:C4:B8:51:23:57:D2 source_clusternode04.us.oracle.com, Feb 1, 2019, trustedCertEntry,  Certificate fingerprint (SHA1): CB:98:23:1F:C0:65:7E:06:40:C4:0C:5E:C3:A9:78:F3:9D:E8:02:9E [root@destination_cluster13 ~]#  l. now do the same on the others node of destination cluster   Step 3.1.5 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Create replication user On the destination cluster you will need to configure replication peer. [root@destination_cluster15 ~]# dcli -C "useradd bdruser -u 2000" [root@destination_cluster15 ~]# dcli -C "groupadd supergroup -g 2000" [root@destination_cluster15 ~]# dcli -C "usermod -g supergroup bdruser" and after this verify that this user belongs to the supergroup: [root@destination_cluster15 ~]# hdfs groups bdruser bdruser : supergroup Step 3.1.6 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Create separate job for encrypted zones It's possible to copy data from encrypted zone, but there is the trick with it. If you will try to do this, you will find the error in the BDR logs: java.io.IOException: Checksum mismatch between hdfs://distcpSourceNS/tmp/EZ/parq.db/customer/000001_0 and hdfs://cdhs49-ns/tmp/EZ/parq.db/customer/.distcp.tmp.4101922333172283041 Fortunately, this problem could easily be solved. You just need to skip calculating checksums for Encrypted Zones: This is a good practice to create separate Job to copy data from encrypted zones and exclude directories with Encryption from general backup job. Example. You have some directory, which you want to exclude (/tmp/excltest/bad) from common copy job. For do this, you need go to "Advanced" settings and add "Path Exclusion": In my example you need to put .*\/tmp\/excltest\/bad+.* you may this regexp it by creating follow directory structure and add Path Exclusion. [root@source_clusternode05 ~]# hadoop fs -mkdir /tmp/excltest/ [root@source_clusternode05 ~]# hadoop fs -mkdir /tmp/excltest/good1 [root@source_clusternode05 ~]# hadoop fs -mkdir /tmp/excltest/good2 [root@source_clusternode05 ~]# hadoop fs -mkdir /tmp/excltest/bad Note: it maybe quite hard to create and validate regular expression (this is Java), for this purposes you may use this on-line resource. Step 3.1.7 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Enable Snapshots on the Source Cluster Replication without snapshots may fail. Distcp automatically created snapshot before coping. Some replications, especially those that require a long time to finish, can fail because source files are modified during the replication process. You can prevent such failures by using Snapshots in conjunction with Replication. This use of snapshots is automatic with CDH versions 5.0 and higher. To take advantage of this, you must enable the relevant directories for snapshots (also called making the directory snapshottable).  When the replication job runs, it checks to see whether the specified source directory is snapshottable. Before replicating any files, the replication job creates point-in-time snapshots of these directories and uses them as the source for file copies.  What happens when you copy data with out snapshots. test case: 1) Start coping (decent amount of data) 2) in the middle of the copy process, delete from the source files 3) get an error: ERROR distcp.DistCp: Job failed to copy 443 files/dirs. Please check Copy Status.csv file or Error Status.csv file for error messages INFO distcp.DistCp: Used diff: false WARN distcp.SnapshotMgr: No snapshottable directories have found. Reason: either run-as-user does not have permissions to get snapshottable directories or source path is not snapshottable. ERROR org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /bdax7_4/store_sales/000003_0 to overcome this do: 1) On the source go to CM -> HDFS -> File Browser, pick right directory and click on: 2) after this when you will run the job, it automatically will take a snapshot and will copy from it: 3) if you will delete data, your copy job will finish it from the snapshot that it took. Note, you can't delete entire directory, but you could delete all files from it: [root@destination_cluster15 ~]# hadoop fs -rm -r -skipTrash /bdax7_4 rm: The directory /bdax7_4 cannot be deleted since /bdax7_4 is snapshottable and already has snapshots [root@destination_cluster15 ~]# hadoop fs -rm -r -skipTrash /bdax7_4/* Deleted /bdax7_4/new1.file Deleted /bdax7_4/store_sales Deleted /bdax7_4/test.file [root@destination_cluster15 ~]# hadoop fs -ls /bdax7_4 [root@destination_cluster15 ~]# 4) copy is successfully completed! Step 3.1.8 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Rebalancing data HDFS tries to keep data evenly distributed across all nodes in a cluster. But after intensive write it maybe useful to run rebalance. Default rebalancing threshold is 10%, which is a bit high. It make sense to change "Rebalancing Threshold" from 10 to 2 (Cloudera Manager -> HDFS -> Instances -> Balancer -> configuration) also, in order to speed up rebalance speed, we could increase value of "dfs.datanode.balance.max.concurrent.moves" from 10 to 1000 (Number of block moves to permit in parallel). After make this changes, save it and run rebalancing: In case of heavily used clusters for reading/writing/deleting HDFS data we may hit disbalance within one node (when data unevenly distributed across multiple disks within one node). Here is the Cloudera Blog about it. Shortly, we have to go to "Cloudera Manager - > HDFS -> Configuration -> HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" and add dfs.disk.balancer.enabled as a name and true as the value. Sometimes, you may have real data skew problem, which could easily be fixed by running rebalance: Note: If you want to visualize data distribution, you could check this tool developed at CERN. Step 3.1.9 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Hive There is also some option for coping Hive Metadata and as well as actual Data. Note: if you would like to copy some particular database schema you need to specify it in the copy wizard as well as specify regular expression which tables you want to copy ([\w].+ for all tables).  For example, this replication policy will copy all tables from database "parq". if you will leave it blank, you will not copy anything. More examples of regular expression you could find here. 3.1.10 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Sentry Sentry is the default authorization mechanism in Cloudera and you may want to replicate authorization rules on the second cluster. Unfortunately, there is no any mechanism embedded into BDR and you have to come with own solution.  Note: you have to configure Sentry in certain way to make it work with BDR. Please refer here to get more details. Step 3.1.11 HDFS Disaster Recovery. Big Data Disaster Recovery (BDR). Advantages and Disadvantages of BDR Generally speaking, I could to recommend use BDR as long as it meet your needs and requirements. Here is a brief summary of advantages and disadvantages of BDR. Advantages (+): - It's available out of the box, no need to install - It's free of charge - It's relative easy to configure and start working with basic examples Disadvantages (-): - It's not real-time tool. User have to schedule batch jobs, which will run every time period - There is no transparent way to fail over. In case of failing of primary side, uses have to manually switch over their applications into the new cluster - BDR (distcp under the cover) is MapReduce job, which takes significant resources. - Because of previous one and because of MR nature in case of coping one big file it will not be parallelized (will be copied in one thread) - Hive changes not fully replicated (drop tables have to be backup manually) - In case of replication big number of files (or Hive table with big number of partitions) it takes long time to finish. I can say that it's near to impossible to replicate directory if it has around 1 million objects (files/directories)  - It's only Cloudera to Cloudera or Cloudera to Object Store copy. No way copy to Hortonworks (but after merging this companies it's not a huge problem anymore) Step 3.2 HDFS Disaster Recovery. Wandisco If you met one of the challenges, that I've explained before, it make sense to take a look not the alternative solution called Wandisco.  Step 3.2.1 HDFS Disaster Recovery. Wandisco. Edge (proxy) node In case of Wandisco you will need to prepare some proxy nodes on the source and destination side. We do recommend to use one of the Big Data Appliance node and here you may refer to MOS note, which will guide you on how to make one of the node available for been proxy node: How to Remove Non-Critical Nodes from a BDA Cluster (Doc ID 2244458.1) Step 3.2.2 HDFS Disaster Recovery. Wandisco. Installation WANdisco Fusion is enterprise class software. It requires careful environment requirements gathering for the installation, especially with multi-homed networking as in Oracle BDA.  Once the environment is fully understood, care must be taken in completing the installation screens by following the documentation closely. Note: if you have some clusters which you want deal by BDR and they don't have Wandisco fusion software, you have to install Fusion client on it Step 3.2.3 HDFS Disaster Recovery. Wandisco. Architecture For my tests I've used two Big Data Appliances (Starter Rack - 6 nodes). Wandisco required to install their software on the edge nodes and I've converted Node06 into the Edge node for Wandisco purposes. Final architecture looks like this: Step 3.2.4 HDFS Disaster Recovery. Wandisco. Replication by example Here I'd like to show how you have to set up replication between two clusters. You need to install Wandisco fusion software on both clusters. As soon as you install fusion on the second (DR) cluster you need to do Induction (peering) with the first (Prod) cluster. As a result of installation, you have WebUI for Wandisco Fusion (it's recommended to be installed in on the Edge Node), you have to go there and setup replication rules.  go to replication bookmark and click on "create" button: after this specify path, which you would like to replicate and choose source of truth: after this click on "make consistent" button to kick-off replication: you could monitor list of the files and permissions which is not replicated yet: and you could monitor performance of the replication in real time: on the destination cluster you may see files, which not been replicated yet (metadata only) with prefix "_REPAIR_" [root@dstclusterNode01 ~]#  hadoop fs -ls /tmp/test_parq_fusion/store_returns/ 19/03/26 22:48:38 INFO client.FusionUriUtils: fs.fusion.check.underlyingFs: [true], URI: [hdfs://gcs-lab-bdax72orl-ns], useFusionForURI: [true] 19/03/26 22:48:38 INFO client.FusionCommonFactory: Initialized FusionHdfs with URI: hdfs://gcs-lab-bdax72orl-ns, FileSystem: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_142575995_1, ugi=hdfs/dstclusterNode01.us.oracle.com@US.ORACLE (auth:KERBEROS)]], instance: 1429351083, version: 2.12.4.3 Found 26 items drwxrwxrwx   - hdfs supergroup          0 2019-03-26 18:29 /tmp/test_parq_fusion/store_returns/.fusion -rw-r--r--   3 hdfs supergroup          0 2019-03-26 22:48 /tmp/test_parq_fusion/store_returns/000000_0._REPAIR_ -rw-r--r--   3 hdfs supergroup          0 2019-03-26 22:48 /tmp/test_parq_fusion/store_returns/000001_0._REPAIR_ -rw-r--r--   3 hdfs supergroup          0 2019-03-26 22:48 /tmp/test_parq_fusion/store_returns/000002_0._REPAIR_ -rw-r--r--   3 hdfs supergroup          0 2019-03-26 22:48 /tmp/test_parq_fusion/store_returns/000003_0._REPAIR_ if you put some file on one of the side, it will appear automatically on the other side (no action required) Step 3.2.5 HDFS Disaster Recovery. Wandisco. Hive The Fusion Plugin for Live Hive enables WANdisco Fusion to replicate Hive’s metastore, allowing WANdisco Fusion to maintain a replicated instance of Hive’s metadata and, in future, support Hive deployments that are distributed between data centers. The Fusion Plugin for Live Hive extends WANdisco Fusion by replicating Apache Hive metadata. With it, WANdisco Fusion maintains a Live Data environment including Hive content, so that applications can access, use, and modify a consistent view of data everywhere, spanning platforms and locations, even at petabyte scale. WANdisco Fusion ensures the availability and accessibility of critical data everywhere. Here you could find more details. Step 3.2.6 HDFS Disaster Recovery. Wandisco. Sentry Use the Fusion Plugin for Live Sentry to extend the WANdisco Fusion server with the ability to replicate policies among Apache Sentry Policy Provider instances. Coordinate activities that modify Sentry policy definitions among multiple instances of the Sentry Policy Provider across separate clusters to maintain common policy enforcement in each cluster. The Fusion Plugin for Live Sentry uses WANdisco Fusion for coordination and replication. Here you could find more details. Step 3.2.7 HDFS Disaster Recovery. Wandisco. Advantages and Disadvantages Talking about Wandisco disadvantages I have to say that it was very hard to install it. Wandisco folks promised that it will be enhanced in a future, but time will show. Advantages (+): - It's realtime. You just load data into the one cluster and another cluster immediately pick up the changes - it's Active-Active replication. You could load data in both clusters and data-sync will be done automatically - Sentry policies replication - Use less resources than BDR - Easy to manage replication policies by WebUI - Wandisco supports cross Hadoop distros replication - Wandisco is multiple endpoint (or multi-target) replication.  A replication rule isn't limited to just source and target (e.g. Prod, DR, Object Store) Disadvantages (-): - A common trade-off for additional features can often be additional complexity during installation.  This is the case with WANdisco Fusion - it costs extra money (BDR is free) - It requires special Hadoop client. As consequence if you want to replicate data with BDR on some remote clusters, you need to install Wandisco Fusion Hadoop client on it Step 3.3 HDFS Disaster Recovery. Conclusion I'll leave it to customer to decide which replication approach is better, I'd just say that it's good approach start with Big Data Disaster Recovery (because it's free and ready to use out of the box) and if customer will have some challenges with it try Wandisco software.  Step 4.1 HBase Disaster Recovery in this blog post I've focused on the HDFS and Hive data replication. If you want to replicate HBase on the remote cluster, all details on how to do this you could find here. Step 5.1 Kafka Disaster Recovery Kafka is another place where users may store the data and want to replicate it. Cloudera recommends to use Mirror Maker in order to do this. Step 6.1 Kudu Disaster Recovery There is another option available for customers to store their data - Kudu. As of today (03/01/2019) Kudu doesn't have solution to replicate data on the Disaster Recovery side. Step 7.1 Solr Disaster Recovery Solr or Cloudera search is another one engine to store data. You may get familiar with DR best practices, by reading this blog from Cloudera. 

Earlier I've written about Big Data High Availability in different aspects and I intentionally avoided the Disaster Recovery topic. High Availability answers on the question how system should process...

Autonomous

Querying external CSV, TSV and JSON files from your Autonomous Data Warehouse

I would like to provide here some practical examples and best practices of how to make use of the powerful data loading and querying features of the Autonomous Data Warehouse (ADW) in the Oracle Cloud. We will dive into the meaning of the more widely used parameters, which will help you and your teams derive business value out of your Data Warehouse in a jiffy! An extremely useful feature of this fully managed service is the ability to directly query data lying in your external object store, without incurring the time and cost of physically loading your data from an object store into your Data Warehouse instance. The DBMS_CLOUD.CREATE_EXTERNAL_TABLE package & procedure enables this behavior, creating a table structure over your external object store data, and allowing your ADW instance to directly run queries and analyses on it. A few pre-cursor requirements to get us running these analyses: Make sure you have a running ADW instance, a credentials wallet and a working connection to your instance. If you haven’t done this already follow Lab 1 in this tutorial. Use this link to download the data files for the following examples. You will need to unzip and upload these files to your Object Store. Once again, if you don’t know how to do this, follow Lab 3 Step 4 in this tutorial, which uploads files to a bucket in the Oracle Cloud Object Store, the most streamlined option. You may also use AWS or Azure object stores if required, you may refer to the documentation for more information on this. If you are using the Oracle Cloud Object Store as in Lab 3 above, you will need Swift URLs for the files lying in your object store. If you already created your object store bucket’s URL in the lab you may use that, else to create this use the URL below and replace the placeholders <region_name>, <tenancy_name> and <bucket_name> with your object store bucket’s region, tenancy and bucket names. The easiest way to find this information is to look at your object’s details in the object store, by opening the right-hand menu and clicking “Object details” (see screenshot below).   https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<tenancy_name>/<bucket_name> Note: In coming updates you will be able to use this object store URL directly in the DBMS_CLOUD API calls, instead of a SWIFT URL. Have the latest version of SQL Developer preferably (ADW requires v18.3 and above).   Comma Separated Value (CSV) Files   CSV files are one of the most common file formats out there. We will begin by using a plain and simple CSV format file for Charlotte’s (NC) Weather History dataset, which we will use as the data for our first external table. Open this Weather History ‘.csv’ file in a text editor to have a look at the data. Notice each field is separated by a comma, and each row ends by going to the next line. (ie. Which implies a newline ‘\n’ character). Also note that the first line is not data, but metadata (column names).   Let us now write a script to create an external table in our ADW over  this data file lying in our object store. We will specify all the column names, and the format of the file as CSV. The format parameter in the DBMS_CLOUD.CREATE_EXTERNAL_TABLE procedure takes a JSON object, which can be provided in two possible formats. format => '{"format_option" : “format_value” }' format => json_object('format_option' value 'format_value')) The second format option has been used in the script below. set define on define base_URL = <paste SWIFT URL created above here> BEGIN DBMS_CLOUD.CREATE_EXTERNAL_TABLE( table_name =>'WEATHER_REPORT_CSV', credential_name =>'OBJ_STORE_CRED', file_uri_list =>'&base_URL/Charlotte_NC_Weather_History.csv', format => json_object('type' value 'csv', 'skipheaders' value '1',   'dateformat' value 'mm/dd/yy'), column_list => 'REPORT_DATE DATE, ACTUAL_MEAN_TEMP NUMBER, ACTUAL_MIN_TEMP NUMBER, ACTUAL_MAX_TEMP NUMBER, AVERAGE_MIN_TEMP NUMBER, AVERAGE_MAX_TEMP NUMBER, AVERAGE_PRECIPITATION NUMBER' ); END; / Let us breakdown and understand this script. We are invoking the “CREATE_EXTERNAL_TABLE” procedure in the DBMS_CLOUD API  package. We are then providing the table name we want in our Data Warehouse, our user credentials (we created this in the pre-requisites), the object store file list that contains our data, a format JSON object that describes the format of our file to the API, and a list of named columns for our destination table. The format parameter is a constructed JSON object with format options ‘type’ and ‘skipheaders’. The type specifies the file format as CSV, while skipheaders tells the API how many rows are metadata headers which should be skipped. In our file, that is 1 row of headers. The 'dateformat' parameter specifies the format of the date column in the file we are reading from; We will look at this parameter in more detailed examples below. Great! If this was successful, we have our first external table. Once you have created an external table, it’s a good idea to validate that this external table structure works with your underlying data, before directly querying the table and possibly hitting a runtime error. Validating the table creates logs of any errors in case your external table was created incorrectly, which helps debug and fix any issues. Use the rowcount option in VALIDATE_EXTERNAL_TABLE if your data is large, to limit the validation to the specified number of rows. BEGIN  DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE (       table_name => 'WEATHER_REPORT_CSV'  ); END; / If you do see errors during validation, follow Lab 3 Step 12 to troubleshoot them with the help of the log file. If required, you can also drop this table like you would any other table with the “DROP TABLE” command. On running this validation without errors, you now have a working external table which sits on top of the data in your object store. You may now query and join the WEATHER_REPORT_CSV table as though it is any other table in your Data Warehouse! Let us find the days in our dataset during which it was pleasant in Charlotte. SELECT * FROM WEATHER_REPORT_CSV where actual_mean_temp > 69 and        actual_mean_temp < 74;   Tab Separated Value (TSV) Files   Another popular file format involves tab delimiters or TSV files. In the files you downloaded look for the Charlotte Weather History ‘.gz’ file. Unzip, open and have look at the ".tsv" file in it in a text editor as before. You will notice each row in this file is ended by a pipe ‘|’ character instead of a newline character, and the fields are separated by tabspaces. Oftentimes applications you might work with will output data in less intelligible formats such as this one, and so below is a slightly more advanced example of how to pass such data into DBMS_CLOUD. Let’s run the following script: BEGIN   DBMS_CLOUD.CREATE_EXTERNAL_TABLE (     table_name =>'WEATHER_REPORT_TSV',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'&base_URL/Charlotte_NC_Weather_History.gz',     format => json_object('removequotes' value 'true',                           'dateformat' value 'mm/dd/yy',                           'delimiter' value '\t',                           'recorddelimiter' value '''|''',                           'skipheaders' value '1'),     column_list => 'REPORT_DATE DATE,     ACTUAL_MEAN_TEMP NUMBER,     ACTUAL_MIN_TEMP NUMBER,     ACTUAL_MAX_TEMP NUMBER,     AVERAGE_MIN_TEMP NUMBER,     AVERAGE_MAX_TEMP NUMBER,     AVERAGE_PRECIPITATION NUMBER'  ); END; / SELECT * FROM WEATHER_REPORT_TSV where actual_mean_temp > 69 and        actual_mean_temp < 74; Whoops! You just hit a runtime error. An important lesson here is that we ran a query directly, without validating the external table like in the previous example. Thus we ran into an error even though the “CREATE_EXTERNAL_TABLE” went through without errors. This is because the “CREATE_EXTERNAL_TABLE” procedure simply creates a table structure (or metadata) over the data, but will not actually check to see whether the data itself is valid; That occurs at validation or runtime. Without validation, our only option would be to visually decipher the problem with the code. Here’s the real working script this time: DROP TABLE WEATHER_REPORT_TSV; BEGIN  DBMS_CLOUD.CREATE_EXTERNAL_TABLE (     table_name =>'WEATHER_REPORT_TSV',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'&base_URL/Charlotte_NC_Weather_History.gz',     format => json_object('ignoremissingcolumns' value 'true',                           'removequotes' value 'true',                           'dateformat' value 'mm/dd/yy',                           'delimiter' value '\t',                           'recorddelimiter' value '''|''',                           'skipheaders' value '1',                           'rejectlimit' value '1',                           'compression' value 'gzip'),     column_list => 'REPORT_DATE DATE,     ACTUAL_MEAN_TEMP NUMBER,     ACTUAL_MIN_TEMP NUMBER,     ACTUAL_MAX_TEMP NUMBER,     AVERAGE_MIN_TEMP NUMBER,     AVERAGE_MAX_TEMP NUMBER,     AVERAGE_PRECIPITATION NUMBER' ); END; / SELECT * FROM WEATHER_REPORT_TSV where actual_mean_temp > 69 and        actual_mean_temp < 74; Let us understand the new parameters here, and why our previous script failed: 'ignoremissingcolumns' value 'true': Notice there is no data for the last column “AVERAGE_PRECIPITATION”. This parameter allows the create external table script to skip over columns from the column list, that have no data in the data file. 'removequotes' value 'true': The first column ‘date’ has data surrounded by double quotes. For this data to be converted to an Oracle date type, these quotes need to be removed. Note that when using the type parameter for CSV files as we did in the first example, this removequotes option is true by default. 'dateformat' value 'mm/dd/yy': If we expect a date column to be converted and stored into an Oracle date column (after removing the double quotes of course), we should provide the date column’s format. If we don’t provide a format, the date column will look for the database's default date format. You can see the dateformat documentation here. 'delimiter' value '\t': Fields in this file are tab delimited, so the delimiter we specify is the special character. 'recorddelimiter' value '''|''': Each record or row in our file is delimited by a pipe ‘|’ symbol, and so we specify this parameter which separates out each row. Note that unlike the delimiter parameter, the recorddelimiter must be enclosed in single quotes as shown here. A nuance here is that the last row in your dataset doesn’t need the record delimiter when it is the default newline character, however it does for other character record delimiters to indicate the end of that row. Also note that since ADW is LINUX/UNIX based, source data files with newline as record delimiters, that have been created on Windows, must use “\r\n” as the format option. Both these nuances will likely have updated functionality in future releases. 'rejectlimit' value '1': We need this parameter here to fix an interesting problem. Unlike with the newline character, if we don’t specify a pipe record delimiter here at the very end of the file, we get an error because the API doesn’t recognize where the last row’s, last column ends. If we do specify the pipe record delimiter however, the API expects a new line because the record has been delimited, and we get a null error for the last non-existent row. To fix situations like this, where we know we might have one or more problem rows, we use the reject limit parameter to allow some number of rows to be rejected. If we use ‘unlimited’ as our reject limit, then any number of rows may be rejected. The default reject limit is 0. 'compression' value 'gzip': Notice the .tsv file is zipped into a gzip “.gz” file, which we have used in the URL. We use this parameter so the file will be unzipped appropriately before the table is created. As before, once this is successful, the external table structure has been created on top of the data in the object store. It may be validated using the VALIDATE_EXTERNAL_TABLE procedure. In the script above we have already queried it as a table in your Data Warehouse.   Field Lists - For more Granular parsing options:   A more advanced feature of the DBMS_CLOUD.CREATE_EXTERNAL_TABLE is the Field_List parameter, which borrows it’s feature set from the Field_List parameter of the Oracle Loader access driver. This parameter allows you to specify more granular information about the fields being loaded. For example, let’s use “Charlotte_NC_Weather_History_Double_Dates.csv” from the list of files in our object store. This file is similar to our first CSV example, except it has a copy of the date column in a different date format. Now, if we were to specify a date format in the format parameter, it would apply to universally to all date columns. With the field_list parameter, we can specify two different date formats for the two date columns. We do need to list all the columns and their types when including the field_list; Not mentioning any type parameters simply uses default Varchar2 values. BEGIN   DBMS_CLOUD.CREATE_EXTERNAL_TABLE (    table_name =>'WEATHER_REPORT_DOUBLE_DATE',    credential_name =>'OBJ_STORE_CRED',    file_uri_list =>'&base_URL/Charlotte_NC_Weather_History_Double_Dates.csv',    format => json_object('type' value 'csv',  'skipheaders' value '1'),    field_list => 'REPORT_DATE DATE ''mm/dd/yy'',                   REPORT_DATE_COPY DATE ''yyyy-mm-dd'',                   ACTUAL_MEAN_TEMP,                   ACTUAL_MIN_TEMP,                   ACTUAL_MAX_TEMP,                   AVERAGE_MIN_TEMP,                   AVERAGE_MAX_TEMP,                   AVERAGE_PRECIPITATION',    column_list => 'REPORT_DATE DATE,                    REPORT_DATE_COPY DATE,                    ACTUAL_MEAN_TEMP NUMBER,                    ACTUAL_MIN_TEMP NUMBER,                    ACTUAL_MAX_TEMP NUMBER,                    AVERAGE_MIN_TEMP NUMBER,                    AVERAGE_MAX_TEMP NUMBER,                    AVERAGE_PRECIPITATION NUMBER'  ); END; /   SELECT * FROM WEATHER_REPORT_DOUBLE_DATE where actual_mean_temp > 69 and actual_mean_temp < 74; It's important to recognize that the date format parameters are to provide the API with the information to read the data file. The output format from your query will be your Database default (based on your NLS Parameters). This can also be formatted in your query using TO_CHAR.   JSON Files   You may be familiar with JSON files for unstructured and semi-structured data. The "PurchaseOrders.txt" file contains JSON Purhcase Order data, which when parsed and formatted looks like the following. Using JSON data in an ADW instance can be as simple as putting each JSON document into a table row as a BLOB, and using the powerful, native JSON features that the Oracle Database provides to parse and query it. You can also view the JSON documentation for additional features here.  Let’s try this! Copy and run the script below: BEGIN   DBMS_CLOUD.CREATE_EXTERNAL_TABLE (    table_name =>'JSON_FILE_CONTENTS',    credential_name =>'OBJ_STORE_CRED',    file_uri_list =>'&base_URL/PurchaseOrders.txt',    column_list => 'json_document blob',    field_list => 'json_document CHAR(5000)' ); END; /   COLUMN Requestor FORMAT A30 COLUMN ShippingInstructions FORMAT A30 SELECT JSON_VALUE(json_document,'$.Requestor') as Requestor,        JSON_VALUE(json_document,'$.ShippingInstructions.Address.city') as ShippingInstructions FROM JSON_FILE_CONTENTS where rownum < 50; The query above lists all the PO requestors and the city where their shipment is to be delivered. Here, we have simply created one column ‘json_document’ in the external table ‘JSON_FILE_CONTENTS’. We do not incur the time it takes to validate these JSON document, and are instead directly querying the external table using the Database’s JSON_VALUE feature. This means the check for well-formed JSON data will be on the fly, which would fail unless you properly skip over the failed data. Here, ‘VALIDATE_EXTERNAL_TABLE’ will not check for valid JSON data, but will simply check that the data is of the correct native datatype (less than 5000 characters long), that is the datatype of the table’s column. For better performance on large JSON data files, using this external table we can also make use of the Database’s JSON features to parse and insert the JSON data into a new table ‘j_purchaseorder’ ahead of time, as below. Note that this insert statement actually brings the data into your ADW instance. You benefit from doing this as it checks to make sure your JSON data is well-formed and valid ahead of time, and therefore incur less of a performance impact when you query this JSON data from your ADW instance. CREATE TABLE j_purchaseorder  (id          VARCHAR2 (32) NOT NULL,   date_loaded TIMESTAMP (6) WITH TIME ZONE,   po_document BLOB   CONSTRAINT ensure_json CHECK (po_document IS JSON));   INSERT INTO j_purchaseorder (id, date_loaded, po_document) SELECT SYS_GUID(), SYSTIMESTAMP, json_document FROM json_file_contents    WHERE json_document IS JSON; We can now query down JSON paths using the JSON simplified syntax as with the following query:   SELECT po.po_document.Requestor,          po.po_document.ShippingInstructions.Address.city                  FROM j_purchaseorder po;   Copying Data into your Autonomous Data Warehouse Here, we've gone through examples to access your object store data via external tables in your ADW. In following posts, I will walk you through examples on how to use the DBMS_CLOUD.COPY_DATA API to copy that data from your files directly into your Data Warehouse, as well as how to diagnose issues while loading your ADW with data using the bad and log files. See you in the next one!

I would like to provide here some practical examples and best practices of how to make use of the powerful data loading and querying features of the Autonomous Data Warehouse (ADW) in the Oracle...

Autonomous

Oracle Autonomous Data Warehouse - Access Parquet Files in Object Stores

Parquet is a file format that is commonly used by the Hadoop ecosystem.  Unlike CSV, which may be easy to generate but not necessarily efficient to process, parquet is really a “database” file type.  Data is stored in compressed, columnar format and has been designed for efficient data access.  It provides predicate pushdown (i.e. extract data based on a filter expression), column pruning and other optimizations. Autonomous Database now supports querying and loading data from parquet files stored in object stores and takes advantage of these query optimizations.  Let’s take a look at how to create a table over a parquet source and then show an example of a data access optimization – column pruning. We’ll start with a parquet file that was generated from the ADW sample data used for tutorials (download here).  This file was created using Hive on Oracle Big Data Cloud Service.  To make it a little more interesting, a few other fields from the customer file were added (denormalizing data is fairly common with Hadoop and parquet).   Review the Parquet File A CSV file can be read by any tool (including the human eye ) – whereas you need a little help with parquet.  To see the structure of the file, you can use a tool to parse its contents.  Here, we’ll use parquet-tools (I installed it on a Mac using brew – but it can also be installed from github): $ parquet-tools schema sales_extended.parquet message hive_schema {   optional int32 prod_id;   optional int32 cust_id;   optional binary time_id (UTF8);   optional int32 channel_id;   optional int32 promo_id;   optional int32 quantity_sold;   optional fixed_len_byte_array(5) amount_sold (DECIMAL(10,2));   optional binary gender (UTF8);   optional binary city (UTF8);   optional binary state_province (UTF8);   optional binary income_level (UTF8); } You  can see the parquet file’s columns and data types, including prod_id, cust_id, income_level and more.  To view the actual contents of the file, we’ll use another option to the parquet-tools utility: $ parquet-tools head sales_extended.parquet prod_id = 13 cust_id = 987 time_id = 1998-01-10 channel_id = 3 promo_id = 999 quantity_sold = 1 amount_sold = 1232.16 gender = M city = Adelaide state_province = South Australia income_level = K: 250,000 - 299,999 prod_id = 13 cust_id = 1660 time_id = 1998-01-10 channel_id = 3 promo_id = 999 quantity_sold = 1 amount_sold = 1232.16 gender = M city = Dolores state_province = CO income_level = L: 300,000 and above prod_id = 13 cust_id = 1762 time_id = 1998-01-10 channel_id = 3 promo_id = 999 quantity_sold = 1 amount_sold = 1232.16 gender = M city = Cayuga state_province = ND income_level = F: 110,000 - 129,999 The output is truncated – but you can get a sense for the data contained in the file. Create an ADW Table We want to make this data available to our data warehouse.  ADW makes it really easy to access parquet data stored in object stores using external tables.   You don’t need to know the structure of the data (ADW will figure that out by examining the file) – only the location of the data and an auth token that provides access to the source.  In this example, the data is stored in an Oracle Cloud Infrastructure Object Store bucket called “tutorial_load_adw”: Using the DBMS_CLOUD package, we will first create a credential using an auth token that has access to the data:   begin   DBMS_CLOUD.create_credential (     credential_name => 'OBJ_STORE_CRED',     username => user@oracle.com',     password => 'the-password'   ) ; end; / Next, create the external table.  Notice, you don’t need to know anything about the structure of the data.  Simply point to the file, and ADW will examine its properties and automatically derive the schema: begin     dbms_cloud.create_external_table (     table_name =>'sales_extended_ext',     credential_name =>'OBJ_STORE_CRED',     file_uri_list =>'https://swiftobjectstorage.<datacenter>.oraclecloud.com/v1/<obj-store-namespace>/<bucket>/sales_extended.parquet',     format =>  '{"type":"parquet",  "schema": "first"}'     ); end; / A couple of things to be aware of.  First, the URI for the file needs to follow a specific format – and this is well documented here (https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/user/dbmscloud-reference.html#GUID-5D3E1614-ADF2-4DB5-B2B2-D5613F10E4FA ).  Here, we’re pointing to a particular file.  But, you can also use wildcards (“*” and “?”) or simply list the files using comma separated values. Second, notice the format parameter.  Specify the type of file is “parquet”.  Then, you can instruct ADW how to derive the schema (columns and their data types):  1) analyze the schema of the first parquet file that ADW finds in the file_uri_list or 2) analyze all the schemas for all the parquet files found in the file_uri_list.  Because these are simply files captured in an object store – there is no guarantee that each file’s metadata is exactly the same.  “File1” may contain a field called “address” – while “File2” may be missing that field.  Examining each file to derive the columns is a bit more expensive (but it is only run one time) – but may be required if the first file does not contain all the required fields. The data is now available for query: desc sales_extended_ext; Name           Null? Type            -------------- ----- --------------  PROD_ID              NUMBER(10)      CUST_ID              NUMBER(10)      TIME_ID              VARCHAR2(4000)  CHANNEL_ID           NUMBER(10)      PROMO_ID             NUMBER(10)      QUANTITY_SOLD        NUMBER(10)      AMOUNT_SOLD          NUMBER(10,2)    GENDER               VARCHAR2(4000)  CITY                 VARCHAR2(4000)  STATE_PROVINCE       VARCHAR2(4000)  INCOME_LEVEL         VARCHAR2(4000) select prod_id, quantity_sold, gender, city, income_level from sales_extended_ext where rownum < 10; Query Optimizations with Parquet Files As mentioned at the beginning of this post, parquet files support column pruning and predicate pushdown.  This can drastically reduce the amount of data that is scanned and returned by a query and improve query performance.  Let’s take a look at an example of column pruning.  This file has 11 columns – but imagine there were 911 columns instead and you were interested in querying only one.  Instead of scanning and returning all 911 columns in the file – column pruning will only process the single column that was selected by the query. Here, we’ll query similar data – one file is delimited text while the other is parquet (interestingly, the parquet file is a superset of the text file – yet is one-fourth the size due to compression).  We will vary the number of columns used for each query:  Query a single parquet column Query all parquet columns Query a single text column Query all the text columns The above table was captured from the ADW Monitored SQL Activity page.  Notice that the I/O bytes for text remains unchanged – regardless of the number of columns processed.  The parquet queries on the other hand process the columnar source efficiently – only retrieving the columns that were requested by the query.  As a result, the parquet query eliminated nearly 80% of the data stored in the file.  Predicate pushdown can have similar results with large data sets – filtering the data returned by the query. We know that people will want to query this data frequently and will require optimized access.  After examining the data, we now know it looks good and will load it into a table using another DBMS_CLOUD procedure – COPY_DATA.  First, create the table and load it from the source: CREATE TABLE SALES_EXTENDED ( PROD_ID NUMBER, CUST_ID NUMBER, TIME_ID VARCHAR2(30), CHANNEL_ID NUMBER, PROMO_ID NUMBER, QUANTITY_SOLD NUMBER(10,0), AMOUNT_SOLD NUMBER(10,2), GENDER VARCHAR2(1), CITY VARCHAR2(30), STATE_PROVINCE VARCHAR2(40), INCOME_LEVEL VARCHAR2(30) ); -- Load data begin dbms_cloud.copy_data( table_name => SALES_EXTENDED', credential_name =>'OBJ_STORE_CRED', file_uri_list =>'https://swiftobjectstorage.<datacenter>.oraclecloud.com/v1/<obj-store-namespace>/<bucket>/sales_extended.parquet', format => '{"type":"parquet", "schema": "first"}' ); end; / The data has now been loaded.  There is no mapping between source and target columns required; the procedure will do a column name match.  If a match is not found, the column will be ignored. That’s it!  ADW can now access the data directly from the object store – providing people the ability to access data as soon as it lands – and then for optimized access load it into the database.  

Parquet is a file format that is commonly used by the Hadoop ecosystem.  Unlike CSV, which may be easy to generate but not necessarily efficient to process, parquet is really a “database” file type. ...

Hadoop Best Practices

Big Data Resource management Looks Hard, But it isn't

Hadoop is an ecosystem consisting of multiple different components in which each component (or engine) consumes certain resources. There are a few resource management techniques that allow administrators to define how finite resources are divided across multiple engines. In this post, I'm going to talk about these different techniques in detail. Talking about resource management, I'll divide the topic in these sub-topics: 1. Dealing with low latency engines (realtime/near-realtime) 2. Division of resources across multiple engines within a single cluster 3. Division of resources between different users within a single technology 1. Low latency engines These engines assume low latency response times. Examples are NoSQL databases (HBase, MongoDB, Cassandra...) or message based systems like Kafka. These systems should be placed on a dedicated cluster if you have real and specific low latency Service Level Agreements (SLAs). Yes, for highly utilized HBase or highly utilized Kafka with the strict SLA we do recommend to put it on a dedicated cluster, otherwise the SLAs can't be met. 2. Division of resources across multiple engines in a single cluster It's quite common to put multiple processing engines (such as YARN, Impala, etc.) in a single cluster. As soon as this happens, administrators face the challenge of how to divide resources among these engines. The short answer for Cloudera clusters is "Static Service Pools". In Cloudera Manager you can find the "Static Service Pool" configuration option here: You use this functionality to divide resources between different processing engines, such as: YARN Impala Big Data SQL Etc. When applied, under the hood, these engines are using Linux cgroup to partition resources across these engines.  I've already explained how to setup Static Service Pools in context of Big Data SQL, please review this for more details on configuring Static Resource Pools. 3. Division of resources between different users within a single technology 3.1 Resource division within YARN. Dynamic service pool To work with resource allocation in Yarn, Cloudera Manager offers "Dynamic Service Pools". Its purpose is to divide resources within different user groups inside YARN (a single engine): Because you work on Yarn, many different engines are impacted - e.g. those engines that run inside the Yarn framework. For example Spark, Hive (MapReduce) etc. The following steps are planned to be automated for Big Data Appliance and Big Data Cloud Service in an upcoming but if you want to apply this beforehand or on your own Cloudera clusters, here are the high level steps for this: a) Enable Fair Scheduler Preemption b) Configure example pools: for Low Priority, Medium Priority and High Priority jobs c) Setup placement rules The following sections dive deeper into the details. 3.1.1. Enable Fair Scheduler Preemption To enable fair scheduler preemption go to Cloudera Manager -> YARN -> Configuration. Then set: - yarn.scheduler.fair.preemption=true - yarn.scheduler.fair.preemption.cluster-utilization-threshold=0.7 The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources. Next we must configure: - yarn.scheduler.fair.allow-undeclared-pools = false When set to true, pools specified in applications but not explicitly configured, are created at runtime with default settings. When set to false, applications specifying pools not explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named with the username associated with the application - yarn.scheduler.fair.user-as-default-queue = false When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool name is not specified. When set to false, all applications are run in a shared pool, called default. Note: parameter "Enable ResourceManager ACLs" should be set to true by default, but its worth checking it, just in case. "yarn.admin.acl" shouldn't be equal '*'. set it equal to "yarn" After modifying these configuration settings you will need to restart the YARN cluster to activate these settings in your cluster. Next step is the configuration of the Dynamic Service Pools. 3.1.2. Configure example pools Go to Cloudera Manager -> Dynamic Resource Pool Configuration. Here we recommend (in future BDA/BDCS versions we will create these by default for you) to create three pools: low medium high We also recommend that you remove the root.default pool as shown below: Different pools will use different resources (CPU and Memory). To illustrate this I'll run 3 jobs and put these into the different pools: high, medium, low: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.low 1000 1000000000000 p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.medium 1000 1000000000000 p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.high 1000 1000000000000 after running for a while, navigate to Cloudera Manager -> YARN -> Resource Pool and take a look at "Fair Share VCores" (or memory): In this diagram we can see that vCores are allocated according to our configured proportion: 220/147/73 roughly the same as 15/10/5. Second important configuration is limit of maximum pool usage: we recommend to put a cap on the resource pool so small applications can jump into the cluster even if a long-running job has been launched. Here is a small test case: - Run Long running job in root.low pool (which takes days to be done): p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.low 1000 1000000000000 Check resource usage: This graph shows that we have some unused CPU, as we wanted. Also, notice below that we have some pending containers which shows that our application wants to run more jobs, but as expected YARN disallows to do this: So, despite that we have spare resource in our cluster, YARN disallows to use it because of the capping maximum resource usage for certain pools. - Now run some small job, which belong to the different pool: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} hive> set mapred.job.queue.name=root.medium; hive> select count(1) from date_dim; ... Now, jump to the resource usage page (Cloudera Manager -> YARN -> Resource Pools): Here we can see that the number of pending containers for the first job hasn't changed. This is the reserve we allocated for newly running small jobs. This enables short start up times for small jobs so, no preemption needed and end users will not feel like their jobs hangs. Third key things of Resource Management configuration is Preemption configuration. We recommend to configure different preemption levels for each different pool (double check you've enabled preemption earlier at the cluster level). There are two configuration settings to change: Fair Share Preemption Threshold - This is a value between 0 and 1. If set to x and the fair share of the resource pool is F, we start preempting resources from other resource pools if the allocation is under (x * F). In other words, it defines in lack of which portion of the resources we start to do preemption. Fair Share Preemption Timeout - The number of seconds a resource pool is under its fair share before it will try to preempt containers to take resources from other resource pools. In other words, this setting defines when YARN will Start to preempt.  To configure, go to Cloudera Manager -> Dynamic Service Pools -> Edit for certain pool -> Preemption. We suggest the following settings: For high. Immediately start preemption if job didn't get all requested resources, which is implemented as below: For medium. Wait 60 seconds before starting preemption if the job didn't get at least 80% of the requested resources: And for low. Wait a 180 seconds before starting preemption if a job didn't get at least 50% of its requested resources: These parameters define how aggressively containers will be preempted (how quickly job will get required resources). Here is my test case - I've run some long running job in root.low pool and run some job in parallel, assign it to low, medium and high pool respectively. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} hive> set mapred.job.queue.name=root.low; hive> select count(1) from store_sales; ... p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} hive> set mapred.job.queue.name=root.medium; hive> select count(1) from store_sales; ... p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} hive> set mapred.job.queue.name=root.high; hive> select count(1) from store_sales; ... as a measure of the result we could consider elapsed time (which will consist of waiting time, which will be different, according our configuration, plus elapsed time which also will be different because of resource usage). This table shows the result: Pool name Elapsed time, min Low 14.5 Medium 8.8 High 8 In the graph below you can see how preemption was accomplished: There is another aspect of preemption, which is whether a pool can be preempted or not. To set this up, go to: Cloudera Manager -> Dynamic Resource Pool -> root.high pool, there click on "Edit": After this click on "Preemption" and disable preemption from root.high pool. That will mean that nobody can preempt tasks from this pool: Note: this setting makes root.high pool incredibly strong and you may have to consider enabling preemption again. 3.1.3. Setup Placement rules Another key component of Dynamic Resource management is Placement rules. Placement rules define which pool a job will be assigned to. By default, we suggest keeping it simple To configure, go to Cloudera Manager -> Dynamic Service Pools -> Placement Rules: With this configuration your user may belong to one of the secondary groups, which we named low/medium/high. If not, you can define the pool assigned for the job at runtime. If you don't do that, by default the job will be allocated resources in the low pool. So, if administrator knows what to do she will put user in certain pool, if users know what to do, they will specify certain pool (low, medium or high). We recommend administrators defining this for the system. For example, I do have user alex, who belongs to secondary group "medium": p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# id alex uid=1002(alex) gid=1006(alex) groups=1006(alex),1009(medium) so, if I'll try to specify a different group (consider it as user wants to cheat the system settings) at runtime it will not overwrite medium group: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# sudo -u alex hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.high 1000 1000000000000 3.1.4. How to specify resource pool at runtime? While there are a few engines, let's focus on MapReduce (Hive) and Spark. Earlier, in this blog I've showed how to specify a pool for MapReduce job, with mapred.job.queue.name parameter. You can specify it with the -D parameter when you launch the job from the console: [root@bdax72bur09node01]# hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.low 1000 1000000000000 Or in case of hive you can set it up as a parameter: hive> set mapred.job.queue.name=root.low;                                             Another engine is Spark, and here you can simply add the "queue" parameter: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01]# spark-submit --class org.apache.spark.examples.SparkPi --queue root.high spark-examples.jar in Spark2-Shell console you need to specify the same parameter: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01 lib]# spark2-shell --queue root.high 3.1.5. Group Mapping The first thing that Resource Manager is checking is user secondary group. How do you define this? I've posted it earlier in my security Blog Post, but in the nutshell it is defined either with LDAP mapping or UnixShell and defined under "hadoop.security.group.mapping" parameter in Cloudera Manager (HDFS -> Configuration): Below is a list of useful commands, which could be used for managing user and groups on BDA/BDCS/BDCC (note all users and groups have to exist on each cluster node and have same id): p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} // Add new user # dcli -C "useradd  -u 1102 user1" // Add new group # dcli -C "groupadd -g 1111 high" // Add user to the group # dcli -C "usermod -a -G high user1" // Remove user from the group # dcli -C "gpasswd -d user1 high" // Delete user # dcli -C "userdel user1" // Delete group # dcli -C "groupdel high" Oracle and Cloudera recommend to use "org.apache.hadoop.security.ShellBasedUnixGroupsMapping" plus SSSD to replicate users from Active Directory. From Cloudera's documentation: "The LdapGroupsMapping library may not be as robust a solution needed for large organizations in terms of scalability and manageability, especially for organizations managing identity across multiple systems and not exclusively for Hadoop clusters. Support for the LdapGroupsMapping library is not consistent across all operating systems." 3.1.6. Control user access to certain pool (ACL) You may want to restrict the group of users that has access to certain pools (especially to high and medium). You accomplish this with ACLs in Dynamic Resource Pools. First, just an a reminder, that you have to set up "yarn.admin.acl" to something different than '*'. Set it equal to "yarn". Before setting restrictions for certain pools, you need to set up a restriction for the root pool: due Cloudera Manager bug, you can't leave ACL for submission and admin empty (otherwise everyone will allow to submit jobs in any pool), so set it up equal to desired groups or as workaround set it up to ",". After this you are ready to create rules for the other pools. Let's start from low. Here as soon as it plays role of default pool in our config, we should allow everyone to submit jobs there: next, let's move to medium. There we will config access for users, who belongs to groups medium and etl: and finally, we are ready to config high pool. Here we will allow to submit jobs for users who belong to groups managers or high: let's do a quick test. let's take some user, who is not belongs to privileged groups: medium, high, etl, managers. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01 hadoop-mapreduce]# hdfs groups user2 user2 : user2 and run some jobs after: [root@bdax72bur09node01 hadoop-mapreduce]# sudo -u user2 hadoop jar hadoop-mapreduce-examples.jar pi 1 10 ... 18/11/19 21:22:31 INFO mapreduce.Job: Job job_1542663460911_0026 running in uber mode : false 18/11/19 21:22:31 INFO mapreduce.Job:  map 0% reduce 0% 18/11/19 21:22:36 INFO mapreduce.Job:  map 100% reduce 0% after this check that this user was place in root.low queue (everyone is allowed to run jobs there): so far so good. Now, let's try to submit job from the same user to high priority pool: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} [root@bdax72bur09node01 hadoop-mapreduce]# sudo -u user2 hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.high 1 10 ... 18/11/19 21:27:33 WARN security.UserGroupInformation: PriviledgedActionException as:user2 (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1542663460911_0027 to YARN : User user2 cannot submit applications to queue root.high to put this job in pool root.high, we need to add this user to any group, which is listed in ACL for pool root.high, let's use managers (create it first): p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01 hadoop-mapreduce]# dcli -C "groupadd -g 1112 managers" [root@bdax72bur09node01 hadoop-mapreduce]# dcli -C "usermod -a -G managers user2" second try and validation: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01 hadoop-mapreduce]# hdfs groups user2 user2 : user2 managers [root@bdax72bur09node01 hadoop-mapreduce]# sudo -u user2 hadoop jar hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.high 1 10 ... 18/11/19 21:34:00 INFO mapreduce.Job: Job job_1542663460911_0029 running in uber mode : false 18/11/19 21:34:00 INFO mapreduce.Job:  map 0% reduce 0% 18/11/19 21:34:05 INFO mapreduce.Job:  map 100% reduce 0% wonderful! everything works as expected. Once again, here is important to set up root pool to some values. If you don't want put the list of available groups in the root pool and want to put it later or you may want to have a pool, like root.low where everyone could submit their jobs, simply use workaround with "," character. 3.1.7. Analyzing of resource usage The tricky thing with the pools, that sometimes many people or divisions use the same pool and it's hard to define who get which portion.  p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85); min-height: 16.0px} span.s1 {font-variant-ligatures: no-common-ligatures} [root@bdax72bur09node01 hadoop-mapreduce]# cat getresource_usage.sh #!/bin/bash   STARTDATE=`date -d " -1 day " +%s%N | cut -b1-13` ENDDATE=`date +%s%N | cut -b1-13` result=`curl -s http://bdax72bur09node04:8088/ws/v1/cluster/apps?finishedTimeBegin=$STARTDATE&finishedTimeEnd=$ENDDATE` if [[ $result =~ "standby RM" ]]; then result=`curl -s http://bdax72bur09node05:8088/ws/v1/cluster/apps?finishedTimeBegin=$STARTDATE&finishedTimeEnd=$ENDDATE` fi #echo $result echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "user|coreSeconds" | awk ' /user/ { user = $2 } /vcoreSeconds/ { arr[user]+=$2 ; } END { for (x in arr) {print "yarn." x ".cpums="arr[x]} } '   echo $result | python -m json.tool | sed 's/["|,]//g' | grep -E "user|memorySeconds" | awk ' /user/ { user = $2 } /memorySeconds/ { arr1[user]+=$2 ; } END { for (y in arr1) {print "yarn." y ".memorySeconds="arr1[y]} } '   3.2.1. Impala Admission Control Another popular engine in the Hadoop world is Impala. Impala has own mechanism to control resources - called admission control. Many MPP systems recommend queueing queries in case of high concurrency instead of running it in parallel. Impala is not an exception. For config this, go to Cloudera Manager -> Dynamic Resource Pools -> Impala Admission Control: Admission Control has few key parameters for configuring queue: Max Running Queries - Maximum number of concurrently running queries in this pool Max Queued Queries - Maximum number of queries that can be queued in this pool Queue Timeout - The maximum time a query can be queued waiting for execution in this pool before timing out so, you will be able to place up to Max Running Queries queries, after this rest Max Queued Queries queries will be queued. They will stay in the queue for Queue Timeout, after this, it will be canceled. Example. I've config Max Running Queries =3 (allow to run 3 simultaneous SQLs), Max Queued Queries =2, which allows running two simultaneous queries and Queue Timeout is default 60 seconds.  After this, I've run 6 queries and wait for a minute. Three queries were successfully executed, three other queries failed for different reasons: One query been rejected right away, because it were not place for it in a queue (3 running, 2 queued next rejected). Two others were in queue for a 60 seconds, but as soon as rest queries were not executed within this timeout, they were failed as well.

Hadoop is an ecosystem consisting of multiple different components in which each component (or engine) consumes certain resources. There are a few resource management techniques that allow...