X

Recent Posts

Exadata Database Machine

Exadata System Software 19.1 – The Foundation for the Autonomous Database

Exadata System Software Release 19.1.0.0.0 is now generally available. This release lays the foundation for the Autonomous Database and improves every aspect of Exadata with 20+ features and enhancements. All new features and enhancements are available on all supported Exadata generations, protecting customer investment across all Exadata deployment models: on-premises, Cloud Service, and Cloud @ Customer. In this post we outline key 19.1.0.0.0 enhancements for automated performance management, improved operations via Oracle Linux 7, and improved availability and security. We also discuss new improvements for even faster Smart Scan performance. Later posts will discuss individual features, and the impatient can also review the full documentation or tune in to this webcast. Automated, Cloud-scale Performance Monitoring The ability to automatically detect and report anomalies to enable software or persons to take corrective actions is crucial to cloud-scale autonomous infrastructure. Release 19.1.0.0.0 automates monitoring of CPU, memory, Input/Output, file system, and network – customers need only configure the notification mechanism. This automation combines machine learning techniques with the deep lessons learned from thousands of mission critical real-world deployments. For example, Exadata can detect that hogging of system resources is affecting database performance, identify the culprit runaway process, and issue an alert, without any pre-existing set-up. To benefit from automatic performance monitoring, all you need to do is configure the notifications. Oracle Linux 7 With Release 19.1.0.0.0, both database and storage servers now run on Oracle Linux 7 Update 5, which brings further performance and reliability improvements. Upgrading to Oracle Linux 7 does not require re-imaging the database servers, and can be done in a rolling fashion, without requiring any downtime. Our customers can now easily upgrade to the most modern Operating System for running the Oracle Database with no outage, no re-imaging, period. Oracle Linux 7 enables Exadata servers to boot faster, and it improves ksplice and kexec, key enablers to nonstop Exadata operation. Also, with Oracle Linux 7, Exadata moves to chrony to further reduce jitter via better and faster time synchronization with NTP servers. Even Higher Availability For OLTP workloads, Release 19.1.0.0.0 makes the addition of new Flash devices to Exadata more transparent, for even higher availability. When a Flash card is replaced, I/O from that card can be delayed as data must be read from disk.  This delay is now mitigated by prefetching, in the background, the most frequently accessed OLTP data into the Flash cache. The key is using algorithms similar to the ones introduced for In-Memory OLTP Acceleration, which ensure that the blocks that were most important to the database are fetched first. This feature provides better application performance after the failure of a Flash device or a storage server by partially duplicating hot data in the Flash cache of multiple storage servers. It works for write-back Flash cache (as there is no secondary mirroring for write-through Flash cache), and it is useful for OLTP workloads, as it does not cache scan data. Even Better Security Release 19.1.0.0.0 security improvements include support for Advanced Intrusion Detection Environment (AIDE), enhanced Secure Eraser, and the addition of access control lists to the Exadata RESTful service. See documentation for the full list of security improvements. AIDE tracks and ensures the integrity of files on the system to detect system intrusions. Secure Eraser now supports erasing data disks only (i.e., excluding system disks), and erasing individual devices. On supported hardware, Secure-Erase is automatically started during the re-imaging of hardware, which simplifies re-imaging without performance penalty. Also, when users specify the disk erasure method 1pass, 3pass, or 7pass, Exadata System Software 19.1.0.0.0 uses Secure Erase if the hardware supports it, for a dramatically shorter erasure time, especially on today’s very large disks. Customers can now specify a list of IP addresses or IP subnet masks to control access to the Exadata RESTful service, or they can disable access altogether. Another security-related change is that Smart Scan and ExaWatcher processes now run under new, less privileged, operating system users. For details, see documentation. Even Faster Smart Scans We call out further improvements to Smart Scan, a flagship innovation in the first Exadata Database Machine, as a good example of Oracle’s commitment to continuing technical innovation. Release 19.1.0.0.0 increases Smart Scan performance by enabling checksum computation at the column level. Checksum validation is key to Exadata’s comprehensive error detection, and it may happen during data storage or retrieval. For in-memory columnar format data on Exadata Smart Flash Cache, the checksum is now computed and validated at the column level rather than full block, enabling two important optimizations, as follows. Selective checksum computation: When Smart Scan reads a Flash block checksum, it now performs checksum verification only on the specific (scan) column Compression Units (CUs), ignoring other columns in the cache line, reducing CPU usage. For example: SQL> SELECT temperature FROM weather WHERE city = 'NASHUA' AND weather = 'SNOW' AND weather_date BETWEEN '01-JUN-18' AND '30-DEC-18'; Checksums are computed only on columns temperature, city, weather, and weather_date, even though the table may have many other columns. Just-in-time checksum computation: checksums are performed only when and if a column is processed. For example: SQL> SELECT temperature FROM weather WHERE city = 'REDWOOD CITY' AND weather = 'SNOW' AND weather_date BETWEEN '01-JUN-18' AND '30-DEC-18'; Since city = ‘REDWOOD CITY’ and weather = ‘SNOW’ returns no results, no checksum is done for columns weather_date and temperature. Less data on which to perform checksum reduces CPU usage. This feature is automatically enabled when you configure the INMEMORY_SIZE database initialization parameter and upgrade to Oracle Exadata System Software release 19.1.0.0.0 For more details see documentation. Summary This post covered the main innovations and enhancements in the new Exadata 19.1.0.0.0 software release. You get everything outlined here, and more, just by upgrading your Exadata software, with no additional costs. Heck, you don't even need to do that yourself: Oracle Platinum Services or our Cloud Operations team will take care of it at no charge. We are always interested in your feedback. You are welcome to engage with us via Twitter @ExadataPM and by comments here.

Exadata System Software Release 19.1.0.0.0 is now generally available. This release lays the foundation for the Autonomous Database and improves every aspect of Exadata with 20+ features...

Exadata Cloud for Small and Medium Businesses

Information Technology Requirements for Small and Medium Businesses Like their large company brethren, Small and Medium Businesses (SMBs) too are concerned about profitability, growth, lowering costs, improving productivity, and business continuity (data protection, disaster recovery, etc.). While the scale of their IT is smaller than that of large enterprises, SMBs must increasingly meet the same stringent IT requirements expected of large enterprises. As SMBs cannot justify a complex IT organization (for example, most SMB IT organizations report to the CFO) they value solutions that meet their requirements with a small number of vendors and reduced management complexity. In addition, SMBs are increasingly considering cloud solutions because they enable SMBs to: Pay for IT capabilities as needed and pay as operational (OpEx) instead of capital expenditures (CapEx) Integrate with existing applications and IT environment Increase the productivity of their IT The database is the cornerstone of IT, and Exadata is the best proven platform to support both mission critical database workloads and database consolidation. Exadata Cloud offerings enable SMBs to take full advantage of Exadata for all their database needs – from mission-critical databases to database consolidation as well as the management and provisioning of multiple "small" databases.  Exadata Cloud (Cloud at Customer and Cloud Service) Benefits  Exadata delivers performance, availability, scalability and better manageability for all database workloads at lower total cost while delivering improved productivity to application users. Exadata Cloud benefits for SMBs include: Pay for the capabilities you need as operational expenses (OpEx) -- more capital for your core business Exadata Cloud brings a much lower upfront cost than acquiring the hardware, enabling rapid adoption among this customer set. Oracle’s Universal Cloud Credits with Exadata Cloud eliminates the core minimums associated with Exadata Cloud non-metered subscriptions. (In the non-metered model Oracle required customer to subscribe to a minimum number of OCPUs per month.  With Universal Credits there no longer is the minimum subscription.) This dramatically decreases the upfront cost. Additionally, compute cores can be expanded and contracted as needed with no cost penalty, enabling customers to match capacity to workload and optimize costs.   Integrate with your existing applications and IT environment Because it delivers the Oracle Database, Exadata Cloud integrates seamlessly with existing applications and into the existing IT environment. Applications remain unchanged, because the Oracle Database is the same database whether on premises or in the cloud. What does change is that infrastructure work is now offloaded from the IT organization to Oracle. Oracle's Exadata Cloud delivers database services according to a Service Level Objective, and Oracle manages database servers,  networking, and storage servers as part of Exadata. Exadata cloud services are designed to keep the database service up and running through all events and updates so that You (the customer) can focus on value added tasks, innovation and running the business. No longer will IT need to respond to a flashing red light; for example,  for Cloud at Customer you'll be notified of Oracle's need to access the machine in the event there is a component failure. If there is a need to take a database node offline, you'll be notified. And, with RAC the service will continue to run.  Increase productivity by reducing your IT management workload A hurdle in the SMB market with an on premises solution was the need for the customer to administer and maintain Exadata and to develop those resources. With the Exadata Cloud Oracle administers and maintains the Exadata Infrastructure leaving the database administration to the customer. Exadata Cloud optimizes SMB productivity by enabling the existing DBAs to operate at higher efficiency and allows those DBA to focus on running the business and innovation Customer Experience Small and Medium Business Customers that have adopted Exadata Cloud are able to obtain the same benefits that large businesses have experienced last decade with Exadata: Performance and Scale – meet growth demands, whether organic growth or growth through mergers and acquisitions Lowest total cost of ownership – Exadata was often perceived as too expensive by SMBs Lower IT costs through Database consolidation and IT standardization Productivity – Oracle runs the infrastructure, Customers focus on the business and innovation Database Options – Ability to take advantage of all database options through PaaS subscriptions Business Continuity through High Availability and Disaster Recovery Security of Cloud at Customer behind the Customer (Your) firewall The move to a cloud consumption model for Exadata opens up Exadata’s ability to improve productivity and resource utilization to Small and Medium Businesses where Exadata might have been out of reach in the past. Find out more: Exadata overview, Case studies (PDF) (University Salzburg; video: Orbitz, WestJet, Customers section here), business value (IDC paper) Exadata Cloud Service (video: Macy's, Forbes article: Irish Postal Services) and Cloud at Customer (video: Glintt), Exadata Database Machine technical overview.

Information Technology Requirements for Small and Medium Businesses Like their large company brethren, Small and Medium Businesses (SMBs) too are concerned about profitability, growth, lowering costs,...

Sizing for the Cloud: Optimize Exadata Cloud Costs Using Universal Credits

The Universal Credits Model allows Oracle to dramatically alter cloud costs and deployment scenarios. Using Universal Credits, all OCPUs are treated the same, and all OCPUs can be elastically scaled to meet compute demand. This enables the most cost-effective Exadata Cloud deployments that Oracle has offered to date.   Databases, including consolidated databases, have a typical workload that is much less than the peak workload. In this post we discuss how workload variability offers the opportunity for dramatic cost savings when using Universal Credits with sizing appropriate to different scenarios. Production Workloads Corporations are busiest in the weeks approaching and just after the quarter close; Retailers in are busiest from October through January for the holiday season; Manufacturers are busiest during a planning cycle. All businesses have peak activity periods. The rest of the time workloads remain at their typical non-peak level, but on-premises customers must still size their databases for the peak workload plus some extra safety buffer. As an example, say that a typical production workload requires up to 16 cores with a peak workload of 32, plus 4 extra cores. This results in 36 cores provisioned and licensed when only 16 are needed the majority of the time.  Disaster Recovery Assume that the customer provisions database disaster recovery (DR) the same size as production – 36 cores. However, for the vast majority of the time the DR servers are applying redo from production, while waiting for an event that will shift the production workload to DR. Applying redo is a light workload compared to production. The amount of compute capacity required for the redo apply is a function of the amount of redo. If the redo workload requires 6 cores, an additional 30 cores will be provisioned and licensed which will be idle waiting for a rare DR event. Development and Test (Dev/Test) We see Dev/Test environments that vary in size from 1x to 10x the size of their corresponding production environment. Many Dev/Test activities do not exercise the full potential of the production environment, so compute resources are often idle. Dev/Test can often tolerate lower performance than production, as long as functionality is correct. The Dev/Test environment does requires at least the peak performance production size for full peak-performance testing, so here let us assume 36 cores in this case. Note that the rest of the time fewer 36 cores are required. Elastic Sizing with Universal Credits We bring together the different requirements outlined above in an example use case from a financial institution. They want to migrate from a commodity environment to a consolidated environment on Exadata Cloud at Customer. While the peak workload requires 164 OCPUs, the typical workload for 95% of the time is only 68 OCPUs. Said differently, in a month of 744 hours, 707 hours require 68 OCPUs and the remaining 37 hours require an additional 96 cores (obtained as elastic OCPUs or bursting) to meet the peak workload requirement of 164 OCPUs. We can now compare sizing scenarios: Sizing 64 OCPUs for 707 hours of typical workload plus an extra peak 96 OCPUs for 37 hours using Universal Credits results in a 49% decrease in price compared to 164 OCPUs for 744 hours. The significant savings of elastic sizing with Universal Credits warrants a serious look at Exadata Cloud deployment model.

The Universal Credits Model allows Oracle to dramatically alter cloud costs and deployment scenarios. Using Universal Credits, all OCPUs are treated the same, and all OCPUs can be elastically...

Snapshots in the Exadata Cloud

Building on the previous blog entry by Gurmeet (Exadata Snapshots: Simple, Secure, Space Efficient), we are going to see how we can create snapshots in the Exadata Cloud.  Snapshot PDBs With the Exadata Cloud (Cloud Service and Cloud at Customer) we have two ways to create snapshots on our services. The first method is via SQL plus and works exactly like what was shown in the previous blog post. You need to have created a SPARSE disk group when you initially provisioned the service. Once we log into our Exadata Cloud and choose a database to work with, we can choose a PDB we want to create a clone from. We can see what PDBs are available with the following SQL: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9); min-height: 14.0px} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} SQL> show pdbs;       CON_ID CON_NAME                       OPEN MODE  RESTRICTED ---------- ------------------------------ ---------- ----------      2 PDB$SEED                           READ ONLY  NO      3 EIGHTPDB                           READ WRITE NO We see we have the EIGHTPDB to work with. We first need to shut the PDB down on all nodes: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9); min-height: 14.0px} span.s1 {font-variant-ligatures: no-common-ligatures} SQL> alter pluggable database EIGHTPDB close instances=all;   Pluggable database altered. Then we put it into READY ONLY mode: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)} span.s1 {font-variant-ligatures: no-common-ligatures} SQL> alter pluggable database EIGHTPDB open read only instances=all;             Pluggable database altered. And now we create our PDB Snapshot copy using the SPARSE disk group: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)} span.s1 {font-variant-ligatures: no-common-ligatures} SQL> create pluggable database CLONEPDB from EIGHTPDB tempfile reuse create_file_dest='+SPRC4' snapshot copy keystore identified by "PASSWORD"; p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Andale Mono'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)} span.s1 {font-variant-ligatures: no-common-ligatures} Pluggable database created. Being a cloud database, TDE is on by default thus we need the keystore identified by "PASSWORD";  clause with the PASSWORD being the one you supplied upon database creation. Finally, open the snapshot PDB: SQL> alter pluggable database CLONEPDB open read write instances=all;             Pluggable database altered.   Thats it, we have a PDB snapshot copy that uses the SPARSE disk group for changes.   Snapshot Masters The second method is taking a shapshot master in the UI or via a REST service. Once we have that snapshot master, we can create sparse clones using the SPARSE disk group. Here is how you do just that. In the UI, view the database details. On the left side of the page in the Administration tab. Click that tab. Now you click the Snapshot subtab. Here we can see the Snapshot Master details. You can click the Create Snapshot Master button to bring up the create modal. In this modal we can create our snapshot master by giving it a name, a database name and password. We can use the node subsetting feature by clicking the Hostnames field and selecting what nodes of the service we want this snapshot to run on.  On the bottom of this modal is an ACFS checkbox. Using this checkbox we place this clone's oracle binaries on an ACFS mount saving space. Now this isn't for a production environment but would work well for test and development. Lastly we can choose to also clone the source oracle home rather than create a new one.  Once you fill in this modal, we can click create to start the snapshot create process. Once the snapshot is created, we can see it in the UI. Use the popup menu on the left to create a clone from this snapshot master. This clone uses the SPARSE disk group for any changes to the read only snapshot master. Once the clone it created, you can use it just as you would any database on the Exadata Cloud Service. To see a video of this in action, click here.

Building on the previous blog entry by Gurmeet (Exadata Snapshots: Simple, Secure, Space Efficient), we are going to see how we can create snapshots in the Exadata Cloud.  Snapshot PDBs With the Exadata...

Exadata Database Machine

Exadata Snapshots: Simple, Secure, Space Efficient

Database snapshots are widely used to re-create realistic workloads for database testing and development (test/dev). Almost all enterprise-class storage vendors support snapshot functionality that can create space-efficient point-in-time copies. However, using general-purpose snapshots for database snapshots requires compromises for test/dev activities and incurs extra process and management complexity as compared to database-aware snapshots. In this post we examine the requirements for database snapshots and contrast general-purpose snapshots with Exadata’s database-aware snapshots. Database testing and development with general-purpose snapshots Typical database test/dev requires re-creating the production environment as efficiently and simply as possible, often for multiple concurrent activities, with minimal disruption to production, and subject to data protection requirements (e.g., personal information, credit card numbers, etc., cannot percolate outside the production environment). Most general-purpose snapshot implementations migrate data from their primary (production) storage array to a secondary platform. Because a single production environment often requires multiple test/dev environments, secondary storage arrays and platforms typically can't afford to have the same performance or availability characteristics as the production environment. Also, any off-database redaction process introduces additional users and software into the set of entities that must be trusted. These compromises and additional procedures add complexity and reduce the quality of test/dev activities. Besides these snapshot issues, general-purpose storage arrays require ongoing, specialized administration. Database Administrators (DBAs) manage database snapshots with tools provided by the storage array vendor, third party data management tools, or Oracle’s own Enterprise Manager (OEM). While OEM simplifies snapshot management, general-purpose storage arrays also require significant non-database-related (storage) management, including creation of volumes or LUNs, capacity management, backup of the array, software upgrades, hardware maintenance tasks, and storage-level user control and access. In sum, while general-purpose snapshots are useful, they entail significant compromises that limit the quality and value of test/dev activities. Do all-flash arrays enable first-class database testing and development? Some all-flash array vendors propose to host test and development databases and production databases in the same primary (production) all-flash storage array. While this approach addresses the performance and availability compromises discussed in the previous section, it introduces a new set of issues. Combining production and test/dev workloads on the same storage array requires careful planning and active management of the database systems and the storage array, as test/dev databases can easily command a large portion of the array’s resources and degrade the performance of the production  database. One problem is that most all-flash arrays have low throughput, single-digits GB/s at best, which can be saturated by a handful of test databases running data warehousing queries in the shared storage array. More broadly, general-purpose storage arrays are unaware of the context in which read and write requests are issues by the database, and are thus limited in their ability to combine workloads intelligently. For example, if a production database submits a log-write while a test/dev database is issuing a long batch of writes, an all-flash array is unable to identify and prioritize the log-write over the batch writes. Combining test/dev with production workloads on the same all-flash storage arrays trades improvements in dev/test environments for risks to the production workloads. And all-flash storage arrays are still inherently limited in their throughput. Database-aware snapshots The solution is to make snapshots as close to the database -in functionality and location- as possible. Exadata provides a database-aware snapshot solution that is simple, space efficient, and secure, and it doesn’t compromise on the availability and the performance of the production system. Simple: An Exadata Snapshot is based on a test master, which is a full clone of the source database. From a single test master you can create multiple Exadata Snapshots with minimal additional storage and minimal effort. It takes one SQL command to create a snapshot database and it is created in minimal time.    SQL> create pluggable database PDB1S1 from PDB1TM1 tempfile reuse create_file_dest='+SPARSE' snapshot copy; Each snapshot represents a complete Oracle database and can take advantage of all Exadata features – Smart Scan, Smart Flash Cache, Smart Flash Logging to name a few. Exadata documentation explains the process in great detail. Exadata Snapshots are integrated with Enterprise Manager as well and customers can use EM to create snapshots in a few clicks. This demo gives a sneak peak on integration between Enterprise Manager and Exadata Snapshots. Exadata Cloud offerings also allow users to create a snapshot with just a few clicks (more on this in a later post). Space Efficient: Exadata Snapshots are built on sparse disk groups and use physical storage in proportion to new changes made to the database. In addition, hierarchical snapshots enable you to create snapshots from snapshots and any of these snapshots can act a sparse test master. And you can take a sparse backup of the snapshot, preserving only the new changes introduced in that database. In fact, Rent-A-Center, an Exadata customer, achieved ~30X space savings by implementing Exadata snapshots. Secure: Each Exadata snapshot inherits the same security model as the primary database. In addition, the test master database owner can delete or mask sensitive data in the test master before making it available to non-privileged users in the snapshot database. Exadata's fully comprehensive and database-aware resource management ensures that snapshots do not interfere with the performance of production workloads, and test/dev environments also benefit from Exadata's high availability. In later blog posts we will discuss the best practices on creating and managing snapshots, sizing the system to take advantage of them as well how snapshots are deployed in Exadata Cloud. Next: Snapshots in the Exadata Cloud.

Database snapshots are widely used to re-create realistic workloads for database testing and development (test/dev). Almost all enterprise-class storage vendors support snapshot functionality that can...

Announcing Exadata Cloud at Customer Release 18.1.4: Enhanced Database Consolidation

We are excited to announce general availability of the latest update for Exadata Cloud at Customer.  This new release (18.1.4) brings customers some of their most requested features—features that unlock Exadata Cloud at Customer’s true potential as a database consolidation platform. The key features in this release include:  Support for multiple VM Clusters in a single Exadata Machine Network isolation between the VM Clusters Shared Oracle Homes Snapshots and Sparse Clones for test/dev Restoration from a cloud backup Support for Oracle Database 18c, the latest release of our flagship database. Together, these new features solidify Exadata Cloud at Customer as the best database consolidation platform for any customer.  Unlike traditional public cloud solutions, Exadata Cloud at Customer runs in your data center, able to host all your data—even those with strict privacy and security requirements—while still providing the agility and economic benefits of a cloud solution.  Support for all features is integrated into the easy-to-use web-based UI, making their use simple and intuitive. How do these new feature benefit you as you consolidate?  First of all, multiple VM clusters within the environment provide better isolation. Better isolation means you can consolidate databases that according to your security and privacy policies, or even generally accepted best practices, should not be in the same cluster.  For example, different business units or data may legally require separation, to ensure data is accessible by authorized users only.  Production data can be segregate in a different cluster environment than test/dev, preventing administrators from confusing production data from almost identical test data and eliminating undesired impacts on production workloads due to testing operations.  New network isolation features ensure network data is protected from users on other clusters.  Data destined for one cluster cannot be intercepted by another. The new release also introduces shared Oracle homes.  Although the Exadata platform can consolidate hundreds of databases, the Oracle homes take up space on the local disk drives, and if each database required its own home, you’d be limited by the space on the disk.  In addition, managing and maintaining hundreds of Oracle homes would become a burden.  Maximum Availability Architecture (MAA) recommends less than 10 Oracle homes on a single server.  This is a tradeoff space and management, versus having flexibility to independently upgrade databases.  Now with this release, you can follow these best practices with Exadata Cloud at Customer. One common consolidation use case is creating test and development environments.  The new release makes Exadata Cloud at Customer a great test/dev platform.  Using the snapshot and sparse clone feature, you can quickly and effortlessly create many test/dev environments in a single cluster, without taking up a lot of space.  Each sparse clone only requires the space to record changes from the master.  Creating clones is thus fast and space efficient.  Now each developer and tester can have their own independent test and development environment. One of the biggest challenges to consolidation is migrating the many databases.  The new release makes it easier for you to get you databases into the new platform.  Simply back them up using Oracle Database Backup Cloud Service, and you can restore them to the Exadata Cloud at Customer with little fuss. Leverage the easy-to-use wizard in the control plane, or implement custom automation and workflows with the REST API. Finally, this release will enable Oracle Database 18c on the Exadata Cloud at Customer.  This is the latest release of the Oracle database, and it contains a great many features to support consolidation of databases on the platform.  Improve consolidation density by packing pluggable databases (PDBs) into shared container databases.  Improve overall performance with In Memory database.  All Oracle Database 18c features are available on Exadata Cloud at Customer.  The new features in this Exadata Cloud at Customer release will all help you more easily and safely adopt Oracle Database 18c.  Support for multiple VM Clusters will help anyone wishing to first test their applications with this new release.  You can create an independent 18c cluster for test/dev, while your production applications continue to run on an existing 12.2 cluster.  Restore a copy of your production database to your test/dev environment using the new Cloud backup restoration feature, then use the sparse clones to give each developer or tester their own copy of the database.  There is no need to worry about running out of storage cell space with sparse clones, or local disk space with shared Oracle homes. New deployments will get the latest release automatically, and existing deployments will be contacted by Oracle Cloud Operations to schedule the update.  I know many existing customers are anxiously awaiting these new features to enable the next step in your journey to simplify and consolidate your IT infrastructure. For those of you new to Exadata Cloud at Customer, you can find more information on Oracle.com

We are excited to announce general availability of the latest update for Exadata Cloud at Customer.  This new release (18.1.4) brings customers some of their most requested features—features that...

Oracle Database Exadata Cloud Service

Not every company can own and staff a data center; there is alot involved. So where can you turn if you need the highest-performing and most available infrastructure for running Oracle Database? Well the Exadata Cloud Service of course.  Now while we can have a bit of fun with the beginning of this post, it bring some honesty as well. A good majority of our Exadata Cloud Service customers are new to Exadata. They have never owned one on premises and are able to start their engineered systems journey with the cloud. So why is that? How is the cloud making owning and using an Exadata accessible to more and more customers? This entry as well as a few follow up posts will go into precisely how this is possible.  Lets start from the beginning; Provisioning. With an on-premises Exadata, you have some setup stets to perform before you are up and running. You also have shipping times. We are looking at a couple of weeks. Again, nothing to long, but will require, as we stated before, data center space, power and cooling as well as cabling to get started.  What if I told you with the Exadata Cloud Service, I could have you up and running in 4 hours? No data center or prior knowledge needed? This is the Exadata Cloud Service and one of the reasons that anyone who wants the power of an Exadata for their database workload can have one.  Once you get your welcome to the Oracle Cloud email and you log into your account, you can start creating a service immediately. Just click the Create Service Instance button and off we go.  The first and only page in our Create Service Instance flow asks you to name the service and to decide on how you want to backup your database. Start by naming your service, then selecting the Region where you purchased your ExaCS. Today customers can choose from four different data centers with more coming soon. The Rack Size lets you pick from the shapes you purchased; quarter, half or full. Lastly is backing the service. We have two options today. We can use local and cloud backups or cloud only backups. By checking this checkbox, we split up the grid disks to allow seven days of backups. This will allow you to not only backup quickly, but restore and create clones from backups. More on this later. Once you select these options, you are ready to create your service. Click the Create button and in about 2 hours for a quarter rack shape, you are ready to start creating databases. Speaking of creating databases, lets look at that now. On the create database page, all we need to do is click the Create button. On the first page, you are asked some basic information. First, name the database service and select the location or ExaCS you want it to be placed on. Some customers have multiple ExaCS instances and the  Exadata System dropdown will allow you to select the exact system you would like to create the database on. The next option to note here is the Database Version option. With the Exadata Cloud Service, you can create an 11.2, 12.1 or 12.2 database. And not just one, but multiple databases of multiple versions.  On the next page of the Create Database Wizard is the details page. We ask that you provide the service with a public key that only you have the matching private key. The tooling will use that public to secure the service so that only you, the holder of the matching private key has access and no one else. On this page, just enter some basic information like password for the sys and system users, set the character set of the database and select how you want to backup this particular database. We have three choices; Both cloud and local, cloud only and none. If this is a development database or maybe a database where you want to test out a cool new 12.2 feature, you may not want backups, so choose none. Have a database that is going to be used for production or disaster recovery? Choose one of the other two backup options. When using the cloud backups just provide where your oracle database backup service is and the tooling will automatically set it up for you.  Thats it! Hit create and in about 2 hours, you will have a database running on all the nodes of your ExaCS shape ready to be used. Want to create additional databases? Follow the same steps and in 20 minutes for a quarter rack shape, you will be ready to start using it.

Not every company can own and staff a data center; there is alot involved. So where can you turn if you need the highest-performing and most available infrastructure for running Oracle Database?...

Announcement: Oracle Database 12c Release 2 on Exadata and SuperCluster

The last 10 days have been very exciting for Exadata. Firstwe released Exadata Software 12.2.1.10 with over 30 features and enhancements. Thenwe launched Exadata SL6, unleashing the power of Software-in-Silicon forExadata systems.  Now we are releasingthe on-premises version of Oracle Database 12c Release 2 exclusively on Exadataand SuperCluster. Oracle Database 12c Release 2 enhances every aspect of theOracle Database. Let’s take a look at some highlights: Oracle Multitenant getsa boost with support for Application Containers. Multitenant users can now managememory resources for their Pluggable Databases in addition to CPU and IOresources. Database In-Memory gets faster joins. In-Memory users can now runin-memory queries on JSON data. In-Memory formats are now persisted in storagethus enabling faster restarts. This release also makes the database more secureas now tablespaces can be encrypted online without the need to export/importexisting unencrypted tablespaces. Oracle Database 12c Release 2 also includes themuch-anticipated Oracle Native Sharding. There are hundreds of other featuresthat make the database more secure, more available, faster, easier to manage,and more developer friendly. Oracle Database 12c Release 2 further differentiates the Exadataplatform. Building on Exadata’s unique resource management capabilities and IOoptimizations the following two features are exclusively available on Exadataand SuperCluster: In-Memory processing on Active Data Guard standbydatabases Support for more than 252 Pluggable Databases perContainer Database   This release also enables support for many unique Exadatafeatures such as hierarchical snapshots, Smart Scan offload for compressed indexes, Smart Scan enhancements to XML operators, end-to-endIO latency capping, and extended distance clusters. For more details pleaserefer to this slide deck or tune in to this webcast or read our prior blog post To take advantage of the full potential of Oracle Database12c Release 2 we encourage customers to upgrade to the latest Exadata Softwarerelease 12.2.1.10 before upgrading their database to 12c Release 2. ExadataSoftware 12.2.1.1.0 supports Smart Scan functionality for Oracle Database 12cRelease 2 and is available via OSDC. For the Oracle Database 12c Release 2 release schedule on platformsother than Exadata and Supercluster please refer to MOS Note 742060.1.

The last 10 days have been very exciting for Exadata. First we released Exadata Software 12.2.1.10 with over 30 features and enhancements. Thenwe launched Exadata SL6, unleashing the power...

Exadata SL6: A new Era in Software/Hardware Co-Engineering

Since the launch of Exadata in 2008, Intel x86 processorshave powered Exadata database servers.  Todaywe are expanding our Exadata family with the addition of Exadata SL6, our firstoffering with SPARC M7 processors. SLstands for “SPARC Linux”. The Exadata Database Machine SL6 is nearly identicalto the x86 based Exadata, except it uses Oracle SPARC T7-2 database serversbased on the SPARC M7 processor. SPARCM7 is the world’s most advanced processor to run Oracle databases, and thefirst to use a revolutionary technology from Oracle referred to as Software in Silicon. Software in Silicon technology is abreakthrough in microprocessor and server design, enabling databases to runfaster and with unprecedented security and reliability. Even though the database servers are based onSPARC processors, Exadata SL6 runs the exact same Linux Operating System asx86-based Exadata systems. This blog post discusses the highlights of the product as wellas our motivation behind it and what this announcement means to our existingproduct line. To learn more about thefiner details, please refer to the Exadata SL6 datasheet, and please watch JuanLoaiza explaining Exadata SL6 and it’s benefits in this short video.  Now let’s take a deeper look into Software in Silicontechnology. Software in Silicon is comprised of three very unique technologyofferings: SQL in Silicon, Capacity in Silicon and Security in Silicon. Each of these topics is worthy of their own individualblog posts, but for this write-up we will focus on the highlights. The SPARC M7 processor incorporates 32 on-chip DataAnalytics Accelerator (DAX) engines that are specifically designed to speed upanalytic queries. The accelerators offload in-memory query processing andperform real-time data decompression; capabilities that are also referred to asSQL in Silicon and Capacity in Silicon, respectively. TheDAX SQL in Silicon processors accelerate analytic queries by independentlysearching and filtering columns of data while the normal processor cores runother functions. Using Capacity in Silicon. processors perform datadecompression at full memory speeds, allowing large volumes of in-memory datato be kept in highly compressed format without incurring additional overheadfor decompression when accessed. This means you can drastically increase theprocessing speed of analytics, allowing your applications to optimally processmassive amounts of data while using the Oracle Database 12c In-Memory option -while executing OLTP transactions at top speed at the same time. The Security inSilicon functions of SPARC M7 continuously perform validity checks on everymemory reference made by the processor without incurring performance overhead.Security in Silicon helps detect buffer overflow attacks made by malicioussoftware, and enables applications such as the Oracle Database to identify andprevent erroneous memory accesses. The Security in Silicon technologies alsoencompass cryptographic instruction accelerators, which are integrated intoeach processor core of the SPARC M7 processor. These accelerators enablehigh-speed encryption/decryption for more than a dozen industry-standardciphers. In addition, SPARC M7 is the world’s fastest conventionalmicroprocessor. With industry leading 32 cores per processor running at 4.1 GHz,it has shattered virtually all performance benchmarks. Faster processors allowyou to run bigger workloads on smaller systems, thus lowering your software andpersonnel costs. Exadata SL6 ushers in a new era of hardware and softwareco-engineering through transformational Software in Silicon technology and thepure performance power of SPARC M7, combined with Exadata storage and networking.Faster processors enable much higher throughput that reduces the number ofservers required to run a task. Exadata SL6 also delivers 20-30% more IOs thanan equivalent Exadata X6-2 configuration and hence further lowers the totalcost of ownership of the solution. Exadata SL6 also offers elastic configurations.Customers can start with an eighth rack and incrementally add compute orstorage as and when needed. In a future post we will break down the costanalysis for SL6. The storage and networking subsystem in Exadata SL6 isidentical to Exadata X6-2. It uses the same InfiniBand infrastructure and HighCapacity and Extreme Flash storage servers. Since Exadata SL6 runs the exactsame Oracle Linux as its x86 counterpart, it supports all the same APIs andinterfaces. The platform becomes agnostic for most developer tasks andoperations. In summary, Exadata SL6 is the best-of-all-worlds databasesystem. It uses the fastest processors - SPARC M7, with the fastest storage -Exadata Storage. It uses ultra fast low latency InfiniBand networking withspecialized database aware protocols. Using SQL in Silicon it is able todeliver analytics at silicon speeds. Capacity in Silicon further improves theamount of data against which these analytics routines run. All access to systemmemory is secured using Security in Silicon, making this system one of theworld’s most secure. To top it all off, it leverages Exadata Software with thousandsof database-focused innovations such as SmartScan, Smart Flash Cache etc. thatnot only provide ground breaking database performance but also deliver carriergrade high availability. And all this goodness comes at the exact same listprice as Exadata X6-2. One might ask how Eaxadata SL6 fits in with similar offeringssuch as Oracle SuperCluster. SuperCluster also offers all the goodness of theSPARC M7; in fact it was a pioneer. SuperCluster runs Solaris and, with all theadvantages of Exadata Storage, ZFS and many other optimizations, is your ideal platformto run the most demanding database applications. SL6 broadens our portfolio tooffer the same SPARC M7 and Exadata Storage goodness in a Linux platform. Boththe products have their own differentiators and their own target use cases. Atthe end of the day it gives you, the customer, more choices to consume all theinnovation our products provide. We are fully committed to x86 based Exadatasystems as well; they are our flagship offering and we are not digressing fromthat path. Exadata SL6 will continue to co-exist with its older cousinsSuperCluster and Exadata X6. A little bit of sibling rivalry makes familiestighter and stronger. 

Since the launch of Exadata in 2008, Intel x86 processors have powered Exadata database servers.  Today we are expanding our Exadata family with the addition of Exadata SL6, our firstoffering with...

Exadata 12.2.1.1.0: The Best Database Cloud Platform Just Got Better

We are announcing the general availability of ExadataSoftware Release 12.2.1.1.0.  This is avery content rich release with over 30 features and enhancements that improveevery aspect of Exadata: faster analytics, high throughput low latencytransaction processing, and massive consolidation of mixed database workloads. Compellingenhancements in this release make the platform more secure and easier to manage.Most of these new features are available on all supported Exadata hardwaregenerations, thus ensuring complete investment protection in all Exadata deploymenttypes: On-premises, Cloud Service, or Cloud Machine. It’s difficult to do justice to all the enhancements in asingle post, so we’ll discuss some highlights in this one, followed by a seriesof posts discussing individual features. You can always engage with us throughthis platform or find us on Twitter @ExadataPM. The Exadata documentation describes thefeatures in detail and we also recorded a short webcast  coveringthe highlights. Exadata 12.2.1.1.0 implements query offload capabilities forOracle Database 12.2.0.1. All smart IO features - such as Smart Scan and Smart FlashCache - are now extended to Oracle Database 12.2, in addition to existingsupport for 12.1 and 11.2. Customers can run all three database versionssimultaneously on the same platform and leverage all of Exadata’s offloadcapabilities. At the same time they can also host 10.2 databases via ACFS,though we hope these customers will upgrade toOracle Database 12c soon. Let’s take a look at some of the features of this release inmore detail. Starting with analytics, Smart Scan gets an uplift with the abilityto scan compressed indexes. Queries involving compressed index scans benefitfrom this feature. Additionally,XMLExists and XMLCast can now be offloaded for LOBs as well. Speaking of LOBs,operators “LENGTH, SUBSTR, INSTRM CONCAT,LPAD, RPAD, LTRIM, RTRIM, LOWER, UPPER, NLS_LOWER, NLS_UPPER, NVL, REPLACE,REGEXP_INSTR, TO_CHAR” are now offloaded to the Exadata Storage Servers. Storage Indexes have been hidden, and not often talked about,yet are one of the most impactful features of Exadata. With a just simplemin/max summary information about a column, Storage Indexes help users savetons of unnecessary IO, thus improving analytic performance many fold. In thisrelease, Storage Indexes get a makeover as well. In addition to having min/maxinformation, Storage Indexes will now also store set-membership informationabout the column. One might wonder, what is this “set-membership” information?Let’s dig a little deeper. Min/Max works very well for columns with numericvalues, or dates, or columns with high cardinality, but with low cardinalitycolumns it might not work as well. For example, a column that represents stateswithin the U.S. A simplistic min/maxapproach will have a min of Alabama and a max of Wyoming and almost all querieswill result in an IO as all states will fall within the min/max rangeirrespective of the fact that the particular state data is present in thatcolumn. Set-membership addresses this use case. In addition to the min/maxvalue, set-membership stores the Bloom Filter representation of the hashedvalues of all distinct entries in the region. Storage Indexes and Bloom Filters are a perfect match. BloomFilters are very good at informing what’s not a part of the dataset, andStorage Indexes are trying to avoid IOs to a region that will not yield ameaningful result. Merging all of these together, Storage Indexes will now lookup a Bloom Filter to ensure if an IO is necessary for columns with lowcardinality, and all of this happens dynamically in a very efficient manner.Along with the set membership enhancement we also removed the eight-columnconstraint for Storage Indexes, so now we store summary information fortwenty-four columns. How Storage Indexes share space between min/max andset-membership data is a subject of a blog post on its own, so stay tuned. Shifting focus to in-memory databases, Exadata withIn-Memory Fault Tolerance is an ideal platform for running Oracle DatabaseIn-Memory. In fact, I have yet to meet a customer who needs the performance ofan in-memory database and can sustain an outage if something goes wrong withthe server running the instance. In-memory databases derive their phenomenal performance partlyfrom their ability to utilize SIMD vector processing at a CPU level, and theability to interact with columnar datasets in memory. Until today, in-memorydatabases have been constrained by the compute and memory resources of their databaseserver. This software release extends the benefits of Oracle Database In-Memorybeyond the conventional boundaries of the database server to Exadata storage. Thisis uniquely possible because Exadata uses a server centric approach to storage.Our storage servers are very similar in architecture to most 2-socket compute servers;we use the same family of microprocessors and have the same SIMD capabilitiesas the compute server. Here lies the opportunity; if we can use the SIMD vectorprocessing at the storage tier and redesign our flash cache to use In-MemoryColumnar formats, then we can deliver database performance at the speed of in-memoryprocessing, but with the capacity and economics of flash storage. That’sexactly what we are doing in this software release. In fact, we have extendedthe technology to use the offload capabilities in our storage to deliver in-memoryjoins and aggregation at the storage tier as well. But wait, it gets better. OracleDatabase 12.2 on Exadata also uniquely enables us to run in-memory processingon an Active Data Guard standby database, thus further enhancing the return oninvestment of Active Data Guard. One interesting tidbit… our published performance numbers donot even account for possible performance improvements due to technologies likeStorage Indexes or all the optimizations built in for Oracle DatabaseIn-Memory. So the platform can deliver much more analytic performance thanadvertised. Analytic workloads only make up a portion of jobs on Exadata.OLTP and mixed workload consolidation are the other popular workloads. Let’s discuss enhancements in this releasefocused on delivering better transaction processing on Exadata. OLTP databasesare a lot more impacted by variable performance of the IO subsystem. A fewreleases ago we introduced a series of features that targeted variable performanceof storage devices in the form of IO latency capping. We also introduced “instantdeath detection” of compute nodes via the InfiniBand Infrastructure. With this newrelease we are extending that portfolio to address rare but important IO latencyissues that can arise between compute and storage servers. Sometimes due tooutliers in the networking stack, or hardware issues with a storage server, anIO can experience a prolonged response time. In such rare cases the Exadata DatabaseServer will detect the anomaly and redirect Read IOs to another storage server.This maintains consistently low response times which is very critical for anOLTP database. We also added a lot of features to enhance consolidation onthe platform. By leveraging the Database 12c Multitenant option combined with resourcemanagement, we are able to uniquely deploy more than 252 Pluggable Databases (upto 4,000) per Container Database on an Exadata system running OracleMultitenant. Given the scale of consolidation on Exadata, it’s not uncommon tosee hundreds of databases being deployed on one Exadata system, and this combinationwill enable previously-unseen consolidation densities on Exadata. A lot of these databases could be test/dev databases andhence just implementing efficiencies at the compute tier will not suffice. Rather,the efficiencies have to extend to the storage tier as well. To address this weare enhancing the implementation of snapshots on Exadata. As you might know,Exadata snapshots are snapshots developed by database engineers for databaseengineers. You get the performance ofExadata with the storage efficiencies of a copy-on-write solution. Thissoftware release implements hierarchical snapshots (snapshots of snapshots) onExadata storage. A hierarchical architecture enables real time updates for thetest master database, thus allowing tenants to periodically refresh testdatasets without incurring additional storage costs. In addition, the releasealso enables sparse backups of the snapshots, passing down the storage savingsto secondary storage as well. No discussion about Exadata is complete if we don't include theavailability and manageability aspects of the system. However, improving the availabilityof an already “five-nines” system is a tough ask. We did take that on as achallenge and delivered some unique optimizations in this release. With aserver-centric approach to storage we have compute resources that can be deployedto recover from a possible hardware error. This release allows users todynamically allocate more compute resources when data needs to be redistributedacross devices in the event of a failure. In addition, we restore critical information first, as the system has avery clear understanding of what’s most important in terms of availability. Toadd to this, we are now also leveraging data in our flash cache to speed uprestoration of redundancy (mirroring) and availability in the event of a mediafailure. On the manageability front, we challenged ourselves todeliver on many fronts – upgrades, security, monitoring etc. We improved thesoftware upgrade times by 2x over the prior major release 12.1.3.3.0, whichmakes for an overall improvement of 5x in the last 12 months. In addition to upgrade times we also improvedExawatcher by introducing graphs and added the ability for Oracle Installer to nowinstall quorum disks: bye-bye quorum disk utility. IP addresses, DNS settingsand NTP configurations can now be changed online, so you can bring new tenants onboard easily. We now offer “full stack secure erase”, so you can take tenantsoffline easily and securely as well. There are more advantages to secure erasethan mentioned here, but we’ll address them in a future post. A lot of ourcustomers are developing management frameworks using the REST interface. Tofacilitate this we are extending our REST service to manage database servers inaddition to storage servers. Hopefully this post gives you an overview of the innovationsand enhancements we are delivering with the new Exadata 12.2.1.1.0 softwarerelease. Subsequent posts will go into more detail about these features. Almosteverything discussed in this post comes at $0 costs to you; just upgrade yourExadata software and you are good to go. Heck, you don't even need to commithuman resources for doing that; Oracle Platinum Services will take care of itfor you. No charge.

We are announcing the general availability of Exadata Software Release 12.2.1.1.0.  This is a very content rich release with over 30 features and enhancements that improveevery aspect of Exadata:...

Oracle Database Exadata Cloud Machine now available

Continuing with our cloud innovation, we are very pleased to announce the general availability of Oracle Database Exadata Cloud Machine (ExaCM). For the first time in IT, now you can deploy the most powerful Oracle Database - with all options and features, on the most powerful Database platform - Exadata, with all the cloud computing benefits, at your data center, behind your firewall. The associated Exadata infrastructure is managed by Oracle Cloud experts, which means you can focus on your primary business instead of having your limited IT resources stretched with mundane tasks such as infrastructure updates. If you have cloud computing on your strategic IT roadmap, but you couldn't make that leap for whatever reason, now with ExaCM, you can embrace the Cloud without having to compromise SQL functionality, performance, availability, or transactional integrity. For example - you are considering a tech refresh and are starting to like the cloud pay-as-you-go model.  Also, you are just starting to think about migrating some of your Oracle Databases to the Public Cloud. And right then, as Murphy's Law would have it, you are told that the company policy is that those databases cannot leave the company data center, or even your country's political territory. These policies may have something to do with data sovereignty / data residency policies. Or on a more technical level - you identified these databases - but then you find out that there are all these legacy applications talking to those databases, whose developers are no longer with the company and it would take a super human effort to migrate those applications to a cloud service.So this is what Exadata Cloud Machine is about - it brings our Cloud to you. But as the TV ad goes - we ain't stopping there! You see, an inherent beauty of our Oracle Database cloud strategy is that no changes to applications or data models are required either. This means that if you are running Oracle Databases on-premises today - rest assured, your investment is protected - you can continue to do so, but at the same time, you can start your cloud journey - whether in Oracle's public cloud data centers, or in your own data center, without having to make any compromises. And with Exadata, it doesn't matter whether the workload is OLTP, Analytics or a consolidation effort, it's the same unified platform. This is what we refer to as a consistent experience across our various Exadata deployment models - on-premises Exadata, Exadata Cloud Service or Exadata Cloud Machine. Since it is the same Oracle database, you can leverage the same in-house Oracle skillset that you have accumulated over many, many years. And because it is Exadata, it is the same platform that thousands of other customers have deployed across all verticals, around the world, and maybe even yourself. This is what investment protection is all about.Interested? Please get started here. The Exadata Cloud Machine datasheet is located here.There will be additional interesting updates on our Exadata Cloud strategy in this blogspace, so please stay tuned. Meanwhile, let us know how your enterprise cloud journey looks like!

Continuing with our cloud innovation, we are very pleased to announce the general availability of Oracle Database Exadata Cloud Machine (ExaCM). For the first time in IT, now you can deploy the most...

Exadata: Database Engineering at Cloud Scale

In our neck of the woods here in Silicon Valley, it’s not uncommonto find a lot of startups working on unique challenges regarding DatabaseEngineering. Some are attacking the scalability aspect, some are attempting to reducethe latency of an OLTP database, others are trying to make the data warehouserun faster, while others are focused on bringing the latest innovations insilicon technology to database systems. Exadata is the culmination of all thosechallenges and then some. We are constantly pushing the technology onthree frontiers – performance, availability and scalability - while constantlyreducing the cost of system ownership. Our mandate is to build the World’s fastest,most available and most cost effective systems. This mandate usually leads usto adopt newer hardware technologies and harden existing ones, to challenge theconventional wisdom of solving problems and to be proactive and engineer forthe future needs of our customers rather than narrowly focusing on the problemof the day. Speaking of our customers, we might have the industry’s widestspectrum. Some are running Petabytes large warehouses, with billions of queriesper day; some are running mission critical transactional databases, withbillions of updates per day (yeah that’s right, it’s billions with a B); whilesome are just trying to reduce the cost of their test dev platform byconsolidating databases. Over the last year alone I have met nearly one hundredExadata customers worldwide and from every industry vertical. Any geography youpick, you’ll find Exadata well represented. One heavy-user government customer we partnered with during theearly days of Exadata shared their business problem with us. They were buildinga system to screen all the cargo that enters the country every day to identifymalicious shipments. Such a system, for each individual cargo item, will have toquery multiple databases, crosslink varied sets of information, come up with adecision and update various systems instantaneously. Now repeat this workflowfor billions of items with seasonal fluctuations. And one last thing, thesystem has to remain up all the time, even throughout planned and unplannedmaintenance. When an issue with the system you work on has the potential tomake breaking news on CNN even before it hits your mailbox then you have tothink differently. These customers not only help make our products betterbut also collectively push the technology to the next frontier. That's’why every startup in this space aspires to beat our performance numbers and alot of incumbents are constantly trying to match our availability andscalability stats. Exadata has increasingly become the platform of choice for theconsolidation of Oracle databases. Consolidating databases on a single platform makes a lot ofsense from an operational expense and manageability perspective. To some,consolidation may mean packing a dozen databases running the same type ofworkload on some kind of converged infrastructure. When we looked at it we challengedourselves to design a system that allows customers to run competing workloadswithout compromising performance or availability. OLTP and data warehouseworkloads have very different constraints and very different IO and CPU usagepatterns, so running them concurrently requires some precision engineering. Youneed to make sure the data warehouse workloads don't consume the entire IObandwidth. At the same time the latency sensitive OLTP workloads must get theirrequired TLC. But running hundreds of these workloads on a single systemrequires a completely different level of engineering. It exposes a whole newset of problems and challenges, some never before addressed by the industry.It's not uncommon to find customers running over 100 databases at full throttleon an eighth rack Exadata, our smallest configuration. If you are from our neck of the woods - Silicon Valley and the SanFrancisco Bay Area - then you have a unique opportunity to hear how we approachand solve these unique challenges. In particular how we shave off microsecondsof an OLTP IO or are able to achieve hundreds of gigabytes per second of scanbandwidth and at the same time maintain a five-nines availability, and how wedo all of the above at scale. Kodi Umamageswarn, Vice President of Exadata Development,one of the guys who took Exadata from an idea to an industry-leading product,will speak at the Northern California Oracle User Group on August 17, 2016. Youcan register here for theconference. 

In our neck of the woods here in Silicon Valley, it’s not uncommon to find a lot of startups working on unique challenges regarding DatabaseEngineering. Some are attacking the scalability aspect, some...

Applications and Exadata: Why Select Exadata

Exadata is now in its sixth generation with thousands ofExadatas deployed at thousands of customers. The growth rate remains strong while systems become more powerful,pricing remaining relatively flat, and acquisition models (Capacity on Demandand Infrastructure as a Service Private Cloud) have made acquiring Exadata moreattractive. There are many reasons why customers and partners choose runthe application’s database on Exadata. Most of the reasons to are focused on performance. We’ll see that performance takes many forms. We’ll briefly investigate some of the reasons,outside of Simplifying IT, why customers and partners decide to run theirapplication databases on Exadata. Reliability and Availability This largely has to do with application downtime. Performance manifests itself through theimplementation of Oracle’s Maximum Availability Architecture on Exadata. A new Exadata customer (Fortune 400) recentlyoffered that they were a best of breed Dell and EMC shop prior to selectingExadata. “We did not buy Exadata forperformance (speed). We bought it(Exadata) because we can’t afford to have our ERP system down every week.” Because the database is always up andrunning, the customer sees tremendous improvements in productivity in both ITand its lines of business. Scalability Exadata with its scale out architecture demonstratesexcellent and often near linear scalability. With Exadata customers know what to expect as their application workloadgrows. A major retailer was rolling outnew retail management and check out software across 1500 stores. At 500 stores the current compute and storageenvironment suffered a “melt down” and could not support the 500 stores, letalone the 1500 required. This retailcustomer had prior Exadata experience in their supply chain and quickly movedto Exadata to achieve the needed performance and scalability.  In this situation, getting Exadata up andrunning quickly was essential to meeting their roll out plans (see Time to DeployApplications) Time to Deploy Applications Oracle delivers Exadata hardware and softwarepre-configured, pre-tested and pre-tuned. This upfront engineering dramatically decreases the time to stand-upExadata to get the database(s) and application(s) operational. With a singlepatch from Oracle for planned maintenance customers now test only once, asopposed to testing after each patch is applied. Further, applications run on Exadata unchangedresulting in straightforward migrations, testing and turn up. Sherwin Williams reported that it reducedscheduled application downtime from 200 hours (budgeted) to 65 hours as itupgraded from E-Business Suite 11i to E-Business Suite R12.  Productivity and new business process areimplemented sooner along with realizing the benefits of the applications sooner. Supportability Because Exadata is pre-configured and deployed Exadata systemslook alike, Oracle’s has an incomparable ability to monitor, maintain andsupport Exadata. This enables Oracle tooffer Platinum Support. Sprint replacedan aging best of breed system with Exadata and observed “(Exadata is) A platform that is stableenough where I don’t get a call in the middle of the night.” And, followed with: “One of the things thathelped us a great deal was support – Platinum Support Services that come withExadata. Oracle has notified us beforewe actually realized we were having a disk problem.” Pulte Group, a large US based home builder, andmany many other Exadata customers have shared similar experiences. Performance Applications that run with Exadata see dramatic boosts inperformance. These performance boostslead to gains realized by the various lines of business. An insurance company can load its datawarehouse with Exadata 4 times a day instead of once a day so it has fresh datafor its analyses. Sprint improved theproductivity of its call center operators at least 30% with faster queryresponses. A large grocer was able tonegotiate lower pricing and better delivery from its suppliers with faster,more accurate and longer forecasts. Exadata performance manifests itself in many ways: fromSimplifying IT to allowing customers to Work Faster and Smarter with real timebusiness intelligence, deeper analysis, near real-time batch processing and Internetscale OLTP.   The performance gainsdeliver clear business benefits to various lines of business within anorganization - reducing costs and creating new opportunities.

Exadata is now in its sixth generation with thousands of Exadatas deployed at thousands of customers.  The growth rate remains strong while systems become more powerful,pricing remaining relatively...

10 reasons to run Database In-Memory on Exadata

Exadata istouted by Oracle as the “best platform for running the Oracle Database”. Thevirtues of Exadata for dramatically improving database I/O performance are welldocumented and overwhelmingly supported by customer references. But is Exadata alsothe best platform for running Oracle Database In-Memory? After all, bydefinition an in-memory database avoids I/O by holding data in DRAM. Doesn’tthat mean Exadata has no benefits for running an in-memory database compared tosimilar x86 systems? The simpleanswer is “no”. Exadata is definitely the best platform for running OracleDatabase In-Memory, at least for any important database. If your in-memorydatabase is small enough to fit entirely in DRAM and you can tolerate periodswhen the data isn’t available, and all you wish to do is read-only analyticsand occasional data loads, then Exadata may be overkill. It’s unlikely youwould be considering Exadata for such a situation anyway. Let’s discuss the 10reasons to run Database In-Memory on Exadata: #1.On Exadata, you can configure an in-memory database to be fault-tolerant.In-memory fault-tolerance is only available on Oracle Engineered Systems. #2.Your database can exceed the size of DRAM and scale to any size across memory,flash and disk, with complete application transparency. #3.When your data doesn’t all fit in memory, you still get outstanding analytics performanceon Exadata from disk and flash. For the same reasons, populating the in-memorydata store from storage is very fast. #4.OLTP is fastest on Exadata and you can run in-memory analytics directly againstan OLTP database. Exadata enables Real-time analytics better than any otherplatform. #5.Exadata is an ideal database consolidation platform and Oracle DatabaseIn-Memory enhances that value even further. In many ways Oracle DatabaseIn-Memory “completes” Exadata by applying in-memory performance techniques thatare similar to those used on disk and flash. The entire storage hierarchy(DRAM, flash, disk) delivers maximum value. Adding Oracle Database In-Memory toan existing Exadata system makes tremendous sense. #6.Exadata is highly optimized for RAC clustering, with special protocols overInfiniBand for low-latency cluster interconnect. RAC clustering is how DatabaseIn-Memory fault-tolerance is achieved and how large databases scale-out inmemory. No other platform supports RAC with in-memory like Exadata does. #7.On Exadata, the use of Oracle VM Trusted Partitions is allowed, reducingsoftware license costs. This is especially relevant for database options thatrequire less compute power than a full server. #8.Exadata elastic configurations, introduced in Exadata X5, enable custom configurationsthat are perfectly tailored to specific workloads. If you need more memory butnot a lot of storage capacity you can create that configuration and onlypurchase the Exadata servers needed for the job. #9.Exadata is the development platform for Oracle Database In-Memory. Issues arediscovered and fixed on Exadata first. #10.Exadata is also the primary platform for Oracle Database testing, HA bestpractices validation, integration and support. The same reasons it is the bestplatform for Oracle Database apply to Oracle Database In-Memory.

Exadata is touted by Oracle as the “best platform for running the Oracle Database”. The virtues of Exadata for dramatically improving database I/O performance are welldocumented and overwhelmingly...

Controlling Software Costs on Exadata

Users of Exadata Database Machines can now take advantage ofseveral techniques to control database software license costs including Capacityon Demand (CoD), Elastic Configurations, and Trusted Virtual Machine Partitions. Exadata Capacity on Demand allows users to disable a subsetof the cores on Exadata database servers to reduce licensing requirements. Themaximum number of cores that can be disabled on an Exadata X5-2 Database Serveris 60%. In other words, with Exadata Capacity on Demand, users only need tolicense 14 cores per Exadata X5-2 Database Server even though there are 36physical cores, thus greatly reducing the number of software licenses requiredper server. Exadata Capacity on Demand allows additional cores to be enabledand licensed as needed. Exadata comes with all the necessary software tools toprovision and manage Capacity on Demand. On other x86 servers, all the physical cores on the machinemust be licensed.  This means thatcustomers must plan for the maximum possible core usage over several years, andcarefully deploy a server whose number of cores comes close to this instead ofdeploying standard sized servers. Incontrast, Exadata customers can enable just the cores they need at deploymenttime, and enable additional cores as they need them. Exadata Database Machines can now be Elastically expanded byadding compute and storage servers one at a time as they are needed, extendingthe concept of deploying just the processors that are needed to entire serversin addition to the cores within a server. Exadata now supports Oracle Virtual Machines (OVM). OVM isbased on the industry standard XEN virtualization technology that is also thebasis of Amazon.com’s cloud. OVM can be used to pin virtual machines tospecific cores to reduce licensing requirements in a manner similar to CoD.  Further, OVM on Exadata implements “TrustedPartitions”. Trusted Partitions are much easier to use, and are more flexibleand dynamic than core pinning, and they enable licensing of software on Exadataby Virtual Machine instead of physical core. One benefit of licensing by virtual machine is that it facilitatesconsolidation of databases running different database options since the optionscan be licensed on just the virtual machines using them instead of everyphysical core in the server. To learn more about “Trusted Partitions” onExadata please refer to the Oracle Partitioning Document.Note that unlike some other virtualization solutions, there is no licensecharge for OVM software or management, and no additional support fees for OVMsince OS and VM support is included with standard hardware support. Our customer base and the analyst community has respondedvery positively to the unique direction we have taken in providing them withbetter flexibility and improved cost on Exadata. Please refer to the Gartner article“New X5 Generation Will Bring Pricing Improvements to Oracle Exadata” thatdiscusses these topics. For an overview of Exadata X5 and how it enables you tospend less, please see the X5 launch video. Specific comments on how to “Spend Less by Paying Less” can be found inthis section of the video.

Users of Exadata Database Machines can now take advantage of several techniques to control database software license costs including Capacityon Demand (CoD), Elastic Configurations, and Trusted...

Prereq Dependency Check Added for Exadata DB Server Updating

With this blog update some background on new functionality added to the Exadata dbnodeupdate.sh utility.  The recently added 'Prereq Dependency Check' feature in dbnodeupdate.sh eliminates possible unexpected package dependency problems that occur during update by detecting dependency issues during the prerequisite check phase that could occur when the update is performed. (Linux package dependency issues sometimes happen after operators customizing the db server by installing additional software packages or newer releases of packages). The recommended use of this feature is to run dbnodeupdate.sh with the –v flag (to verify prereqs only) days before the scheduled maintenance.  dbnodeupdate.sh will report dependency problems that must be resolved before proceeding with the update. Note that the dependency check functionality is also run by default prior to performing the update. dbnodeupdate.sh will now also list what packages will be removed for your update. Some details: Updates starting from 11.2.3.1 - 11.2.3.2.2 to any release earlier than 11.2.3.3.0: Dependency check is validated against 'standard' dependencies. Updates starting from 11.2.3.1 - 11.2.3.2.2 to any release equal to or later than 11.2.3.3.0: Dependency check is first validated against 'exact' RPM dependency. If 'exact' RPM dependency check passes it is assumed 'minimum' RPM dependency check will also pass. If 'exact' RPM dependency check fails then 'minimum' RPM dependency check is run. Updates starting from release 11.2.2.4.2 do not have Prereq Dependency Check functionality. dbnodeupdate.sh will report what checks were executed for your update and which of them did 'pass' or 'fail' If the dependency check is executed as part of dbnodeupdate.sh –u and only 'minimum' RPM dependency check passes, then the new target will implicitly be changed to 'minimum' (which is equal to dbnodeupdate.sh -m).  If the dependency check is executed as part of dbnodeupdate.sh –u and both 'exact' and ’minimum’ RPM dependency checks fail, then the operator will not be able to proceed with the update. For dependency checks that fail a separate report is generated.This report highlights to the failing package. The operator can then decide to either remove/install/update the failing package depending on what works best for that particular server. Examples: Prereq run here -this is a prereq only run. Notice the ''Exact' package dependency check failed' and the ''Minimum' package dependency check succeeded' Failing dependencies here  - for more details on what package cause the problem and what can be done to resolve it. Update scenario here - see the same dependency checks and notice 'Update target switched to 'minimum'' Existing backups of the current image overwritten by default: Existing backups of the current image on the inactive lvm will be overwritten by default. You can decide to skip (and retain) the backup by using the "-n" flag.  Rene Kundersma 

With this blog update some background on new functionality added to the Exadata dbnodeupdate.sh utility.  The recently added 'Prereq Dependency Check' feature in dbnodeupdate.sh eliminates possible...

Updating database servers using dbnodeupdate.sh - part 3

When running dbnodeupdate.sh for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this  'location' the repository'.  The repository comes in two different flavors: as an ISO and as Oracle ULN channel. Let's start with the ULN channel.   Exadata ULN Channels For the different Exadata releases starting 11.2.3.1.0 a corresponding 'base channel' is available on linux.oracle.com (ULN). Exadata customers can register to ULN with their CSI, subscribe to the channel (e.g. patch/release) they require and then synchronize one or more channels to with the local (in house) repository. This local repository can then be made available to the internal data-center by publishing it via a web server.  Note: it is not recommended to use an Exadata db server as local YUM repository Instructions how to subscribe a system to ULN and synchronize to a local repository are the same as for Oracle Linux and Oracle VM, so generic instructions can be found on OTN here. The README of a specific Exadata release will always mention what channels are made available for that release. You can also find this information via MOS 888828.1 and also in 1473002.1. Additional to the 'base channel', there is also a 'latest channel'. Currently for Exadata there is a latest channel for 11.2 and 12.1 releases. The content of the 'latest' channel will never remain the same (unlike the 'base' channel) as long as there will be updates for that 11.2 or 12.1 release. When for example a new Exadata 12 release will be published this will be added to the existing latest channel (in addition to a 'base channel' also made available). This is the primary reason for the 'latest' channel being much larger (and taking more time to synchonize) than a base channel. For Exadata installations on release later than 11.2.3.2.1, the 'latest' channel brings additional options. With the latest channel it's now possible to specify what release you want to update to. For example when on 11.2.3.3.0 planning to update to a later (but not the latest 12.1.release, just as an example) you can use the 'latest' channel and specify the "-t" flag to specify what release you want to update to.  Note that this can only be done with the 'latest' channel and that without specifying the "-t" argument by default the db server will be updated to the most recent release it can find in that channel. Of course there is also the option to just grab the 'base' channel and update without specifying any "-t' option. Examples updating with a latest channel specifying no argument (latest release in the channel will be used) here updating with the latest channel to a specific release that not exists (a typo) here updating to a specific release here Exadata channel as ISO  For those not able or willing to synchronize repositories with Oracle ULN there is also an ISO image available. The ISO image is built (and zipped) by Oracle and is only available for  'base' channel content. An ISO is ~1GB and about the same size as the sum of all packages of the corresponding base channel on ULN. Using ISO or ULN From an 'update' perspective there isn't much difference between using ISO or a http repository, only the location (-l) changes: For local YUM repositories (synchronized from Oracle ULN): ./dbnodeupdate.sh -u -l http://myrepo/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/ For using an ISO (example with the 11.2.3.3.0 iso): ./dbnodeupdate.sh -u -l ./p17809253_112330_Linux-x86-64.zip  The ISO file should not be unzipped and there is no need to make an local 'loop mount' to use the iso - this is all done by the dbnodeupdate.sh script Validations For each type of repository some validation checks will be done to see if it a usable is a repository, checks are done for expected files and also if the available Exadata release in the repository is a later release than the one currently installed - because if not, an update would not be possible.  When specifying an http repository it's required to specify the top level directory containing the 'YUM metadata repository directory'. Typically this is the directory that has the 'repodir' directory in it. (see example here). When an http location cannot be identified as a valid repository an you would see a suggestion how to locate the right url. Rene Kundersma

When running dbnodeupdate.sh for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this  'location' the...

Updating database servers using dbnodeupdate.sh - part 2

Within the Oracle Exadata Database Machine documentation and README's you will generally find two types of backups for database server OS backup: The Exadata Database Machine Owners Guide (Chapter 7) has instructions to create backups stored outside of the dbserver, for example on an NFS mount (see 'Creating a Snapshot-Based Backup of Oracle Linux Database Server') dbserver_backup.sh - which creates a local copy of your active lvm. In this post I will explain background and usage for both backups and how they integrate with dbnodeupdate.sh dbserver_backup.sh For backing-up and rolling-back Exadata dbserver OS updates  the dbserver_backup.sh script is used by dbnodeupdate.sh. For each upgrade by default the dbserver_backup.sh script is executed. When executed (either manually or via dbnodeupdate), the dbserver_backup.sh script creates a small snapshot of the 'active' sys lvm. The active sys lvm is the primary lvm that your current OS image is running on. For example: [root@mynode ~]# imageinfo Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64Image version: 11.2.3.3.0.131014.1Image activated: 2014-01-13 13:20:52 -0700Image status: successSystem partition on device: /dev/mapper/VGExaDb-LVDbSys2 In the above example the active lvm is /dev/mapper/VGExaDb-LVDbSys2.The snapshot is created to have a 'consistent' 'view' of the root filesystem while the backup is made. After the snapshot is created, it's mounted by the same script and then it's contents are copied over to the inactive lvm. For lvm enabled systems, there are always 2 'sys' lvm's "VGExaDb-LVDbSys1" and "VGExaDb-LVDbSys2". VGExaDb-LVDbSys2 will automatically be created (on lvm enabled system) if not existing yet. For the example above, the 'inactive' lvm will be VGExaDb-LVDbSys1 Now, depending on how many files there are in the root (/) filesystem (based on your active sys lvm) the backup times may vary. Previous Grid and Database home installation zip files in /opt/oracle.SupportTools/onecommand will make the backup take longer (not the restore, which I will explain why later). Same for those who have many small files (like mail messages in /var/spool) - the backup may take longer.  One of the first steps the dbnodeupdate.sh script will doing when executed is making a backup with this script. Now, if you want to shorten your downtime and make this backup before the start of your 'planned maintenance window' you have 2 options: Either execute the dbserver_backup.sh script yourself or use dbnodeupdate.sh with the "-b" flag to make a backup only before hand. Example making a backup with dbnodeupdate.sh here (see 'Backup only' for 'Action') When you then have the downtime for planned maintenance and already have the backup you can then let dbnodeupdate skip the backup using the "-n" flag. Example skipping a backup with dbnodeupdate.sh here (See 'Create a backup: No') Both Sys lvm's are 30GB each. The snapshot that will be created is ~1GB. It is recommended to keep this in mind when claiming the free space in the volume group to make your /u01 filesystem as big as possible. (the script checks for 2 GB free space in the volume group) Now, when the update proceeds, the current active lvm will remain the active lvm. This is different than what happens on the cells where the active lvm becomes inactive with an update.  Typically you will only switch active sys lvm's when a rollback needs to be done on a db server, for example, an upgrade from 11.2.3.3.0 to 12.1.1.1.0 needs to be rolled-back. What happens then is nothing more than 'switching' the filesystem label of the sys lvm's, updating grub (the bootloader) and restoring the /boot directory (backed up earlier also by dbnodeupdate.sh). Then, a next boot will now have the previous inactive lvm as active lvm. Rolling back with dbnodepdate.sh as in the example here (a rollback from 11.2.3.2.1 to 11.2.2.4.2)  After booting the node, it's recommended to run dbnodeupdate.sh again with the "-c" flag to relink the oracle home's again. Remarks: It's important, to make a new backup before attempting a new update. In the above example, there is only talk about the sys lvm's. This means custom partitions including /u01 are not backed up. For regular node updates this is enough to rollback to a previous release but it's recommended to also have a backup of other filesystems inline to your requirements. Nodes deployed without lvm will not have this option available Rolling back db servers to previous Exadata releases with this procedure does not rollback the firmware Backup / restore procedure owners guide chapter 7  The backup made with the procedure in chapter 7 of the Oracle Exadatabase Database owners guide covers total node recovery.  Like the dbserver_backup.sh procedure a snapshot is used for a consistent view, then in this scenario a copy is placed outside of the db server (via NFS in this example).  This procedure allow you to backup every filesystem you require. In case of emergency - such as a non-bootable system, the node can be booted with the diagnostic iso. For non-customized partitions an interactive script will then question you to provide backup details and recover the node completely. For customized partitions steps (which are almost the same) can also be found in the owners guide. Advantages /  Disadvantages Both type of backups serve another goal. Also, these are just examples - of course customized backup and restore scenario's are also possible.The procedure as described in the owners guide requires external storage, while the dbserver_backup.sh script uses space on the node - but that is also where the risk is. The backup made with dbserver_backup.sh works well for the purpose of rolling back upgrades. With the automation of dbnodeupdate.sh rollbacks can be done simple and quickly. However - loss of critical partitions and/or filesystems will not be covered with this type of backup - so you may want to combine both types of OS backup. The general recommendation is to use the default built-in backup procedure when running dbnodeupdate to make easy rollback possible. But also backup the entire OS and customized filesystems outside of the database server with an interval based on your own requirements. Rene Kundersma

Within the Oracle Exadata Database Machine documentation and README's you will generally find two types of backups for database server OS backup: The Exadata Database Machine Owners Guide (Chapter...

Updating database servers using dbnodeupdate.sh - part 1

In this and future posts I am planning to describe some new functionality and background of dbnodeupdate.sh starting with Oracle Exadata Database Machine release 11.2.3.3.0. Some of this functionality will be directly available to the operator via the interface and can actually be used via an argument, however, some of the recent changes are made to make patching even easier, reduce human error and downtime. You may also find some of the 'new features' described in MOS 1553103.1 'Exadata Database Server Patching using the DB Node Update Utility' Exclusion/Obsolete list  With updates to Exadata 11.2.3.3.0 or later some packages on the database server will become obsolete. When updating a db server the dbnodeupdate.sh script will mention an 'RPM exclusion list' and an 'RPM obsolete list' in it's confirmation screen. The 'RPM obsolete list' will list the all packages that will be removed by default during the update to 11.2.3.3.0 (or later) when no action is taken by the operator. As an example - click here If you would like to find out first what obsolete packages will be removed you have to choose 'n' when prompted to 'Continue ? [Y/n]'. This will stop your current patching session. Then look at the contents of the freshly created 'obsolete list' (it should have the runid of your dbnodeupdate session in it's header). Example here All the packages listed in the '/etc/exadata/yum/obsolete.lst' file will be removed by default - this has multiple reasons, mainly this is because these packages are not required anymore for Exadata functioning or they are considered a security risk. In case you would like to keep for example the 'java' package, you should create an 'exclusion file'  which is '/etc/exadata/yum/exclusion.lst' and put the 'java*openjdk' rpm name (or wildcard) in it. Example: [root@mynode u01]# cat /etc/exadata/yum/exclusion.lstjava-*-openjdk When dbnodeupdate.sh  is restarted you would see that the 'exclusion file' is detected (an example here). All packages you have put in the 'exclusion file' will still be listed in the obsolete file, but will not be removed when the confirmation screen says 'RPM exclusion list: In use (rpms listed in /etc/exadata/yum/exclusion.lst)'  with the update to 11.2.3.3.0 or later. Frequent releases of dbnodeupdate.sh - keep an eye on it: dbnodeupdates.sh and it's MOS note were designed / made to be released frequently and quickly when needed.This way dbnodeupdate.sh can provide workarounds for known issues, emergency fixes, new features and best practices to the operator relatively quick.This reduces risk of people not keeping up to date with 'known issues' or not being sure it applies to them. Basically the same idea as with the patchmgr plugins. Also, unlike to the storage servers some customization can be done on the db servers - best practices and lessons learned from this in regards to patching may also benefit your environment. For those engineers involved with upgrading Exadata database nodes, I'd like to emphasis to always check for the most recent release of dbnodeupdate.sh when doing a db server update. I have already seen some people watching the releases closely, which is definitely a good thing. Rene Kundersma

In this and future posts I am planning to describe some new functionality and background of dbnodeupdate.sh starting with Oracle Exadata Database Machine release11.2.3.3.0. Some of this functionality...

Enhanced lights-out patching for Exadata Storage Cells

The recently released 11.2.3.3.0 Oracle Exadata version comes with multiple enhancements in the patchmgr utility. For those who don't know what the patchmgr utility is: the patchmgr utility is a tool Exadata Database Administrators use to apply (or rollback) an update to the Oracle Exadata Storage Cells. The enhanced patchmgr will have an option to send the operator an email for the most significant patch and rollback state changes. This eliminates the need to monitor the screen while the update or rollback is in progress. In order to send the email you need to specify values for the '-smtp_from' and '-smtp_to' flags. Example as follows: ./patchmgr -cells ~/cell_group -patch -rolling \ -smtp_from dbma@yourdomain.com \  -smtp_to your-email@yourdomain.com Patchmgr will use the sendmail package which  is installed by default on the Exadata Database Server. It will start sendmail if it's not already started and assumes it's configured to deliver email for the domains you specify. You will recognize the format of the alerts patchmgr sends as they have the same formatting as the ASR emails. The email you will receive when you enable this option can have the following end states for a cell:  Started Start Failed Waiting Patching Rolling Back Failed Succeeded.  Not Attempted  The majority of the above states probably is self explanatory, however explaining 'Not Attempted' may help: 'Not Attempted' will be the final state of a cell when patching or rolling back (in a rolling fashion) of the current cell has failed. In that case the patching will stop and the remaining cells (which are not touched yet) are in the state 'Not Attempted'. So for example: imagine you are patching cel1,cel2 and cel3. When cel1 fails patching, then the end state for cel2 and cel3 will be 'Not Attempted'. The following example will give you the idea: Patching starts in a rolling fashion from 11.2.3.2.0 to 11.2.3.3.0 for cel1,cel2 and cel3.  Note: this is an example of the type of state changes you will see, actual patch timings are not correct and not relevant for this example. Also actual states or formatting may change in your release. 1. patchmgr was started in a rolling fashion. Initial state for all cells is 'Started'. This should be seen as this initialization phase. You will receive an email as follows: 2. The actual patching has begun since this is in rolling fashion, patchmgr starts with the first cel listed, which is cel1. The other two cells are waiting until cel1 finishes. 3. After some time patching of cel1 has completed. You will receive another status update in your inbox stating cel1 was updated successfully. 4. patchmgr continues with the next cell. 5. When succeeded (or failed) you will receive a state update via mail. 6. Last cell to be patched is cel3: 7. The last email you will receive will give an update stating patching was completed and lists the end state for all cells. In case patching or rolling back would fail, you would receive a pointer to a release specific MOS note where you may find additional information.Rene Kundersma 

The recently released 11.2.3.3.0 Oracle Exadata version comes with multiple enhancements in the patchmgr utility. For those who don't know what the patchmgr utility is: the patchmgr utility is a tool...

Demo on Data Guard Protection From Lost-Write Corruption

Today I received the news a new demo has been made available on OTN for Data Guard protection from lost-write corruption. Since this is a typical MAA solution and a very nice demo I decided to mention this great feature also in this blog even while it's a recommended best practice for some time. When lost writes occur an I/O subsystem acknowledges the completion of the block write even though the write I/O did not occur in the persistent storage. On a subsequent block read on the primary database, the I/O subsystem returns the stale version of the data block, which might be used to update other blocks of the database, thereby corrupting it.  Lost writes can occur after an OS or storage device driver failure, faulty host bus adapters, disk controller failures and volume manager errors. In the demo a data block lost write occurs when an I/O subsystem acknowledges the completion of the block write, while in fact the write did not occur in the persistent storage. When a primary database lost write corruption is detected by a Data Guard physical standby database, Redo Apply (MRP) will stop and the standby will signal an ORA-752 error to explicitly indicate a primary lost write has occurred (preventing corruption from spreading to the standby database). Links: MOS (1302539.1). "Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration" Demo MAA Best Practices Rene Kundersma

Today I received the news a new demo has been made available on OTN for Data Guard protection from lost-write corruption. Since this is a typical MAA solution and a very nice demo I decided to...

Exadata X3, 11.2.3.2 and Oracle Platinum Services

Oracle recently announced an Exadata Hardware Update. The overall architecture will remain the same, however some interesting hardware refreshes are done especially for the storage server. Each cell will now have 1600GB of flash, this means an X3-2 full rack will have 20.3 TB of total flash ! For all the details I would like to refer to the Oracle Exadata product page: www.oracle.com/exadata Together with the announcement of the X3 generation. A new Exadata release, 11.2.3.2 is made available. New Exadata systems will be shipped with this release and existing installations can be updated to that release. As always there is a storage cell patch and a patch for the compute node, which again needs to be applied using YUM. Instructions and requirements for patching existing Exadata compute nodes to 11.2.3.2 using YUM can be found in the patch README. Depending on the release you have installed on your compute nodes the README will direct you to a particular section in MOS note 1473002.1. MOS 1473002.1 should only be followed with the instructions from the 11.2.3.2 patch README. Like with 11.2.3.1.0 and 11.2.3.1.1 instructions are added to prepare your systems to use YUM for the first time in case you are still on release 11.2.2.4.2 and earlier. You will also find these One Time Setup instructions in MOS note 1473002.1 By default all compute nodes that will be updated to 11.2.3.2.0 will now have the UEK kernel. Before 11.2.3.2.0 the 'compatible kernel' was used for all compute nodes other than X2-8. For 11.2.3.2.0 customer will have the choice to replace the UEK kernel with the Exadata standard 'compatible kernel' which is also in the ULN 11.2.3.2 channel. Recommended is to use the kernel that is installed by default. One of the other great new things 11.2.3.2 brings is Writeback Flashcache (wbfc). By default wbfc is disabled after the upgrade to 11.2.3.2. Enable wbfc after patching on the storage servers of your test environment and see the improvements this brings for your applications. Writeback FlashCache can be enabled  by dropping the existing FlashCache, stopping the cellsrv process and changing the FlashCacheMode attribute of the cell. Of course stopping cellsrv can only be done in a controlled manner. Steps: drop flashcache alter cell shutdown services cellsrv again, cellsrv can only be stopped in a controlled manner alter cell flashCacheMode = WriteBack alter cell startup services cellsrv create flashcache all Going back to WriteThrough FlashCache is also possible, but only after flushing the FlashCache: alter cell flashcache all flush Last item I like to highlight in particular is already from a while ago, but a great thing to emphasis: Oracle Platinum Services. On top of the remote fault monitoring with faster response times Oracle has included update and patch deployment services.These services are delivered by Oracle Advanced Customer Support at no additional costs for qualified Oracle Premier Support customers. References: 11.2.3.2.0 README Exadata YUM Repository Population, One-Time Setup Configuration and YUM upgrades  1473002.1 Oracle Platinum Services René Kundersma

Oracle recently announced an Exadata Hardware Update. The overall architecture will remain the same, however some interestinghardware refreshes are done especially for the storage server. Each cell...

New channels for Exadata 11.2.3.1.1

With the release of Exadata 11.2.3.1.0 back in April 2012 Oracle has deprecated the minimal pack for the Exadata Database Servers (compute nodes). From that release the Linux Database Server updates will be done using ULN and YUM. For the 11.2.3.1.0 release the ULN exadata_dbserver_11.2.3.1.0_x86_64_base channel was made available and Exadata operators could subscribe their system to it via linux.oracle.com. With the new 11.2.3.1.1 release two additional channels are added: a 'latest' channel (exadata_dbserver_11.2_x86_64_latest) a 'patch' channel (exadata_dbserver_11.2.3.1.0_x86_64_patch) The patch channel has the new or updated packages updated in 11.2.3.1.1 from the base channel. The latest channel has all the packages from 11.2.3.1.0 base and patch channels combined.  From here there are three possible situations a Database Server can be in before it can be updated to 11.2.3.1.1: Database Server is on Exadata release < 11.2.3.1.0 Database Server is patched to 11.2.3.1.0 Database Server is freshly imaged to 11.2.3.1.0 In order to bring a Database Server to 11.2.3.1.1 for all three cases the same approach for updating can be used (using YUM), but there are some minor differences: For Database Servers on a release < 11.2.3.1.0 the following high-level steps need to be performed: Subscribe to el5_x86_64_addons, ol5_x86_64_latest and  exadata_dbserver_11.2_x86_64_latest Create local repository Point Database Server to the local repository* install the update * during this process a one-time action needs to be done (details in the README) For Database Servers patched to 11.2.3.1.0: Subscribe to patch channel  exadata_dbserver_11.2.3.1.0_x86_64_patch Create local repository Point Database Server to the local repository Update the system For Database Servers freshly imaged to 11.2.3.1.0: Subscribe to patch channel  exadata_dbserver_11.2.3.1.0_x86_64_patch Create local  repository Point Database Server to the local repository Update the system The difference between 'situation 2' (Database Server is patched to 11.2.3.1.0) and 'situation 3' (Database Server is freshly imaged to 11.2.3.1.0) is that in situation 2 the existing Exadata-computenode.repo file needs to be edited while in situation 3 this file is not existing  and needs to be created or copied. Another difference is that you will end up with more OFA packages installed in situation 2. This is because none are removed during the updating process.  The YUM update functionality with the new channels is a great enhancements to the Database Server update procedure. As usual, the updates can be done in a rolling fashion so no database service downtime is required.  For detailed and up-to-date instructions always see the patch README's 1466459.1 patch 13998727 888828.1 Rene Kundersma

With the release of Exadata 11.2.3.1.0 back in April 2012 Oracle has deprecated the minimal pack for the Exadata Database Servers (compute nodes). From that release the Linux Database Server updates...

Updating Exadata Compute Nodes using ULN and YUM starting 11.2.3.1

With this post an update on Exadata Planned Maintenance - The new 11.2.3.1 ULN Updating Procedure for Exadata Compute Nodes.As you may already know, starting with Oracle Exadata Storage Server release 11.2.3.1 the 'minimal pack' is deprecated.From now on the (Linux) Computes Node will use Yellowdog Updater (Modified) (YUM) to apply the new updates as rpm packages.  These RPM packages that come from ULN may also contain firmware updates which will be applied in the same updating procedure. In order to update the Exadata Compute Nodes via YUM, you need a repository to download the software from. Of course Oracle provides its customers ULN access for this to make this really easy. It may however happen this requirement for http access to a web location from the Compute Nodes is not possible for some reason. For these situations Oracle worked out an alternative solution. In this post I like to discuss the solution to setup a YUM repository located in the customers Data Center: This is a local system, that can download the updates from ULN and act as a YUM repository for the Compute Nodes as a 'man in the middle' bertween ULN and the Compute Nodes. For installations planning to setup an internal/local YUM repository and currently not on 11.2.3.1: there are two notes that need to be reviewed carefully before the patching starts: README for patch 13741363 : Especially chapter 3 "Performing One-Time Setup on Database Servers of Oracle Exadata Database Machine" The steps described here are one time only steps to prepare and populate the local YUM repository server that will be used for ALL the Compute Nodes. Best is to install the YUM repository server on a separate Linux machine running OEL 4 or OEL 5. Basically the steps are: go to http://linux.oracle.com, use your Hardware CSI for the registration steps, register the YUM repository server and subscribe to the right channels and populate the repository. After the download is finished, the repository is 'build' and now 'local' it needs to be made available by http for the Compute Nodes to download from on the local network. After the setup of the repository V2/X2-2 and X2-8 Compute Nodes require a One-Time setup so they are able to use YUM from now on. The One-Time steps remove a set of packages that can cause the further setup to fail, but also add some packages to support the installations to be done using YUM onwards. Please pay close attention to one of the most important steps of this One-Time setup: the editing of the repository configuration file /etc/yum.repos.d/Exadata-computenode.repo and making sure it points to your local YUM repository if you don't have direct ULN access. README for 13536739: Especially chapter 6 "Updating Oracle Linux Database Servers in Oracle Exadata Database Machine"  After setting up the repository and enabling the Compute Nodes to use YUM there is one thing to do: the update itself. Key step in this process is enabling each Compute Node to use the new repository. After this some  packages (ofed1) may need to be downgraded depending on your installation and some checks for kernel versions need to be done. After this, from now on the the system can be updated using a simple 'yum install' to install the main Exadata database server rpm with all dependent packages.At the end of the installation the Compute Node is rebooted automatically. At this moment I have to make some remarks/disclaimers: Please see the notes for 13741363  and 13536739 for exact steps, I am only highlighting to explain the overall procedure of setting up a local repository and configuring the Compute Nodes using it. If you are able to connect your Compute Nodes to ULN directly there is no need to setup a repository and the related steps can be skipped. For X2-2 (and earlier) and X2-8 the 'updating Oracle Linux Database Server' steps are slightly different. Oracle has provided 'helper' scripts to automated the steps described above making it even more easy The YUM Updating procedure only applies to Linux Compute Nodes having images > 11.2.2.4.2. (for updates to versions lower than 11.2.3.1 you still need to use the minimal pack) Rene Kundersma

With this post an update on Exadata Planned Maintenance - The new 11.2.3.1 ULN Updating Procedure for Exadata Compute Nodes.As you may already know, starting with Oracle Exadata Storage Server release...

11.2.0.3 available for the Oracle Exadata Database Machine

This post is about upgrades. The first thing I like to mention is the availability of 11.2.0.3 for the Oracle Exadata Database Machine since today (Jan 3rd 2012). With this, it is now possible to upgrade your Oracle Exadata Database Machine to 11.2.0.3. To help you best with the upgrade we have released a MOS note that describes the prerequisites and steps to go from 11.2.0.1 or 11.2.0.2 installations to 11.2.0.3. Please see the MOS note for the best approach on how to apply this upgrade in your environment. The MOS note applies to both Solaris 11 Express and Linux installations and upgrades both the Grid Infrastructure and the Database Home on V2, X2-2 and X2-8. Please see MOS note "11.2.0.1/11.2.0.2 to 11.2.0.3 Database Upgrade on Exadata Database Machine" (Doc ID 1373255.1) for more details.For V1 users we have recently also published MOS note 888834.1. This document contains the steps to upgrade the HP Oracle Database Machine to Oracle Exadata Storage Server Software 11.2.X.  The steps include upgrading Oracle Cluster Ready Services (CRS) 11.1.0.7.X and Oracle Automatic Storage Management (ASM) 11.1.0.7.X to Oracle Grid Infrastucture 11.2.0.2 and Oracle RDBMS from 11.1.0.7 to Oracle RDBMS 11.2.0.2. After upgrading to 11.2.0 on V1 hardware using MOS note 888834.1,  MOS note 1373255.1 can be used to upgrade to 11.2.0.3. Please See MOS note "HP Oracle Database Machine 11.2 Upgrade" (Doc ID 888834.1) René Kundersma

This post is about upgrades. The first thing I like to mention is the availability of 11.2.0.3 for the Oracle Exadata Database Machine since today (Jan 3rd 2012). With this, it is now possible to...

Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata

With this post I like to mention the availability of the Enterprise Manager 12c setup automation kit for Exadata. I also like to explain how easy it is to deploy the agent for a complete Exadata rack of any size with all it's components. After the discovery the system can managed by EM, the deployment I did took only 30 minutes for a quarter rack with all components.You can obtain the "Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata" from MOS by searching for patch 12960596The deployment of the agent starts with the Exadata configurator sheet. This is the sheet ACS already used for the Exadata deployment. The sheet now also creates a configuration output file for automatic agent and OMS deployment. The file I am talking about is called em.params. This file can be used as input for the OMS or EM 12c deployment scripts. In this posting I will only discuss the agent deployment.This em.params file will have the following information: # This is em.param file# Written : 26-10-2011EM_VERSION=1.0OMS_LOCATION=REMOTE_CORP_GCEM_BASE=/u01/app/oracle/product/EMbaseOMS_HOST=my-host.us.oracle.comOMS_PORT=4900EM_CELLS=(cel12 cel13 cel14)EM_COMPUTE_ILOM_NAME=(db07-c db08-c)EM_COMPUTE_ILOM_IP=(a.b.c.d a.b.c.e)machinemodel="X2-2 Quarter rack"EM_USER=oracleEM_PASSWORD=passwdswikvmname=sw-kvmswikvmip=a.b.c.fswiipname=sw-ipswiipip=a.b.c.gswiib2name=sw-ib2swiib2ip=a.b.c.dswiib3name=sw-ibhswiib3ip=a.b.c.ipduaname=pdu1pduaip=a.b.c.jpdubname=pdu2pdubip=a.b.c.kOf course, the names and ipnumbers are changed in this example to hide any specific information.When you install the kit the configuration file is expected in /opt/oracle.SupportTools/onecommand, this is where your OneCommand deployment files will be anyway.During installation some additional checks will be done like:Trying to ping Oracle Management Server Host my-host.us.oracle.com .Checking oracle user infoSearching for KVM Switch information by key swikvmname from /opt/oracle.SupportTools/onecommand/em.paramSearching for KVM Switch IP information by key swikvmip from /opt/oracle.SupportTools/onecommand/em.paramSearching for Cisco Switch information by key swiipname from /opt/oracle.SupportTools/onecommand/em.paramSearching for Cisco Switch IP information by key swiipip from /opt/oracle.SupportTools/onecommand/em.paramSearching for IB2 Switch information by key swiib2name from /opt/oracle.SupportTools/onecommand/em.paramSearching for IB2 Switch IP information by key swiib2ip from /opt/oracle.SupportTools/onecommand/em.paramSearching for IB3 Switch information by key swiib3name from /opt/oracle.SupportTools/onecommand/em.paramSearching for IB3 Switch IP information by key swiib3ip from /opt/oracle.SupportTools/onecommand/em.paramSearching for PDUA Name information by key pduaname from /opt/oracle.SupportTools/onecommand/em.paramSearching for PDUA IP information by key pduaip from /opt/oracle.SupportTools/onecommand/em.paramSearching for PDUA Name information by key pdubname from /opt/oracle.SupportTools/onecommand/em.paramSearching for PDUB IP information by key pdubip from /opt/oracle.SupportTools/onecommand/em.paramSearching for ILON Names information by key EM_COMPUTE_ILOM_NAME from /opt/oracle.SupportTools/onecommand/em.paramSearching for ILOM IPS information by key EM_COMPUTE_ILOM_IP from /opt/oracle.SupportTools/onecommand/em.paramVerifying ssh setup..Verifying SSH setup for rootWhen the validations are done, the 12.1.0.1.0_AgentCore_226.zip file will be transferred to the compute nodes and the 12c agent will be installed.Like the 11.1 installation, still no agents will be installed on the storage cells, EM will use the compute nodes as proxy to the cells, it's up to you if you want EM to choose which nodes or choose yourself.After deployment, there are three easy steps left in the EM GUI: Exadata discovery GI(Cluster) discovery RAC discovery The discovery steps above are guided by a clear GUI.All the discovery process needs is access to a compute node, from there all the Exadata components will be discovered automatically.The file EM needs to be available for that is  /opt/oracle.SupportTools/onecommand/databasemachine.xmlAfter installation of the kit and discovery in EM 12c, you have a nice overview of the rack with all the components in it, easy to drill down and manage from there. The example below is a quarter rack. René Kundersma Oracle MAA Development

With this post I like to mention the availability of the Enterprise Manager 12c setup automation kit for Exadata. I also like to explain how easy it is to deploy the agent for a complete Exadata rack...

Software Updates, Best Practices and Notes

With a very busy time behind and ahead of me, I still like to make mention of a couple of documents recently we published. OPlan is now available for Linux as well as Solaris X86-64 for the Oracle Exadata Database Machine. The current version of OPlan is 11.2.0.2.5. For more details, see my previous post The two notes mentioned below explain about how OPlan can be used and demo how OPLan works to clone the Grid Infrastructure on the Database Machine, patch the cloned home and switch to it. This is called out-of-place patching. MOS Note 1306814.1 - Oracle Software Patching with OPLAN MOS Note 1136544.1 - Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories Supported software versions for Exadata are still listed in MOS note 888828.1, this is where you will find a reference to the latest Exadata Storage Server software release: 11.2.2.3.5, available via patch 12849110. Please see note 1334254.1 for the details. For the Database Server, the latest patch made available for 11.2.0.2 is Bundle Patch 10, available via Patch 12773458 (Linux version). Please know that BP's can be applied by EM, which makes patching more easy.For ASR we now have release version 3.3 Released. Details can be found in Note 1185493.1.The latest PDU metering unit firmware is 1.04 and available as patch 12871297.For MAA Best pracitces releated to the database machine we released a real good document called "Oracle Exadata Database Machine Consolidation: Segregating Databases and Roles", which you can find here Also releated to best practices is the document "MAA Best Practices for Oracle Exadata Database Machine", you can find that hereRene Kundersma

With a very busy time behind and ahead of me, I still like to make mention of a couple of documents recently we published. OPlan is now available for Linux as well as Solaris X86-64 for the Oracle...

Failover capability for plugins Exadata & EMGC Rapid deployment

With this entry two items around Exadata management:First: a pointer to My Oracle Support note 1110675.1. This is note is called "Oracle Database Machine Monitoring Best Practices" and has a lot of information on Exadata management. In this note you will read about the plugins we have to monitor the various Exadata components and how to configure them, also you will read about creating a 'Dashboard'.The topic I like to cover here is about the steps to take to setup automated fail-over of targets. This is, because by default a target monitored using a plug-in in Enterprise Manager Grid Control is bound to the agent used during the initial installation.So, what you basically want to do is make sure another agent (on another node) is available to take over the monitored targets when the first agents fails; providing HA for your monitoring.For this I like to mention two items: "Package to add failover capabiity for Plug-ins"   (bottom of note 1110675.1.) OEM_Exadata_Dashboard_Deployment_v104.pdf In the file 'OEM_Exadata_Dashboard_Deployment' you read about the following steps: adding the PL/SQL package into the EM repository (package mentioned above) registering the package setting up of agents the target can use for fail-over create failover method and notification rule create fail-back method and notification rule Once you have executed the steps above, the second agent should be able to take over the targets initially assigned to the first agent when the first fails. You can test this by stopping the first agent; in Grid Control you will find the targets under the second agent. Making sure the monitored targets can failover to a surviving agent is recommended, but it going that route, it would make sense to also make sure Grid Control (OMS) and database are both also HA.Second topic for this entry is the "EM 11GC Setup automatic kit" for Exadata. With the latest Oracle Exadata Database Machine Configurator a new output file (em.param) is generated. This file can be used as input for installing Oracle Enterprise Manager Grid Control Agents on Oracle Exadata Database Machine With this kit, Exadata customers, having an existing Oracle Enterprise Manager Grid Control environment can have setup agents rapidly by the Oracle or partner teams doing the Exadata deployment. For the sake of completeness: Oracle Enterprise Manager Grid Control is not a requirement for Exadata but a recommendation.References: MOS 1110675.1 Demo: here Updates: Oracle Enterprise Manager Grid Control Setup Automation kit for Exadata Kit Rene Kundersma

With this entry two items around Exadata management: First: a pointer to My Oracle Support note 1110675.1. This is note is called "Oracle Database Machine Monitoring Best Practices" and has a lot of...

Applying bundle patches on Exadata using Enterprise Manager Grid Control

With this post a small first note on applying Exadata Bundle Patches using Enterprise Manager Grid Control 11g.Did you know: Exadata BP's can be applied in a rolling fashion using Grid Control comparable to 'opatch auto' The patching procedure always updates Opatch to the latest version automatically To some extend conflicts are checked to prevent you from problems using patching Before applying a patch you could run 'analyze' only and verify if the BP patch can be applied. When a SQL script has to run as part of the BP, that's also taken care off Interested ? Here some resources to help you forward: Driving Database Patch Management with EM 11g Configuration management and provisioning of Sun oracle Exadata Database Machine Using Enterprise manager More important, check first for required patches and what BP's are tested: Grid Control OMS and Agent Patches required for setting up Provisioning, Patching and Cloning (Doc ID 427577.1) Patch Oracle Exadata Database Machine via Oracle Enterprise Manager 11gR1 (11.1.0.1) (Doc ID 1265998.1) Patch Requirements for Setting up Monitoring and Administration for Exadata (Doc ID 1323298.1) Still, there's only one single source of truth when it comes to which patches are recommended for Exadata: Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions (ID 888828.1) A small demo of a comparable 'RAC Rolling patch' can be found on OTN: Rene Kundersma

With this post a small first note on applying Exadata Bundle Patches using Enterprise Manager Grid Control 11g. Did you know:Exadata BP's can be applied in a rolling fashion using Grid Control...

Patching for Exadata: introducing oplan

With this update I like to introduce you to Oplan. Oplan is a utility that facilitates you with the application of bundle patches on Exadata compute nodes via Opatch. This new utility helps you with the patch process by generating step-by-step instructions telling you how to apply a bundle patch in your environment.Oplan is supported from release 11.2.0.2 and available as download here. The steps to install Oplan are straight forward:download Oplan to one of your oracle home's (for example the GI home) as the oracle home owner. Say /u01/app/11.2.0/grid/oplanappend the Oplan directory to your pathas the oracle home owner generate the Oplan installation instructions, for example:oplan generateApplySteps /home/oracle/11828582 (where 11828582 is BP5)Oplan will now generate the specific installation instructions for your environment.You will notice that Oplan provides you with several options how to apply the patch.In my case Oplan offered :in-place instructions using auto patch or notin-place instructions rolling or non-rollingout-of-place patching instructions.For each 'strategy' Oplan lists the advantages and disadvantages. For example an out-of-place instruction may cost you more time, but is easier to rollback. Also the number of steps for each option is listed.Once you made up your mind and made your choice for a specific patch strategy, Oplan will give you step-by-step instructions.Each patch plan generated by Oplan begins with a pre-patch apply phase, then a patch apply phase and ends with a post-patch apply phase.In the pre-patch apply phase the current situation of the oracle homes to be patched will be verified. Some checks that are done:The version of opatch will be verifiedCheck to see if there are no conflicts pending. Next to this, ocm will be configured In the patch apply process the actual patch is applied. This is followed by the post-patch apply phase where srvctl is used to reconfigure the new Oracle home to be used (in my case) and also modify /etc/oratab.For all of this Oplan generates instructions for all of the nodes in the cluster, even how to copy over the patch itself.Last but no least some small notes:Oplan can create rollback instructions for the patch.There is no support for Oracle DataGuardPatch README files should be used to double check the proposed actions, also in case of any conflict: the patch README should be considered as truthAltogether Oplan comes in very handy by summarizing the different patch strategies available. When you made your choice, Oplan tells you exactly what commands to execute. This will limit errors and reduce the time it takes to prepare.Please see note 1306814.1 and check it out yourself.Rene

With this update I like to introduce you to Oplan. Oplan is a utility that facilitates you with the application of bundle patches on Exadata compute nodes via Opatch. This new utility helps you with...

ASR / SNMP on Exadata

Recently I worked with ASR on Exadata for multiple customers. ASR is a great functionality that enables your 'systems' to alert Oracle when hardware failures occur. Sun hardware is using ASR for sometime and since 2009/2010 this is also available for Exadata. My goal is not to re-write the documentation so for general information I like to refer to this link.So, where is this note about ? Well, it is about two things I experienced around setting up ASR. I like to provide my experience so others can be successful with ASR fast as well. (It is however expected that things will be updated in the latest documentation.)First, imagine yourself configuring SNMP traps to be sent to ASR. In this situation be sure to not erase any existing SNMP Subscribers settings for example the subscription to Enterprise Manager Grid Control or whatever you already subscribed for. So, when you have documentation stating to execute "cellcli -e alter cell snmpSubscriber=(host=, port=)" be sure to add existing snmpSubscribers when they exist.The syntax allows this:snmpSubscriber=((host=host [,port=port] [,community=community][,type=ASR])[,(host=host[,port=port][,community=community][,type=ASR])...)Second, when configuring SnmpSubscribers using DCLI you have to work with a slash to escape the brackets. Be sure to verify your SNMP settings after setting them because you might end up with a bracket in the 'asrs.state' file stating 'public\' in stead of 'public'.Having the extra slash after the word 'public' of course doesn't help when sending SNMP-traps:dcli -g dbs_group -l root -n "/opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -validate_snmp_subscriber -type asr"cn38: Sending test trap to destination - 173.25.100.43:162cn38: (1). count - 50 Failed to run "/usr/bin/snmptrap -v 2c -c public\ -M "+/opt/oracle.cellos/compmon/" -m SUN-HW-TRAP-MIB 173.25.100.43:162 "" SUN-HW-TRAP-MIB::sunHwTrapTestTrap sunHwTrapSystemIdentifier s " Sun Oracle Database Machine secret" sunHwTrapChassisId s "secret" sunHwTrapProductName s "SUN FIRE X4170 SERVER" sunHwTrapTestMessage s "This is a test trap. Exadata Compute Server: cn38.oracle.com ""cn38: getaddrinfo: +/opt/oracle.cellos/compmon/ Name or service not knowncn38: snmptrap: Unknown host (+/opt/oracle.cellos/compmon/)All together ASR is a great addition to Exadata that I highly recommend. Some excellent documentation is written on the implementation details and available on MyOracleSupport. See "Oracle Database Machine Monitoring (Doc ID 1110675.1)"Rene KundersmaTechnical ArchitectOracle Technology Services

Recently I worked with ASR on Exadata for multiple customers. ASR is a great functionality that enables your 'systems' to alert Oracle when hardware failures occur. Sun hardware is using ASR...

11gr2 DataGuard: Restarting DUPLICATE After a Failure

One of the great new features that comes in very handy when databases get larger and larger these days is RMAN's capability to duplicate from an active database and even restart a duplicate when it fails.Imagine yourself the problem I had lately; I used the duplicate from active database feature and had to wait for an hour or 6 before all datafiles where transferred.At the end of the process some error occurred because of the syntax. While this error was easily to solve I was afraid I had to redo the complete procedure and transfer the 2.5 TB again.Well, 11gr2 RMAN surprised me when I re-ran my command with the following output:Using previous duplicated file +DATA/fin2prod/datafile/users.2968.719237649 for datafile 12 with checkpoint SCN of 183289288148Using previous duplicated file +DATA/fin2prod/datafile/users.2703.719237975 for datafile 13 with checkpoint SCN of 183289295823Above I only show a small snippet, but what happend is that RMAN smartly skipped all files that where already transferred !The documentation says this:RMAN automatically optimizes a DUPLICATE command that is a repeat of a previously failed DUPLICATE command. The repeat DUPLICATE command notices which datafiles were successfully copied earlier and does not copy them again. This applies to all forms of duplication, whether they are backup-based (with and without a target connection) or active database duplication. The automatic optimization of the DUPLICATE command can be especially useful when a failure occurs during the duplication of very large databases.If a DUPLICATE operation fails, you need only run the DUPLICATE again, using the same parameters contained in the original DUPLICATE command. Please see chapter 23 of the 11g Release 2 Database Backup and Recovery User's Guide for more details.B.w.t. be very careful with the duplicate command. A small mistake in one of the 'convert' parameters can potentially overwrite your target's controlfile without prompting !Rene KundersmaTechnical ArchitectOracle Technology Services

One of the great new features that comes in very handy when databases get larger and larger these days is RMAN's capability to duplicate from an active database and even restart a duplicate when it...

Installation and testing RAC One Node

Okay time to write something nice about Rac One Node:In order to test RAC One Node, on my Laptop, I just:- installed Oracle VM 2.2- Created two OEL 5.3 imagesThe two images are fully prepared for Oracle 11gr2 Grid Infrastructure and 11gr2 RAC including four shared disks for ASM and private nics.How would I test this all so fast without virtualization ?!In order to view the captures best, I recommend to zoom in a couple of times (control-++)After installation of the Oracle 11gr2 Grid Infrastructure and a "software only installation" of 11gr2 RAC, I installed patch 9004119 as you can see with the opatch lsinv output:This patch has the scripts required to administer RAC One Node, you will see them later.At the moment we have them available for Linux and Solaris.After installation of the patch, I created a RAC database with an instance on one node.Please note that the "Global Database Name" has to be the same as the SID prefix and should be less then or equal to 8 characters:When the database creation is done, first I create a service. This is because RAC One Node needs to be "initialized" each time you add a service:The service configuration details are:After creating the service, a script called raconeinit needs to run from $RDBMS_HOME/bin. This is a script supplied by the patch. I can imagine the next major patch set of 11gr2 has this scripts available by default. The script will configure the database to run on other nodes:After initialization, when you would run raconeinit again, you would see:So, now the configuration is ready and we are ready to run 'Omotion' and move the service around from one node to the other (yes, vm competitor: this is service is available during the migration, nice right ?) .Omotion is started by running Omotion. With Omotion -v you get verbose output:So, during the migration with Omotion you will see the two instance active (RKDB_1 and RKDB_2):And, after the migration, there is only one instance left on the new node:Of course, regular node failures will also initiate a failover, all covered by the default Grid Infrastructure functionalities. The thing that I noticed is that if you would kill a node that runs an active instance, the instance is failed over nicely by RAC Node One, but the name of the instance failing over stays the same, so this is other behavior then the migration:p.s. 1:Some other funny thing I noticed is that after installing 11gr2 Grid Infrastructure, Oracle removes some essential lines from grub.conf. What you get when you try to start a vm after this is: "error: boot loader didn't return any data"Luckily Oracle creates a backup of the grub.conf in /boot/grub named grub.conf.orabackup So, you need to restore that file in the vm images itself.This can be done with the great lomount option.First make sure you create the entry in the fstab of your oracle vm server:/var/ovs/mount/8BF866167A1746FE8FFA0EAA20939C55/running_pool/GRIDNODE01/System.img /tmp/q ext3 defaults 1 1Then execute: lomount -diskimage ./System.img -partition 1This mounts the image to /tmp/q so you can restore the file.p.s. 2:I would like to demo some, hopefully I can do this at the next Planboard Symposium June 8. See this linkRene KundersmaOracle Technology Services, The Netherlands

Okay time to write something nice about Rac One Node: In order to test RAC One Node, on my Laptop, I just: - installed Oracle VM 2.2 - Created two OEL 5.3 images The two images are fully prepared for...

Using DNFS for test purposes

Because of other priorities such as bringing the first v2 Database Machine in the Netherlands into production I did spend less time on my blog that planned. I do however like to tell some things about DNFS, the build-in NFS client we have in Oracle RDBMS since 11.1.What DNFS is and how to set it up can all be found here .As you see this documentation is actually the "Clusterware Installation Guide". I think that is weird, I would expect this to be part of the Admin Guide, especially the "Tablespace" chapter.I do however want to show what I did not find in the documentation that quickly (and solved after talking to my famous colleague "the prutser"):First, a quick setup:1. The standard ODM library needs to be replaced with the NFS ODM library:[oracle@rkdb01 ~]$ cp $ORACLE_HOME/lib/libodm11.so $ORACLE_HOME/lib/libodm11.so_stub[oracle@rkdb01 ~]$ ln -s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.soAfter changing to this library you will notice the following in your alert.log:Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.02. The intention is to mount the datafiles over normal NAS (like NetApp). But, in case you want to test yourself and use an exported NFS filesystem, it should look like the following: [oracle@rkdb01 ~]$ cat /etc/exports /u01/scratch/nfs *(rw,sync,insecure)Please note the "insecure" option in the export, since you will not be able to use DNFS without it if you export a filesystem from a host. Without the "insecure" option the NFS server considers the port used by the database "insecure" and the database is unable to acquire the mount. This is what you will see in the alert.log creating the file: Direct NFS: NFS3ERR 1 Not owner. path rkdb01.nl.oracle.com mntport 930 nfsport 20493. Before configuring the new Oracle stanza for DNFS we still need to configure a regular kernel NFS mount:[root@rkdb01 ~]# cat /etc/fstab | grep nfsrkdb01.nl.oracle.com:/u01/scratch/nfs /incoming nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=6004. Then a so called Oracle-'nfstab' needs to be created that specifies what the available exports to use:[oracle@rkdb01 ~]$ cat /etc/oranfstab server:rkdb01.nl.oracle.compath:192.168.1.40export:/u01/scratch/nfs mount:/incoming5. Creating a tablespace with a datafile on the NFS location:SQL> create tablespace rk datafile '/incoming/rk.dbf' size 10M;Tablespace created.As said, be sure to know that it may happen that if you do not specify the insecure option (like I did) you will still see output from the query v$dnfs_servers:SQL> select * from v$dnfs_servers;ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX-- -------------------- ----------------- --------- ---------- ------ ------1 rkdb01.nl.oracle.com /u01/scratch/nfs 684 2049 32768 32768But, querying v$dnfsfiles and v$dnfs_channels will not return any result, and indeed, you will see the following message in the alert-log whenyou create a file : Direct NFS: NFS3ERR 1 Not owner. path rkdb01.nl.oracle.com mntport 930 nfsport 2049After correcting the export with the "secure" option:SQL> select * from v$dnfs_files;FILENAME FILESIZE PNUM SVR_ID--------------- -------- ------ ------ /incoming/rk.dbf 10493952 20 1 Rene KundersmaOracle Technology Services, The Netherlands

Because of other priorities such as bringing the first v2 Database Machine in the Netherlands into production I did spend less time on my blog that planned. I do however like to tell some things...

Adding iSCSI storage without restarting the iSCSI service

By one colleague I was asked how to add iSCSI storage without stopping the iSCSI service itself. Below how this works. I used tgt as iSCSI service. Create a iSCSI target: (this is what you do on the source):[root@gridnode01 ~]# tgtadm --lld iscsi --op new --mode target --tid 1 -T 192.168.1.172:vol1[root@gridnode01 ~]# dd if=/dev/zero of=/tmp/vol1.dd bs=1M count=1010+0 records in10+0 records out10485760 bytes (10 MB) copied, 0.0285 seconds, 368 MB/s[root@gridnode01 ~]# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /tmp/vol1.dd [root@gridnode01 ~]# tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALLThen, install iSCSI-initiator utils on the target node and configure the service:[root@gridnode02 ~]# rpm -i iscsi-initiator-utils-6.2.0.868-0.7.el5.i386.rpm [root@gridnode02 ~]# chkconfig iscsi on[root@gridnode02 ~]# service iscsi startiscsid is stoppedTurning off network shutdown. Starting iSCSI daemon: [ OK ][ OK ]Setting up iSCSI targets: iscsiadm: No records found![ OK ]Step 3: Discover the target just created and login to the portal"[root@gridnode02 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.1.171192.168.1.171:3260,1 192.168.1.172:vol1[root@gridnode02 ~]# iscsiadm --mode node --targetname 192.168.1.172:vol1 --portal 192.168.1.171:3260 --loginLogging in to [iface: default, target: 192.168.1.172:vol1, portal: 192.168.1.171,3260]Login to [iface: default, target: 192.168.1.172:vol1, portal: 192.168.1.171,3260]: successfulNow, let's see the new scsi disk coming in:[root@gridnode02 ~]# tail -f /var/log/messagesJan 10 21:01:19 gridnode02 kernel: sda: Write Protect is offJan 10 21:01:19 gridnode02 kernel: SCSI device sda: drive cache: write backJan 10 21:01:19 gridnode02 kernel: SCSI device sda: 20480 512-byte hdwr sectors (10 MB)Jan 10 21:01:19 gridnode02 kernel: sda: Write Protect is offJan 10 21:01:19 gridnode02 kernel: SCSI device sda: drive cache: write backJan 10 21:01:19 gridnode02 kernel: sda: unknown partition tableJan 10 21:01:19 gridnode02 kernel: sd 0:0:0:1: Attached scsi disk sdaJan 10 21:01:19 gridnode02 iscsid: received iferror -38Jan 10 21:01:19 gridnode02 last message repeated 2 timesJan 10 21:01:19 gridnode02 iscsid: connection1:0 is operational now[root@gridnode02 ~]# fdisk -l /dev/sdaDisk /dev/sda: 10 MB, 10485760 bytes1 heads, 20 sectors/track, 1024 cylindersUnits = cylinders of 20 * 512 = 10240 bytesDisk /dev/sda doesn't contain a valid partition tableNow, add another volume on the source:[root@gridnode01 ~]# tgtadm --lld iscsi --op new --mode target --tid 2 -T 192.168.1.172:vol2[root@gridnode01 ~]# dd if=/dev/zero of=/tmp/vol2.dd bs=1M count=10[root@gridnode01 ~]# tgtadm --lld iscsi --op new --mode logicalunit --tid 2 --lun 1 -b /tmp/vol2.dd [root@gridnode01 ~]# tgtadm --lld iscsi --op bind --mode target --tid 2 -I ALLOn target, re-run the discovery:[root@gridnode02 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.1.171192.168.1.171:3260,1 192.168.1.172:vol1192.168.1.171:3260,1 192.168.1.172:vol2And login again on the new target:[root@gridnode02 ~]# iscsiadm --mode node --targetname 192.168.1.172:vol2 --portal 192.168.1.171:3260 --loginLogging in to [iface: default, target: 192.168.1.172:vol2, portal: 192.168.1.171,3260]Login to [iface: default, target: 192.168.1.172:vol2, portal: 192.168.1.171,3260]: successfulLet's check the new disk:[root@gridnode02 ~]# tail /var/log/messagesJan 10 21:04:33 gridnode02 kernel: SCSI device sdb: drive cache: write backJan 10 21:04:33 gridnode02 kernel: SCSI device sdb: 20480 512-byte hdwr sectors (10 MB)Jan 10 21:04:33 gridnode02 kernel: sdb: Write Protect is offJan 10 21:04:33 gridnode02 kernel: SCSI device sdb: drive cache: write backJan 10 21:04:33 gridnode02 kernel: sdb: unknown partition tableJan 10 21:04:33 gridnode02 kernel: sd 1:0:0:1: Attached scsi disk sdbJan 10 21:04:33 gridnode02 kernel: sd 1:0:0:1: Attached scsi generic sg3 type 0Jan 10 21:04:33 gridnode02 iscsid: received iferror -38Jan 10 21:04:33 gridnode02 last message repeated 2 timesJan 10 21:04:33 gridnode02 iscsid: connection2:0 is operational now[root@gridnode02 ~]# fdisk -l /dev/sdbDisk /dev/sdb: 10 MB, 10485760 bytes1 heads, 20 sectors/track, 1024 cylindersUnits = cylinders of 20 * 512 = 10240 bytesDisk /dev/sdb doesn't contain a valid partition tableRene KundersmaOracle Technology Services, The Netherlands

By one colleague I was asked how to add iSCSI storage without stopping the iSCSI service itself. Below how this works. I used tgt as iSCSI service. Create a iSCSI target: (this is what you do on the...

Relocating Grid Infrastructure (part 2)

In my previous post, I moved the 11gr2 Grid infrastructure (gi) home to another location. Unfortunately, as I showed, during my actions the re-run of root.sh caused the diskgroup holding the clusterware-files (vote / ocr) to be recreated.Recreating this diskgroup would mean loosing my database data in it.My colleagues from the development organization helped me out and told me how I could solve this. Below you can find my notes. Please note: I am NOT telling is this a supported action. If you perform this actions, you are on your own. I can only recommend you test things first.The current location for the gi is /u01/app/11.2.0/grid:[root@rac1 ~]# cat /etc/oratab | grep app+ASM1:/u01/app/11.2.0/grid:N # line added by AgentORCL:/u01/app/oracle/product/11.2.0/dbhome_1:N # line added by AgentSo, first let's stop all of the clusterware stack (on both nodes)[root@rac1 ~]# . oraenvORACLE_SID = [root] ? +ASM1The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle[root@rac1 ~]# crsctl stop crsCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'CRS-2673: Attempting to stop 'ora.orcl.db' on 'rac1'CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeededCRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeededCRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeededCRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeededCRS-2677: Stop of 'ora.orcl.db' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.asm' on 'rac1'CRS-2677: Stop of 'ora.asm' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.ons' on 'rac1'CRS-2673: Attempting to stop 'ora.eons' on 'rac1'CRS-2677: Stop of 'ora.ons' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeededCRS-2677: Stop of 'ora.eons' on 'rac1' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completedCRS-2677: Stop of 'ora.crsd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'CRS-2673: Attempting to stop 'ora.asm' on 'rac1'CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeededCRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeededCRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeededCRS-2677: Stop of 'ora.ctssd' on 'rac1' succeededCRS-2677: Stop of 'ora.evmd' on 'rac1' succeededCRS-2677: Stop of 'ora.asm' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'rac1'CRS-2677: Stop of 'ora.cssd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeededCRS-2677: Stop of 'ora.diskmon' on 'rac1' succeededCRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completedCRS-4133: Oracle High Availability Services has been stopped.Now, create a new directory for the gi, and move gi home into that:[root@rac1 ~]# mkdir /u01/rk[root@rac1 ~]# mv /u01/app/11.2.0/grid /u01/rk[root@rac1 ~]# . oraenvORACLE_SID = [+ASM1] ? The Oracle base for ORACLE_HOME=/u01/rk/grid is /u01/app/oracleNow, edit the file crsconfig_params, so that it reflects the new gi home path:vi $ORACLE_HOME/crs/install/crsconfig_params => and change ORACLE_HOME=/u01/rk/gridIf you would run $OH/crs/install/rootcrs.pl -patch -hahome /u01/rk/grid now, you are in trouble, like I was, this is what my alert file said:[ohasd(26079)]CRS-1339:Oracle High Availability Service aborted due to an unexpected error [Failed to initialize Oracle Local Registry]. Details at (:OHAS00106:) in /u01/rk/grid/log/rac1/ohasd/ohasd.log.So, what was in my ohasd.log:2010-01-05 14:48:04.516: [ OCROSD][3046704848]utopen:6m':failed in stat OCR file/disk /u01/app/11.2.0/grid/cdata/rac1.olr, errno=2, os err string=No such file or directory2010-01-05 14:48:04.516: [ OCROSD][3046704848]utopen:7:failed to open any OCR file/disk, errno=2, os err string=No such file or directory2010-01-05 14:48:04.516: [ OCRRAW][3046704848]proprinit: Could not open raw deviceAs, you can see, I forgot to change the location of my OLR, so let's do it:vi /etc/oracle/olr.loc and change:olrconfig_loc=/u01/rk/grid/cdata/rac1.olrcrs_home=/u01/rk/gridAfter this, I ran the command again and succeeded:[root@rac1 grid]# $OH/crs/install/rootcrs.pl -patch -hahome /u01/rk/grid2010-01-05 14:58:28: Parsing the host name2010-01-05 14:58:28: Checking for super user privileges2010-01-05 14:58:28: User has super user privilegesUsing configuration parameter file: crs/install/crsconfig_paramsCRS-4123: Oracle High Availability Services has been started.See the new status:[root@rac1 grid]# crsctl status resource -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.eons ONLINE OFFLINE rac1 ONLINE OFFLINE rac2 ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE OFFLINE rac1 ONLINE OFFLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 --------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac1 ora.oc4j 1 OFFLINE OFFLINE ora.orcl.db 1 ONLINE ONLINE rac1 Open 2 ONLINE OFFLINE ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac1 Rene KundersmaOracle Technology Services, The Netherlands

In my previous post, I moved the 11gr2 Grid infrastructure (gi) home to another location. Unfortunately, as I showed, during my actions the re-run of root.sh caused the diskgroup holding...

Relocating Grid Infrastructure

Below I will describe the actions one needs to perform when the Oracle 11gr2 Grid Infrastructure ORACLE HOME needs to be moved to a new location.Please note:- You will loose already registered resources (like databases) from OCR (they need to be added back again).- Default LISTENER needs to be re-configured (re-run netca from Grid Inf. home)- You will have downtime during the action- Your ASM diskgroup that holds your cluster disks will be recreated !So, please again, note the ASM diskgroup will be recreated. In case your data is there, you will loose it.These are my steps:1. On all nodes, but the last run the delete force command. This will stop all cluster resources and deconfigure the Oracle clusterware stack on the node.[root@rac1 ]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -delete -forceOutput:2009-12-29 14:52:16: Parsing the host name2009-12-29 14:52:16: Checking for super user privileges2009-12-29 14:52:16: User has super user privilegesUsing configuration parameter file: ./crsconfig_paramsPRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1PRCR-1068 : Failed to query resourcesCannot communicate with crsdPRCR-1070 : Failed to check if resource ora.gsd is registeredCannot communicate with crsdPRCR-1070 : Failed to check if resource ora.ons is registeredCannot communicate with crsdPRCR-1070 : Failed to check if resource ora.eons is registeredCannot communicate with crsdACFS-9200: SupportedCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4000: Command Stop failed, or completed with errors.CRS-4544: Unable to connect to OHASCRS-4000: Command Stop failed, or completed with errors.Successfully deconfigured Oracle clusterware stack on this node2. On the last node in run command again with the "-lastnode" option. [root@rac2 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -delete -force -lastnode3. On both nodes create a new directory and move the ORACLE HOME to there: [root@rac1-2 ~]# mkdir /u01/rk; mv /u01/app/11.2.0/grid /u01/rk4. On both nodes create a new symlink for JRE; [root@rac1-2 ~]# cd /u01/rk/grid; rm -f JRE; ln -s /u01/rk/grid/jdk/jre/ JRE5. As oracle run clone.pl, this will change all the hard coding: export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/rk/grid cd /u01/rk/grid/clone/bin; run then: perl clone.pl ORACLE_HOME=$ORACLE_HOME output like this:[oracle@rac1 bin]$ perl clone.pl ORACLE_HOME=/u01/rk/grid ORACLE_BASE=/u01/app/oracle./runInstaller -clone -waitForCompletion "ORACLE_HOME=/u01/rk/grid" "ORACLE_BASE=/u01/app/oracle" -defaultHomeName -silent -noConfig -nowait Starting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 1955 MB PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2009-12-29_03-09-42PM. Please wait ...Oracle Universal Installer, Version 11.2.0.1.0 ProductionCopyright (C) 1999, 2009, Oracle. All rights reserved.You can find the log of this install session at: /u01/app/oraInventory/logs/cloneActions2009-12-29_03-09-42PM.log.................................................................................................... 100% Done.Installation in progress (Tuesday, December 29, 2009 3:10:15 PM EST)......................................................................... 73% Done.Install successfulLinking in progress (Tuesday, December 29, 2009 3:10:19 PM EST)Link successfulSetup in progress (Tuesday, December 29, 2009 3:11:39 PM EST)................. 100% Done.Setup successfulEnd of install phases.(Tuesday, December 29, 2009 3:13:38 PM EST)WARNING:The following configuration scripts need to be executed as the "root" user./u01/rk/grid/root.shTo execute the configuration scripts: 1. Open a terminal window 2. Log in as "root" 3. Run the scriptsRun the script on the local node first. After successful completion, you can run the script in parallel on all the other nodes.The cloning of OraHome1 was successful.Please check '/u01/app/oraInventory/logs/cloneActions2009-12-29_03-09-42PM.log' for more details.6. Verify your crsconfig_params (/u01/rk/grid/crs/install/crsconfig_params) and make sure this file is available on both nodes.7. Relink the grid infra home on both nodes As root:# cd /u01/rk/grid/crs/install# perl rootcrs.pl -unlockAs the grid infrastructure for a cluster owner:$ export ORACLE_HOME=cd /u01/rk/grid$ cd /u01/rk/grid/bin/relink8. On each node run root.sh, begin with node1: cd /u01/rk/grid; ./root.sh Check /u01/rk/grid/install/root_rac1_2009-12-29_15-37-11.log for the output of root script run rootcrs as requested on each node:/u01/rk/grid/perl/bin/perl -I/u01/rk/grid/perl/lib -I/u01/rk/grid/crs/install /u01/rk/grid/crs/install/rootcrs.plOutput node1:[root@rac1 grid]# /u01/rk/grid/perl/bin/perl -I/u01/rk/grid/perl/lib -I/u01/rk/grid/crs/install /u01/rk/grid/crs/install/rootcrs.pl2009-12-29 16:05:51: Parsing the host name2009-12-29 16:05:51: Checking for super user privileges2009-12-29 16:05:51: User has super user privilegesUsing configuration parameter file: /u01/rk/grid/crs/install/crsconfig_paramsLOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'..Operation successful.Adding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingCRS-2672: Attempting to start 'ora.gipcd' on 'rac1'CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'CRS-2676: Start of 'ora.gipcd' on 'rac1' succeededCRS-2676: Start of 'ora.mdnsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'rac1'CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'CRS-2676: Start of 'ora.diskmon' on 'rac1' succeededCRS-2676: Start of 'ora.cssd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'rac1'CRS-2676: Start of 'ora.ctssd' on 'rac1' succeededASM created and started successfully.DiskGroup DATA created successfully.clscfg: -install mode specifiedSuccessfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.CRS-2672: Attempting to start 'ora.crsd' on 'rac1'CRS-2676: Start of 'ora.crsd' on 'rac1' succeededSuccessful addition of voting disk 034bbf3dcd1f4f9fbf1afa38db67caad.Successfully replaced voting disk group with +DATA.CRS-4266: Voting file(s) successfully replaced## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 034bbf3dcd1f4f9fbf1afa38db67caad (/dev/sdb1) [DATA]Located 1 voting disk(s).CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'CRS-2677: Stop of 'ora.crsd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.asm' on 'rac1'CRS-2677: Stop of 'ora.asm' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'rac1'CRS-2677: Stop of 'ora.cssd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.gipcd' on 'rac1'CRS-2676: Start of 'ora.gipcd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'rac1'CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'CRS-2676: Start of 'ora.diskmon' on 'rac1' succeededCRS-2676: Start of 'ora.cssd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'rac1'CRS-2676: Start of 'ora.ctssd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.asm' on 'rac1'CRS-2676: Start of 'ora.asm' on 'rac1' succeededCRS-2672: Attempting to start 'ora.crsd' on 'rac1'CRS-2676: Start of 'ora.crsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.evmd' on 'rac1'CRS-2676: Start of 'ora.evmd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.asm' on 'rac1'CRS-2676: Start of 'ora.asm' on 'rac1' succeededCRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeededCRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeededrac1 2009/12/29 16:11:02 /u01/rk/grid/cdata/rac1/backup_20091229_161102.olrPreparing packages for installation...cvuqdisk-1.0.7-1Configure Oracle Grid Infrastructure for a Cluster ... succeeded Output node 2:[root@rac2 grid]# /u01/rk/grid/perl/bin/perl -I/u01/rk/grid/perl/lib -I/u01/rk/grid/crs/install /u01/rk/grid/crs/install/rootcrs.pl2009-12-29 16:13:33: Parsing the host name2009-12-29 16:13:33: Checking for super user privileges2009-12-29 16:13:33: User has super user privilegesUsing configuration parameter file: /u01/rk/grid/crs/install/crsconfig_paramsLOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'..Operation successful.Adding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingCRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminatingAn active cluster was found during exclusive startup, restarting to join the clusterCRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.gipcd' on 'rac2'CRS-2676: Start of 'ora.gipcd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeededCRS-2672: Attempting to start 'ora.cssd' on 'rac2'CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'CRS-2676: Start of 'ora.diskmon' on 'rac2' succeededCRS-2676: Start of 'ora.cssd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'rac2'CRS-2676: Start of 'ora.ctssd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeededCRS-2672: Attempting to start 'ora.asm' on 'rac2'CRS-2676: Start of 'ora.asm' on 'rac2' succeededCRS-2672: Attempting to start 'ora.crsd' on 'rac2'CRS-2676: Start of 'ora.crsd' on 'rac2' succeededCRS-2672: Attempting to start 'ora.evmd' on 'rac2'CRS-2676: Start of 'ora.evmd' on 'rac2' succeededrac2 2009/12/29 16:16:42 /u01/rk/grid/cdata/rac2/backup_20091229_161642.olrPreparing packages for installation...cvuqdisk-1.0.7-1Configure Oracle Grid Infrastructure for a Cluster ... succeeded9. Finished: [root@rac1 trace]# crsctl status resource -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.eons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 --------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac1 ora.oc4j 1 OFFLINE OFFLINE ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac1 Rene KundersmaOracle Technology Services, The Netherlands

Below I will describe the actions one needs to perform when the Oracle 11gr2 Grid Infrastructure ORACLE HOME needs to be moved to a new location. Please note:- You will loose already registered...

11gR2 Grid Infrastructure Installation

There is so much to tell about the new features that come with 11gR2, this new release gives me input for years ! Since the "Oracle Database New Features Guide 11g Release 2" does a good job here, I am not even trying to cover some of that.I will however try to discuss some highlights or cool new things that changed since the previous (11gR1) release. 11gR2 Grid Infrastructure is one of those things.11gR2 Grid Infrastructure combines Clusterware and ASM in one Oracle home and can be described as the next step in Grid Computing. If you are familiar with previous Clusterware and ASM releases, you will recognize the new functionality and way of working and realize this is indeed the next step in what we need for enabling Enterprise Grid. Deployment is simpler, faster and we are not talking about nodes anymore, but about services that live on resources.One of the new features of 11gR2 is Grid Plug and Play, also called GPnP. Let me repeat what the documentation says about GPnP:"Grid Plug and Play (GPnP) eliminates per-node configuration data and the need for explicit add and delete nodes steps. This allows a system administrator to take a template system image and run it on a new node with no further configuration. This removes many manual operations, reduces the opportunity for errors, and encourages configurations that can be changed easily. Removal of the per-node configuration makes the nodes easier to replace, because they do not need to contain individually-managed state.Grid Plug and Play reduces the cost of installing, configuring, and managing database nodes by making their per-node state disposable. It allows nodes to be easily replaced with regenerated state"Some of the key enablers for GPnP are GNS and DHCP. GNS, the Grid Naming Service is described here. Since all of the requirements for a Grid Infrastructure installation are clearly documented in the "Grid Infrastructure installation guide", there is no need to discuss this. This posting however is made to demo how to do an "Advanced Installation" of the Grid Infrastructure your self and show how to do an installation for education purposes, for example a situation at home where you want to test the setup of Oracle Grid Infrastructure with your own DNS and DHCP server. In real life, at customer sites, DNS and DHCP servers are all in place and Oracle Grid Infrastructure can leverage from these existing services.Since most steps of the Oracle Grid Infrastructure installation are easy I will only only focus on the details I want to discuss regarding GNS and DHCP.Oracle Grid Infrastructure can be downloaded here and when you made sure all prerequisites are checked you can start the installation by executing runInstaller.It does makes sense to install the Oracle Grid Infrastructure with a different user id then the Oracle database. For this the Oracle documentation again has some sound examples. Because of this I had to make sure permissions for directories and for example ASM disks are setup with 'grid' permissions instead of 'oracle' (and both oinstall as group)I used user "grid" to install the Oracle Grid Infrastructure and since I wanted to install and configure the Oracle Grid Infrastructure I chose the first option. A typical installation does not have GNS and since the purpose of the posting is to explain about the setup with GNS, the "Advanced Installation" option was chosen.Language, speaks for itself.Okay, this basically is the most important step of the setup. At this step you have to define the name for your cluster. In my case "cluster01", that is an easy one as there are no relations for this. The SCAN name however, is the "Simple Client Access Name" and will be setup by the Oracle Grid Infrastructure. This SCAN name will resolve to three ip addresses within the cluster. The good news is that you don't have to do much for it, just make up a name that your clients will use later to acces databases in your cluster. SCAN Port 1521 is the default port we always use for SQLNet. The SCAN name has to be in the GNS Sub Domain as explained below:The option "Configure GNS" was checked. If this box was not checked, still SCAN could be used, but then, I had to setup the SCAN entries in DNS myself, with the SCAN name resolving to three different ip addresses.However, since GNS is checked, the Grid Naming Service will be configured and GNS will setup my SCAN name. The only requirement is that a GNS Sub domain must be made and the DNS must be configured so that each request for this Sub Domain will be delegated to the GNS Sub Domain, so that GNS can handle the request.The GNS VIP address is the ip address of the server that will host the GNS. You need to make sure this one is available for use.You may ask yourself why this all is needed. Well, imagine yourself a cluster, where nodes are added and removed dynamically. In this situation, the complete administration with ip address management and name resolution management is done by the cluster itself. No need to do any manual work in updating connection strings, configuring ip numbers etc.So how does it work:First, my (named, linux) DNS is running on 10.161.102.40.This DNS does the naming for cluster01.nl.oracle.com and pts.local.For cluster01.nl.oracle.com a "delegation" is made, so that every request to a machine in the domain .cluster01.nl.oracle.com is delegated to the GNS. (with the GNS VIP).In DNS:cluster01.nl.oracle.com NS gns.cluster01.nl.oracle.comgns.cluster01.nl.oracle.com. 10.161.102.55So, once the cluster installation is done, the GNS in the cluster will be stared and a request to scan.cluster01.nl.oracle.com will be forwarded to the GNS. The GNS will then take care of the request and answer which three nodes in the cluster will serve as scan listeners:[root@gridnode01pts05 ~]# nslookup scan.cluster01.nl.oracle.comServer: 10.161.102.40Address: 10.161.102.40#53Non-authoritative answer:Name: scan.cluster01.nl.oracle.comAddress: 10.161.102.78Name: scan.cluster01.nl.oracle.comAddress: 10.161.102.79Name: scan.cluster01.nl.oracle.comAddress: 10.161.102.77Also, with dig, you can see all information coming from GNS:[root@dns-dhcp ~]# dig scan.cluster01.nl.oracle.com; DiG 9.3.4-P1 scan.cluster01.nl.oracle.com;; global options: printcmd;; Got answer:;; ->>HEADER

There is so much to tell about the new features that come with 11gR2, this new release gives me input for years ! Since the "Oracle Database New Features Guide 11g Release 2" does a good job here, I...

Cold failover for a single instance RAC database

This blog posting is about protecting an instance from a 10.2/11.1 single instance RAC database so that it can act in a cold-failover situation. I want to refer to the great document "Using Oracle Clusterware to Protect a single instance oracle database 11g" written by my collegue Philip Newlan since most input comes from here.The mentioned pdf describes how to make sure a single instance database can failover with the use of Oracle Clusterware.With this posting I want to show how to do this for a single instance RAC database where you have to manage instance1' and 'instance2' instead of just one instance.Since the grid control agent may have some problems with an instance[number] travelling from node1 to node2 this choice was made on purpose for some customer. Other reason for this awkward solution is that the application for some reason cannot run with two instances concurrently. Also, the fact that instance deployment with sequential instance numbers is standard in their grid environment the choice has been made to do this with different instance numbers instead of one.Within this posting I will also show the required updates in tnsnames and spfile.First, I made sure the OCR entries of the RAC database, the Services and the instances are removed from the OCR. Then a "resource group" will be created. This is the container for all the resources.oracle@pts0138([crs]):/ora/product/11.1.0/crs> crs_profile -create \oss.xdbprk.rg -t application -a \/ora/product/11.1.0/crs/crs/public/act_resgroup.pl -o "ci=600" oracle@pts0138([crs]):/ora/product/11.1.0/crs> crs_register oss.xdbprk.rgNow, let's verify the new entry:oracle@pts0138([crs]):/ora/product/11.1.0/crs/crs/public> crsstat | grep rgHA Resource Target Stateoss.xdbprk.rg OFFLINE OFFLINEThe scripts mentioned in the pdf are placed in $CRS_HOME/crs/public on both nodes. I made sure the scripts are executable and tested them:export CLUSTERWARE_HOME=/ora/product/11.1.0/crs/export ORACLE_HOME=/ora/product/10.2.0/db_2export _USR_ORA_LANG=$ORACLE_HOME export _USR_ORA_SRV=xdbprk2 export _USR_ORA_FLAGS=1 oracle@pts0138(xdbprk2):/tmp> $CLUSTERWARE_HOME/crs/public/act_db.pl startSQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 10:12:21 2009Copyright (c) 1982, 2007, Oracle. All Rights Reserved.SQL> Connected to an idle instance.SQL> ORACLE instance started.Total System Global Area 536870912 bytesFixed Size 2085360 bytesVariable Size 150998544 bytesDatabase Buffers 377487360 bytesRedo Buffers 6299648 bytesDatabase mounted.Database opened.SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,Data Mining and Real Application Testing optionsAlso tested the stop function.oracle@pts0138(xdbprk2):/tmp> $CLUSTERWARE_HOME/crs/public/act_db.pl stopSQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 10:12:37 2009Copyright (c) 1982, 2007, Oracle. All Rights Reserved.SQL> Connected.SQL> Database closed.Database dismounted.ORACLE instance shut down.SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,Data Mining and Real Application Testing optionsoracle@pts0138(xdbprk2):/ora/product/11.1.0/crs/crs/public>The action above was executed on both nodes. Just to verify if the instance could be started/stoped with the scripts. Only problem was: for each node, the ORACLE_SID had to be changed, as both nodes have another ORACLE_SID for that database. With only one SID you would not have the problem.Then, the failover resource was created and registered.oracle@pts0138([crs]):/tmp> crs_profile -create oss.xdbprk.db-cold-failover \-t application -r oss.xdbprk.rg -a /ora/product/11.1.0/crs/crs/public/act_db.pl -o "ci=20,ra=5,osrv=xdbprk,ol=/ora/product/10.2.0/db_2,oflags=1,rt=600"oracle@pts0138([crs]):/tmp> crs_register oss.xdbprk.db-cold-failoverThe value osrv=xdbprk will never work as this is not the correct instance name for any of the nodes. Even if I made the value osrv=xdbprk1, then the script would only work on one of the nodes i.e. the node that had the appropriate init.ora etc.So, leaving the value osrv=xdbprk to this, I actually hard-coded the ORACLE_SID on both sides of the cluster in the act_db.pl. Since the value is now hard coded, it should work. This clearly limits the option to re-use the script for other database, so I'd better change the name of the script to act_db_xdbprk.pl if I do this for real.So, how will the resource profiles look now ?oracle@pts0101([crs]):/var/opt/oracle> crs_profile -print oss.xdbprk.rgNAME=oss.xdbprk.rgTYPE=applicationACTION_SCRIPT=/ora/product/11.1.0/crs/crs/public/act_resgroup.plACTIVE_PLACEMENT=0AUTO_START=restoreCHECK_INTERVAL=600DESCRIPTION=oss.xdbprk.rgFAILOVER_DELAY=0FAILURE_INTERVAL=0FAILURE_THRESHOLD=0HOSTING_MEMBERS=OPTIONAL_RESOURCES=PLACEMENT=balancedREQUIRED_RESOURCES=RESTART_ATTEMPTS=1SCRIPT_TIMEOUT=60START_TIMEOUT=0STOP_TIMEOUT=0UPTIME_THRESHOLD=7dUSR_ORA_ALERT_NAME=USR_ORA_CHECK_TIMEOUT=0USR_ORA_CONNECT_STR=/ as sysdbaUSR_ORA_DEBUG=0USR_ORA_DISCONNECT=falseUSR_ORA_FLAGS=USR_ORA_IF=USR_ORA_INST_NOT_SHUTDOWN=USR_ORA_LANG=USR_ORA_NETMASK=USR_ORA_OPEN_MODE=USR_ORA_OPI=falseUSR_ORA_PFILE=USR_ORA_PRECONNECT=noneUSR_ORA_SRV=USR_ORA_START_TIMEOUT=0USR_ORA_STOP_MODE=immediateUSR_ORA_STOP_TIMEOUT=0USR_ORA_VIP=oracle@pts0101([crs]):/var/opt/oracle> crs_profile -print oss.xdbprk.db-cold-failoverNAME=oss.xdbprk.db-cold-failoverTYPE=applicationACTION_SCRIPT=/ora/product/11.1.0/crs/crs/public/act_db.plACTIVE_PLACEMENT=0AUTO_START=restoreCHECK_INTERVAL=20DESCRIPTION=oss.xdbprk.db-cold-failoverFAILOVER_DELAY=0FAILURE_INTERVAL=0FAILURE_THRESHOLD=0HOSTING_MEMBERS=OPTIONAL_RESOURCES=PLACEMENT=balancedREQUIRED_RESOURCES=oss.xdbprk.rgRESTART_ATTEMPTS=5SCRIPT_TIMEOUT=60START_TIMEOUT=600STOP_TIMEOUT=0UPTIME_THRESHOLD=7dUSR_ORA_ALERT_NAME=USR_ORA_CHECK_TIMEOUT=0USR_ORA_CONNECT_STR=/ as sysdbaUSR_ORA_DEBUG=0USR_ORA_DISCONNECT=falseUSR_ORA_FLAGS=1USR_ORA_IF=USR_ORA_INST_NOT_SHUTDOWN=USR_ORA_LANG=/ora/product/10.2.0/db_2USR_ORA_NETMASK=USR_ORA_OPEN_MODE=USR_ORA_OPI=falseUSR_ORA_PFILE=USR_ORA_PRECONNECT=noneUSR_ORA_SRV=xdbprkUSR_ORA_START_TIMEOUT=0USR_ORA_STOP_MODE=immediateUSR_ORA_STOP_TIMEOUT=0USR_ORA_VIP=Okay, and then the basic test of starting the resource, first all is down:oracle@pts0101([crs]):/var/opt/oracle> crsstatHA Resource Target State----------- ------ -----oss.xdbprk.db-cold-failover OFFLINE OFFLINEoss.xdbprk.rg OFFLINE OFFLINEThen, the start:oracle@pts0101([crs]):/var/opt/oracle> crs_start oss.xdbprk.db-cold-failoverAttempting to start `oss.xdbprk.db-cold-failover` on member `pts0138`Start of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.oracle@pts0138([crs]):/var/opt/oracle> ps -ef | grep smon | grep dbprkoracle 31568 1 0 11:51 ? 00:00:00 ora_smon_xdbprk2And the relocate:oracle@pts0101([crs]):/var/opt/oracle> crs_relocate -f oss.xdbprk.db-cold-failoverAttempting to stop `oss.xdbprk.db-cold-failover` on member `pts0138`Stop of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.Attempting to stop `oss.xdbprk.rg` on member `pts0138`Stop of `oss.xdbprk.rg` on member `pts0138` succeeded.Attempting to start `oss.xdbprk.rg` on member `pts0101`Start of `oss.xdbprk.rg` on member `pts0101` succeeded.Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0101`Start of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.Let's verify if it runs on the other node:oracle@pts0101([crs]):/var/opt/oracle> ps -ef | grep smon | grep dbprkoracle 11568 1 0 11:55 ? 00:00:00 ora_smon_xdbprk1And stopped on the original:oracle@pts0138(xdbprk2):/ora/product/10.2.0/db_2/dbs> ps -ef | grep smon | grep dbprkOkay, what is left to do from here:Service names that used to be managed by CRS, now have to be coded hard in the spfile so that they register each time with the listener:service_names in spfile added:dbprk.xe.griddbprk.xe.suppdbprk.xe.linkAlso, for each node, another local listener will be used. I made sure this is in the spfile:xdbprk1.local_listener='pts0101-LOCAL-LISTENER'xdbprk2.local_listener='pts0138-LOCAL-LISTENER'In order to be sure there can only be started one instance the cluster_database_instances_parameter is set to 1.*.cluster_database_instances=1For tnsnames a normail failover entry is created, if the first instance is down, the next will be found (and should be running):oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> tnsping dbprk.xe.gridTNS Ping Utility for Linux: Version 10.2.0.4.0 - Production on 18-SEP-2009 12:39:14Copyright (c) 1997, 2007, Oracle. All rights reserved.Used parameter files:/etc/oss/sqlnet.oraUsed TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=pts0101-grid.nl.eu.abnamro.com)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=pts0138-grid.nl.eu.abnamro.com)(PORT=1521))(LOAD_BALANCE=ON))(CONNECT_DATA=(SERVICE_NAME=dbprk.xe.grid)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20))))OK (10 msec)So, another basic test, what is the situation:HA Resource Target State----------- ------ -----oss.xdbprk.db-cold-failover ONLINE ONLINE on pts0101oss.xdbprk.rg ONLINE ONLINE on pts0101oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crs_start oss.xdbprk.db-cold-failoverAttempting to start `oss.xdbprk.db-cold-failover` on member `pts0101`Start of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.In which instance will my session end ?oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> sqlplus rk/rk@dbprk.xe.gridSQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:39:55 2009Copyright (c) 1982, 2007, Oracle. All Rights Reserved.Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,Data Mining and Real Application Testing optionsSQL> select instance_name from v$instance;INSTANCE_NAME----------------xdbprk1And after the relocate, the session should go to the other instance:oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crs_relocate -f oss.xdbprk.db-cold-failoverAttempting to stop `oss.xdbprk.db-cold-failover` on member `pts0101`Stop of `oss.xdbprk.db-cold-failover` on member `pts0101` succeeded.Attempting to stop `oss.xdbprk.rg` on member `pts0101`Stop of `oss.xdbprk.rg` on member `pts0101` succeeded.Attempting to start `oss.xdbprk.rg` on member `pts0138`Start of `oss.xdbprk.rg` on member `pts0138` succeeded.Attempting to start `oss.xdbprk.db-cold-failover` on member `pts0138`Start of `oss.xdbprk.db-cold-failover` on member `pts0138` succeeded.oracle@pts0101([crs]):/ora/dbprk/admin/xdbprk1/bdump> crsstatHA Resource Target State----------- ------ -----oss.xdbprk.db-cold-failover ONLINE ONLINE on pts0138oss.xdbprk.rg ONLINE ONLINE on pts0138oracle@pts0101(xdbprk1):/ora/dbprk/admin/xdbprk1/bdump> sqlplus rk/rk@dbprk.xe.gridSQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:41:22 2009Copyright (c) 1982, 2007, Oracle. All Rights Reserved.Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,Data Mining and Real Application Testing optionsSQL> select instance_name from v$instance;INSTANCE_NAME----------------xdbprk2As an extra test, to make sure the two instances cannot be started concurrently, started with a test to start the second instance after starting the first.As you can see this is not possible.oracle@pts0101(*):/var/opt/oracle> db xdbprk1ORACLE_SID=xdbprk1ORACLE_HOME=/ora/product/10.2.0/db_2oracle@pts0101(xdbprk1):/var/opt/oracle> sqlplus / as sysdbaSQL*Plus: Release 10.2.0.4.0 - Production on Fri Sep 18 12:42:02 2009Copyright (c) 1982, 2007, Oracle. All Rights Reserved.Connected to an idle instance.SQL> startup;ORACLE instance started.Total System Global Area 536870912 bytesFixed Size 2085360 bytesVariable Size 331353616 bytesDatabase Buffers 197132288 bytesRedo Buffers 6299648 bytesDatabase mounted.ORA-01092: ORACLE instance terminated. Disconnection forcedRene KundersmaOracle Technology Services, The Netherlands

This blog posting is about protecting an instance from a 10.2/11.1 single instance RAC database so that it can act in a cold-failover situation. I want to refer to the great document "Using Oracle...

Mounting Oracle VM Templates without LVMs in it.

In blog entry "Provisioning your GRID with Oracle VM Templates" I explained how to mount a filesystem from an Oracle VM template that was build on a logical volume. It seems however, that Oracle VM templates are also shipped without logical volumes in it, just plain ext3.So how would one manage to mount that then ?First, setup the loop, to see what's in it.[root@pts05 ] losetup /dev/loop99 /OVS/running_pool/gridnode01/System.img[root@pts05 ]# fdisk -lu /dev/loop99Disk /dev/loop99: 6530 MB, 6530871808 bytes255 heads, 63 sectors/track, 793 cylinders, total 12755609 sectorsUnits = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System/dev/loop99p1 * 63 64259 32098+ 83 Linux/dev/loop99p2 64260 8562644 4249192+ 83 Linux/dev/loop99p3 8562645 12739544 2088450 82 Linux swap / SolarisSo, three plain partitions, no logical volumes in it.What to do next ? it seems NOT possible to inform the kernel about that three partitions:[root@pts05 p]# partprobe -s /dev/loop99Error: Error informing the kernel about modifications to partition /dev/loop99p1 -- Invalid argument. This means Linux won't know about any changes you made to /dev/loop99p1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.Error: Error informing the kernel about modifications to partition /dev/loop99p2 -- Invalid argument. This means Linux won't know about any changes you made to /dev/loop99p2 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.Error: Error informing the kernel about modifications to partition /dev/loop99p3 -- Invalid argument. This means Linux won't know about any changes you made to /dev/loop99p3 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.Warning: The kernel was unable to re-read the partition table on /dev/loop99 (Invalid argument). This means Linux won't know anything about the modifications you made until you reboot. You should reboot your computer before doing anything with /dev/loop99.Okay, since I don't want to reboot, this is not an option.Maybe we need to setup the loop, but with an offset to the partition I need.Since the sector size is 512 bytes and I start at 64260, my offset will be: 512 * 64260 = 32901120. Why do I start at 64260, since I guess partition one is the /boot partition.[root@pts05 tools]# losetup /dev/loop98 /OVS/running_pool/gridnode01/System.img -o 32901120[root@pts05 tools]# losetup -a | grep offset/dev/loop98: [0811]:6127628 (/OVS/running_pool/gridnode01/System.img), offset 32901120Yes, there it is ! Mounting is easy now ![root@pts05 p]# mkdir /tmp/p; mount /dev/loop98 /tmp/p/[root@pts05 p]# df -m | loop98/dev/loop98 4013 2071 1901 53% /tmp/p[root@pts05 p]# ls -l /tmp/p/total 196drwxr-xr-x 2 root root 4096 Sep 3 00:03 bindrwxr-xr-x 2 root root 4096 Mar 25 05:44 bootdrwxr-xr-x 2 root root 4096 Sep 4 09:58 crsdrwxr-xr-x 2 root root 4096 Apr 10 07:14 devdrwxr-xr-x 96 root root 12288 Sep 10 07:03 etcdrwxr-xr-x 4 root root 4096 Sep 8 21:30 homedrwxr-xr-x 13 root root 4096 Sep 3 00:03 libdrwx------ 2 root root 16384 Mar 25 05:44 lost+founddrwxr-xr-x 2 root root 4096 Jan 9 2009 mediadrwxr-xr-x 2 root root 4096 Jan 21 2009 miscdrwxr-xr-x 3 root root 4096 Sep 4 09:40 mntdr-xr-xr-x 2 root root 4096 Sep 8 21:22 netdrwxr-xr-x 4 root root 4096 Sep 9 22:03 optdrwxr-xr-x 2 root root 4096 Mar 25 05:44 procdrwxr-x--- 2 root root 4096 Sep 8 22:30 rootdrwxr-xr-x 2 root root 12288 Sep 9 22:05 sbindrwxr-xr-x 2 root root 4096 Mar 25 05:44 selinuxdrwxr-xr-x 2 root root 4096 Jan 9 2009 srvdrwxr-xr-x 2 root root 4096 Mar 25 05:44 sysdrwxr-xr-x 3 root root 4096 Apr 10 07:03 tftpbootdrwxrwxrwt 17 root root 4096 Sep 10 14:57 tmpdrwxr-xr-x 2 root root 4096 Sep 2 22:13 u01drwxr-xr-x 14 root root 4096 Apr 10 07:03 usrdrwxr-xr-x 21 root root 4096 Apr 10 07:05 varRene KundersmaOracle Technology Services, The Netherlands

In blog entry "Provisioning your GRID with Oracle VM Templates" I explained how to mount a filesystem from an Oracle VM template that was build on a logical volume. It seems however, that Oracle VM...

Para-virtualizing disk and network drivers for Linux HVM Guests

Lately one of Oracle's customers decided to run their RedHat Enterprise Linux 3 systems onto Oracle VM. Running hardware virtualized is very straightforward and easy to do. However, you may want to improve performance by using para-virtualized drivers. On the topic on para-virtualized drivers, this entry will describe how do do it, what's possible, what's not and where to pay attention to.To begin a short description: the picture below shows us Oracle VM can run both hardware and para-virtualized guests. Hardware virtualization uses device emulation. In general people get more performance using para-virtualization, as the quests VMs can use these as native drivers. First one should verify if Oracle VM supports the platform. So, in this situation the Oracle VM online documentation says RedHat Enterprise Linux 3 is certified for hardware virtualization and can even run with para-virtualized drivers.In order to use para-virtualized drivers the requirement is that at least RedHat Enterprise Linux 3 update 9 is used.Having para-virtualized drivers in place makes the fully-virtualized guests VM 'para-virtualization-aware'. The benefit of this is the significant I/O performance improvement say 'speed' of the vm.Please note that having para-virtualized drivers != running a para-virtualized VM. In this situation the only improvement you will see is on block (disk) and network performance.In order to start using the paravirtualized drivers, you should download them first from the Oracle Unbreakable Linux Network (ULN). The file you need is kmod-xenpv-*el3.i686.rpm and you can find it in the appropriate channel. The file needs to be installed into each RHEL 3 HVM guest. (rpm -ivh kmod-xenpv-smp-0.1-9.el3.i686.rpm)After this make sure the modules just installed with rpm are copied to /lib/modules/'uname -r'/extra/xenpv. After this the modules need to be loaded:[root@gridnode01 /]# insmod xen-platform-pci.o[root@gridnode01 /]# insmod xen-balloon.o[root@gridnode01 /]# insmod xen-vbd.o[root@gridnode01 /]# insmod xen-vnif.oAlso make sure the eth0 alias is setup in /etc/modules.confalias eth0 xen-vnifSo, for example, your vm.cfg for the HVM guest may look like this:acpi = 1apic = 1boot = 'c'bootloader = '/usr/bin/pygrub'builder = 'hvm'device_model = '/usr/lib/xen/bin/qemu-dm'disk = ['file:/OVS/running_pool/112_pdrtest/system.img,hda,w','file:/OVS/running_pool/112_pdrtest/oh0.img,hdb,w',]disk_other_config = []kernel = '/usr/lib/xen/boot/hvmloader'keymap = 'en-us'maxmem = 1024memory = 1024name = '12_pdrtest'on_crash = 'restart'on_reboot = 'restart'pae = 1serial = 'pty'uuid = 'f2e29b25-391f-74e8-b0cf-cbc69bb7905'vcpus = 1vif = ['type=ioemu, mac=00:16:4E:1C:42:7E, bridge=xenbr1']vnc = 1vncconsole = 1vnclisten = '0.0.0.0'vncpasswd = oraclevncunused = 1What now needs to change is the entry "type=ioemu" from the vif configuration line.The other item to change is for the IO. New physical devices need to be added to the vm.cfg file and one has to make sure they use the xen-vdb disk driver. Please note that once the new disk is created the data needs to be moved to that disk.Now disk and network drivers are para-virtualized performance will be better.The ability to keep running older operating systems in a virtualized environment is great.Oracle Certifies most common operating systems on Oracle VM. The Oracle software, for instance Real Application Clusters still needs to be certified with Oracle VM and the guest VM OS image. See also this link for best practices and certification.Rene KundersmaOracle Technology Services, The Netherlands

Lately one of Oracle's customers decided to run their RedHat Enterprise Linux 3 systems onto Oracle VM. Running hardware virtualized is very straightforward and easy to do. However, you may want to...

Feature enthusiasm: Oracle Flashback Database

IntroductionOn a daily basis database changes to production database systems happen everywhere. In order to provide rollback options for failed changes people most often use the regular rman restore (point in time recovery) strategy. However, restoring databases (or parts of databases) can take long, especially as databases are getting larger and larger. As longer downtime obviously is not something people look for, Flashback database maybe something to think of. I am very enthusiastic about this underrated featureBackgroundSince Oracle version 10.1, "Oracle Flashback Database" was introduced. In the documentation Oracle stated the following:"This feature introduces the FLASHBACK DATABASE statement in SQL. It let you quickly bring your database to a prior point in time by undoing all the changes that have taken place since that time. This operation is fast, because you do not need to restore the backups. This in turn results in much less downtime following data corruption or human error."So, Flashback database is simpler and faster to use then point in time recovery because it does not require restoring datafiles from backup and it requires applying fewer changes from the archived redo logs.Although most of us already know about availability of this option, it isn't used as often as rollback strategy for database changes. To be complete in this background details, Flashback technology brings more options to the table:·Oracle Flashback Table, which returns one or more tables to their contents at a previous time;·Oracle Flashback Drop, which undoes the effects of the DROP TABLE operation;·Oracle Flashback Query, which is used to query the contents of the database at a past time;·Oracle Flashback Version Query, which lets you view past states of data;·Oracle Flashback Transaction Query, which is used to review transactions affecting a table over time.This blog entry is about database Flashback only because it is that functionality that comes in very handy for undoing database changes or testing cycles. Normal and guaranteed restore pointsFor Flashback Database there are two type of restore points:1. Normal restore points2. Guaranteed restore pointsThe documentation says "Restore points provide capabilities related to Flashback Database as well as other recovery operations. Guaranteed restore points, in particular, provide a complementary capability to Flashback Database, allowing you to select an SCN and enforce the requirement that Flashback Database be usable to that SCN, though not necessarily to SCNs between the guaranteed restore point and the present SCN"So, in order to make sure changes can be rolled back you can imagine the guarantee option is the one we need. Because, if space in the Flash recovery area is low, then Flashback logs may be deleted to free space for files required by the configured retention policy and that is not what we want.During space pressure, the Flashback logs are deleted under two conditions: 1. When an archivelog is deleted, it would also delete the Flashback logs that are dependent on the archived log. 2. When quota is shrunk, then Flashback logs are deleted to reclaim diskspace. This is decreasing the DB_RECOVERY_FILE_DEST_SIZE to such a value that the Flashback log itself will run into a space pressure and than the old Flashback logs will get deleted.So, using guaranteed restore points is the only way to ensure that you can use Flashback Database to return to a specific point in time and guarantee the size of the Flashback window. I.e. NO DELETION of Flashback logs.In the end, when a database is 'Flashed back' to its state is at some past target time using Flashback Database, each block changed since that time is restored from the copy of the block in the Flashback logs most immediately prior to the desired target time. The redo log is then used to re-apply changes since the time that block was copied to the Flashback logs. In one of the examples below you will see that redo is applied to roll forward within the Flashback procedure.Be careful of the following remark: "if no files are eligible for deletion from the Flash recovery area because of the requirements imposed by your retention policy and the guaranteed restore point, then the database behaves as if it has encountered a disk full condition. In many circumstances, this causes your database to halt". You may get messages about a full Flash Recovery Area (FRA) in your alert.log, also you may see this kind of error messages when your diskgroup is filled up with Flashback logs:ORA-19624: operation failed, retry possibleORA-19504: failed to create file "+RK"ORA-17502: ksfdcre:4 Failed to create file +RKORA-15041: diskgroup space exhaustedAn extra advantage of guaranteed restore points is that each block that changes is logged only once. Hence space requirement and performance overhead is lesser with guaranteed restore points then with normal restore points.How to setup and use FlashbackUsing Flashback requires extra Flash Recovery Area diskspace as Flashback logs are not recycled when a Guaranteed Restore Point is in effect. Actually, you don't want any Flashback file from a restore point to be dropped until you are sure you don't need it again.Remaining question is: How large should my FRA be sized extra in order to use Flashback ? Well, as a general rule of thumb the answer is: The total size of all the archive logs that would be created for the same change will suffice as required space for flashback logs. Please note that you need this size as additional space and this is just an estimate because how much space Flashback logs will take depends on your actions. In one test I did, that only inserted data into a table 1.5GB of archives was created with only 85 megs of Flashback. Results of testing with updates and deletes will be added later.In order to use Flashback, the two init.ora parameters DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE should be set. The DB_RECOVERY_FILE_DEST_SIZE parameter specifies the maximum total bytes to be used for the FRA. DB_RECOVERY_FILE_DEST specifies the location of the FRA. Please note, in case of RAC databases the value requires to be the same on all instances. Most of us already are using the DB_RECOVERY_FILE_DEST* parameters for their RMAN backups and archive logs. However, by default Flashback is disabled as you can see by querying v$database, so you still need to enable that.SQL> SELECT flashback_on from v$database;FLASHBACK_ON------------NO So, no Flashback is enabled and no Flashback logs exist:SQL> SELECT * FROM v$flashback_database_log;no rows selectedIf you try to create your first guaranteed restore point while the database is running (open) you receive the following error:SQL> create restore point rk1 guarantee flashback database;create restore point rk1 guarantee flashback database*ERROR at line 1:ORA-38784: Cannot create restore point 'RK1'.ORA-38787: Creating the first guaranteed restore point requires mount mode when flashback database is off.So, before creating the guaranteed Flashback the database has to be brought into mount mode:oracle@node01(xrku1):/var/opt/oracle> srvctl stop database -d xrku -o immediateoracle@node01(xrku1):/var/opt/oracle> srvctl start instance -i xrku1 -d xrku -o mountNow the restore point can be made and the database can be opened :SQL> create restore point rk1 guarantee flashback database;Restore point created.SQL> alter database open;You should now be able to see the restore point and identify the Flashback logfile:SQL> SELECT flashback_on from v$database;FLASHBACK_ON------------RESTORE POINT ONLYSQL> SELECT NAME, SCN, TIME,SQL> GUARANTEE_FLASHBACK_DATABASESQL> FROM V$RESTORE_POINTSQL> WHERE GUARANTEE_FLASHBACK_DATABASE='YES';NAME SCN TIME GUARANTEE_FL---------- ---------- --------------------------------------------------------------------------- ------------RK1 2066217 29-JUL-09 10.34.05.000000000 AMSQL> select name from V$FLASHBACK_DATABASE_LOGFILE;NAME--------------------------------------------+FLASH_RECOVERY_AREA/xrku/flashback/log_1.931.693484445So, as an example, after creating or dropping some tables, in order to execute a Flashback database, the database has to be placed into mount mode again:SQL> shutdown immediateSQL> startup mountSQL> flashback database to restore point rk1;Flashback complete.This is what you will see in the alert.log of your instance:Wed Jul 29 12:03:38 2009flashback database to restore point rk1Wed Jul 29 12:03:38 2009Flashback Restore StartFlashback Restore CompleteCompleted: flashback database to restore point rk1After Flashing back your database, open it with an "alter database open resetlogs" command, you should now have the situation as of before the creation or dropping of the tables.Benefits, specials and things to be careful of with Flashback database As said, Flashback database can be used as a strategy for point in time recovery (PITR) and as an alternative to the rman restore and recover. As a matter of fact, the Flashback log captures old versions of changed blocks. You could look at this as a continuous backup. Then, the logs are 'replayed' to restore the database to a certain time. Flashback 'restores' just changed blocks. The process responsible for writing the Flashback log is the rvwr process:oracle@node01(xrku1):/ora/rku/admin/xrku1/bdump> ps -ef | grep oracle | grep rvwroracle 2677 1 0 12:30 ? 00:00:00 ora_rvwr_xrku1You will only see this process if Flashback is enabled.Flashback database also has limitations that you should be aware of before using it: 1. Flashback Database can only undo changes to a datafile made by an Oracle database. For example:Users created will be Flashed back as these are database changes. sysdba users however, will not be removed from password file (v$pwfile_users). The same counts for the init.ora file where Flashback doesn't Flashback init.ora settings. 2. Flashback Database cannot be used to repair media failures, or to recover from accidental deletion of datafiles.3. You cannot use Flashback Database to undo a shrink datafile operation, actually, shrinking a datafile or dropping a tablespace can prevent Flashing back the Database. For example:SQL> create tablespace x datafile size 10M;Tablespace createdSQL> select file_name from dba_data_files where tablespace_name = 'X';FILE_NAME---------------------------------------------------------+DATA_AREA01/xrku/datafile/x.329.6934914571 row selected.SQL> alter database datafile '+DATA_AREA01/xrku/datafile/x.329.693491457' resize 5M;Database altered.SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.SQL> startup mount;ORACLE instance started.Total System Global Area 536870912 bytesFixed Size 2085360 bytesVariable Size 331353616 bytesDatabase Buffers 197132288 bytesRedo Buffers 6299648 bytesDatabase mounted.SQL> flashback database to restore point rk2;flashback database to restore point rk2*ERROR at line 1:ORA-38766: cannot flashback data file 8; file resized smallerORA-01110: data file 8: '+DATA_AREA01/xrku/datafile/x.329.693491457'As another example, tablespaces created after the restore point will be dropped and files will be deleted upon Flashback execution (as expected):Wed Jul 29 12:18:20 2009flashback database to restore point rk1Wed Jul 29 12:18:20 2009Flashback Restore StartDeleted Oracle managed file +DATA_AREA01/xrku/datafile/y.305.693490587Flashback: deleted datafile #9 in tablespace #9 from control file.Flashback: dropped tablespace #9: 'Y' from the control file.Deleted Oracle managed file +DATA_AREA01/xrku/datafile/x.329.693490579Flashback: deleted datafile #8 in tablespace #8 from control file.Flashback: dropped tablespace #8: 'X' from the control file.Flashback Restore CompleteCompleted: flashback database to restore point rk1Wed Jul 29 12:18:25 2009Tablespaces created after restore point X and before restore point X+1 will be Flashed back and 'recovered' automatically, Think about this and realize what a great idea it is from Oracle to combine rolling forwards and rolling backwards:flashback database to restore point rk2Wed Jul 29 12:30:57 2009Flashback Restore StartFlashback Restore CompleteFlashback Media Recovery StartMedia Recovery apply resetlogs offline range for datafile 1, incarnation : 0Media Recovery apply resetlogs offline range for datafile 2, incarnation : 0Media Recovery apply resetlogs offline range for datafile 3, incarnation : 0Media Recovery apply resetlogs offline range for datafile 4, incarnation : 0Media Recovery apply resetlogs offline range for datafile 5, incarnation : 0Media Recovery apply resetlogs offline range for datafile 6, incarnation : 0Media Recovery apply resetlogs offline range for datafile 7, incarnation : 0 parallel recovery started with 3 processesWed Jul 29 12:30:57 2009Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0 Mem# 0: +DATA_AREA01/xrku/onlinelog/group_1.354.689779921 Mem# 1: +DATA_AREA01/xrku/onlinelog/group_1.307.689779921Successfully added datafile 8 to media recoveryDatafile #8: '+DATA_AREA01/xrku/datafile/x.329.693491457'Wed Jul 29 12:30:57 2009Incomplete Recovery applied until change 2066572Flashback Media Recovery CompleteCompleted: flashback database to restore point rk24. If the database control file is restored from backup or re-created, all accumulated Flashback log information is discarded. You cannot use FLASHBACK DATABASE to return to a point in time before the restore or re-creation of a control file.5. When using Flashback Database with a target time at which a NOLOGGING operation was in progress, block corruption is likely in the database objects and datafiles affected by the NOLOGGING operation. For example, if you perform a direct-path INSERT operation in NOLOGGING mode, and that operation runs from 9:00 to 9:15 on April 3, 2005, and you later need to use Flashback Database to return to the target time 09:07 on that date, the objects and datafiles updated by the direct-path INSERT may be left with block corruption after the Flashback Database operation completes.6. For combinations Flashback and features like streams and dataguard with Flashback you should be aware of some extra constraints:Having the ability to enable your source (primary) database with Flashback Guaranteed Restore Points (and actually using Flashback) requires the physical standby database to be in full Flashback mode. Flashing back the 'primary' to 'SCN' requires to Flashback the target (to SCN-20) so that the log sequence can catch up. Note that enabling full Flashback on the target database requires extra disk space.As a general guideline Oracle advices 2x the total size of archives that will be created as a requirement for flashback data. You have to test this for your environment.Another thing to watch out for: "If the switchover involved a physical standby database, the primary and standby database roles are preserved during the Flashback operation. That is, the role in which the database is running does not change when the database is Flashed back to the target SCN or time to which you Flashed back the database." i.e. you have to do this your self. A last note on the combination flashback and dataguard: be sure to use the command 'flashback standby database ...' on your standy databaseFor streams this can of course only be done when there is a 1:1 relation ship between source and target. Note that the with Streams source and target database have their own log thread. It's obviously more difficult to find an SCN in source and in target that are on exact the same time.Disk usageThe information about required disk space is collected after some basic tests and is only applicable to Flashback database with a guaranteed restore points. Each action (delete, update, insert, drop) on a database generates an different amount of flashback logs. Also, as with normal actions, the amount of flashback data also depends on whether complete tables are updated or only specific columns. Depending on the action and the amount of data you may need more or less space, though, in general this list of values can be used as a guide.For my test a table with a size of 1000M was created, without an index.For insertion, deletion, updating (and dropping) of the complete table information was collected on the amount of flashback logs. For each action the most possible redo was generated:SummaryAs discussed guaranteed restore points can come in very helpful to rollback database changes. It's a requirement to configure your Flash Recovery Area (FRA) if that's not already done. In order to use FRA DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE need to be set. Flashback database could also come in very handy in testing cycles where you want to re-test performance for example over and over with the same set of data.In order to Flashback, bring your database down to mount state and Flashback to that restore point. Also, be sure to understand the mentioned limitations of this wonderful option so that you know in what state your database is after Flashing it back. Also know how to handle Flashback in combination with solutions like Dataguard and Streams. Carefully test this before heading for production. Make yourself aware with the technology.Be sure to reserve at least 1 x the amount you would need for archives to hold your Flashback logs, since Flashback logs requires storage also. Active monitoring off free FRA space is highly advised because a full FRA will halt your database. For this use the view v$flash_recovery_area_usage;Not using the FRA for you archives (i.e. using log_archive_dest_*) is not recommended, but brings us an extra option which maybe interesting for you as a temporary extra safety machanism: Since the location for archived logs is not FRA in this situation, a separate (temporary) diskgroup could be made for the FRA that will only be used for flashback logs. The advantage of this solution is that filling up the diskgroup (with Flashback logs) will not risk database availability for other databases. In order to create this temporary diskgroup of course free disks need to be available. It's however recommend by Oracle to put archives as well as flashback logs all in FRA.References464542.1 Can I Open A Physical Standby Db For R/W And Use Flashback to Put It Back?728374.1 How To Flashback Primary Database In Standby Configuration330535.1 Restore Points in Oracle10g Release2565535.1 Flashback Database Best Practices & Performance305648.1 What is a Flash Recovery Area and how to configure it369759.1 FLASH RECOVERY AREA and FLASHBACK database369755.1 Flashback Logs-Space managementOracle® Database Backup and Recovery Basics 10g Release 2 (10.2) - 7.5 Reversing Database Changes with Flashback DatabaseRene KundersmaOracle Expert Services, The Netherlands

Introduction On a daily basis database changes to production database systems happen everywhere. In order to provide rollback options for failed changes people most often use the regular rman restore...

Jumbo Frames for RAC Interconnect

At the moment for a customer I am investigating whether it's a good idea to use "Jumbo Frames" within a RAC environment. In the next couple of days I will post some results here.Until that time, I like to share some thoughts.First: why do you possibly want "Jumbo Frames" ?Answer: Jumbo Frames (in my Oracle world) are Ethernet frames that do not have a conventional payload of 1500 bytes, but 9000 bytes. As you can imagine the standard 1500 bytes is too small for our 'regular' Oracle database block of 8k. When you make sure the frames are 9000 byte, a block should fit in and this way eliminate the overhead.Eliminating the overhead would potentially improve performance and decrease CPU usage. This sound like a good theory and very applicable to use for the RAC interconnect where we do sent 8k (UDP) blocks from instance A to instance B.Investigation tells us Jumbo Frames are not (yet) an IEEE standard. This means, you could (I am not saying you will) have problems between the devices that supposed to be configured to handle frames of this size. The OS, network card as well as the switch all need to 'talk' the same size of "Jumbo Frames". Note that some vendors may call 4000 bytes Jumbo, some call 9000 bytes Jumbo.Theory tells us properly configured Jumbo Frames can eliminate 10% of overhead on UDP traffic. So how to test ? I guess an 'end to end' test would be best way. So my first test is a 30 minute Swingbench run against a two node RAC, not too much stress in the begin. The MTU configuration of the network bond (and the slave nics will be 1500 initially).After the test, collect the results on the total transactions, the average transactions per second, the maximum transaction rate (results.xml), interconnect traffic (awr) and cpu usage. Then, do exactly the same, but now with an MTU of 9000 bytes. For this we need to make sure the switch settings are also modified to use an MTU of 9000.B.t.w.: yes, it's possible to measure network only, but real-life end-to-end testing with a real Oracle application talking to RAC feels like the best approach to see what the impact is on for example the avg. transactions per second.In order to make the test as reliable as possible some remarks:- use guaranteed snapshots to flashback the database to its original state.- stop/start the database (clean the cache)B.t.w: before starting the test with an MTU of 9000 bytes the correct setting had to be proofed. One way to do this is using ping with a packet size (-s) of 8972 and prohibiting fragmentation (-M do).One could send Jumbo Frames and see if they can be sent without fragmentation.[root@node01 rk]# ping -s 8972 -M do node02-ic -c 5 PING node02-ic. (192.168.23.32) 8972(9000) bytes of data.8980 bytes from node02-ic. (192.168.23.32): icmp_seq=0 ttl=64 time=0.914 msAs you can see this is not a problem. While for packages larger then 9000 bytes, this is a problem:[root@node01 rk]# ping -s 8973 -M do node02-ic -c 5 --- node02-ic. ping statistics ---5 packets transmitted, 5 received, 0% packet loss, time 4003msrtt min/avg/max/mdev = 0.859/0.955/1.167/0.109 ms, pipe 2PING node02-ic. (192.168.23.32) 8973(9001) bytes of data.From node02-ic. (192.168.23.52) icmp_seq=0 Frag needed and DF set (mtu = 9000)Bringing back the MTU size to 1500 should also prohibit sending of fragmented 9000 packages:[root@node01 rk]# ping -s 8972 -M do node02-ic -c 5 PING node02-ic. (192.168.23.32) 8972(9000) bytes of data.--- node02-ic. ping statistics ---5 packets transmitted, 0 received, 100% packet loss, time 3999msBringing back the MTU size to 1500 and sending 'normal' packages should work again:[root@node01 rk]# ping node02-ic -M do -c 5 PING node02-ic. (192.168.23.32) 56(84) bytes of data.64 bytes from node02-ic. (192.168.23.32): icmp_seq=0 ttl=64 time=0.174 ms--- node02-ic. ping statistics ---5 packets transmitted, 5 received, 0% packet loss, time 3999msrtt min/avg/max/mdev = 0.174/0.186/0.198/0.008 ms, pipe 2An other way to verify the correct usage of the MTU size is the command 'netstat -a -i -n' (the column MTU size should be 9000 when you are performing tests on Jumbo Frames):Kernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgbond0 1500 0 10371535 0 0 0 15338093 0 0 0 BMmRUbond0:1 1500 0 - no statistics available - BMmRUbond1 9000 0 83383378 0 0 0 89645149 0 0 0 BMmRUeth0 9000 0 36 0 0 0 88805888 0 0 0 BMsRUeth1 1500 0 8036210 0 0 0 14235498 0 0 0 BMsRUeth2 9000 0 83383342 0 0 0 839261 0 0 0 BMsRUeth3 1500 0 2335325 0 0 0 1102595 0 0 0 BMsRUeth4 1500 0 252075239 0 0 0 252020454 0 0 0 BMRUeth5 1500 0 0 0 0 0 0 0 0 0 BMAs you can see my interconnect in on bond1 (build on eth0 and eth2). All 9000 bytes.Not finished yet, no conclusions yet, but here is my first result.You will notice the results are not that significantly.MTU 1500: TotalFailedTransactions : 0AverageTransactionsPerSecond : 1364MaximumTransactionRate : 107767TotalCompletedTransactions : 4910834MTU 9000: TotalFailedTransactions : 1AverageTransactionsPerSecond : 1336MaximumTransactionRate : 109775TotalCompletedTransactions : 4812122In a chart this will look like this:As you can see, the number of transactions between the two tests isn't really that significant, but the UDP traffic is less ! Still, I expected more from this test, so I have to put more stress to the test.I noticed the failed transaction, and found "ORA-12155 TNS-received bad datatype in NSWMARKER packet". I did verify this and I am sure this is not related to the MTU size. This is because I only changed the MTU size for the interconnect and there is no TNS traffic on that network.As said, I will now continue with tests that have much more stress on the systems:- number of users changed from 80 to 150 per database- number of databases changed from 1 to 2- more network traffic: - rebuild the Swingbench indexes without the 'REVERSE' option - altered the sequences and lowered increment by value to 1 and cache size to 3. (in stead of 800) - full table scans all the time on each instance- run longer (4 hours in stead of half an hour)Now, what you see is already improving. For the 4 hour test, the amount of extra UDP packets sent with an MTU size of 1500 compared to an MTU size of 9000 is about 2.5 to 3 million, see this chart:Imagine yourself what an impact this has. Each package you not send save you the network-overhead of the package itself and a lot of CPU cycles that you don't need to spend.The load average of the Linux box also decreases from an avg of 16 to 14.In terms of completed transactions on different MTU sizes within the same timeframe, the chart looks like this:To conclude this test two very high load runs are performed. Again, one with an MTU of 1500 and one with an MTU of 9000.In the charts below you will see less CPU consumption when using 9000 bytes for MTU.Also less packets are sent, although I think that number is not that significant compared to the total number of packets sent.My final thoughts on this test:1. you will hardly notice the benefits of using Jumbo on a system with no stress2. you will notice the benefits of Jumbo using Frames on a stressed system and such a system will then use less CPU and will have less network overhead. This means Jumbo Frames help you scaling out better then regular frames.Depending on the interconnect usage of your applications the results may vary of course. With interconnect traffic intensive applications you will see the benefits earlier then with application that have relatively less interconnect activity.I would use Jumbo Frames to scale better, since it saves CPU and reduces network traffic and this way leaves space for growth.Rene KundersmaOracle Expert Services, The Netherlands

At the moment for a customer I am investigating whether it's a good idea to use "Jumbo Frames" within a RAC environment. In the next couple of days I will post some results here. Until that time, I...

Migrating to Exadata / HP Oracle DB Machine

IntroPresuming you have to migrate your data to Exadata and you already did your capacity planning.Then, probably, your next step is plan how to actually move your existing database to Exadata / HP Oracle DB-Machine (from now on called Exadata in this blog). For this scenario, there are some small things to take care of. My personal experience is that, preparing such a migration is all about knowing Oracle's MAA strategy. With that knowledge you know what options you have and don't have.Off-line or On-lineFirst of al one has to decide whether the migration is 'off-line' or on-line.'On-line' in this case is a matter of a couple of minutes downtime, you always have to switch from source database 'A' to target database 'B'. Consider this time, for example, as the time it takes to switch over to a standby database.'Off-line' is the most comfy solution, but only of course when the off-line window is large enough to migrate your data. In case of migrations to Exadata the data bulk you have to migrate will probably be 'large', so consider yourself lucky if you have a 'off-line' option and a window large enough to do the action.For Exadata it is recommend to have an extent size that is a multiple of 4M. The ASM AU size should be 4M also. This is to make sure that at database level (for the extents) and at ASM level (for the AU's) at least 4M of contiguous chunks will be read. This is required to make Exadata perform best.So, if you want to stick to this recommendation, you have to check the current size of your AU's and extents. When you find out the size is not the recommend size then choice number two pops-up: will you do a 'logical' migration and stick to the recommendation or not and do a 'physical' migration. Of course, when your source system 'A' is not on ASM there isn't much AU's to check.For the extents sizes the query should look like this:select segment_name,from dba_exentswhere bytes < (4*1024*1024)and owner = 'the name of the schema you are migrating';If the extents already have the recommended size, you of course still can choose for a logical migration, but my personal feeling would be to do a physical migration in that case. Physical migrationPhysical migration to me means 'transferring the datafiles from platform A to platform B'. This situation leads us to new choices, like the choice of how to transfer the datafiles from A to B. This depends because platform A en B can both have a different architecture. As we know Exadata/DB-Machine is based on Linux on Intel X86-64 and the source can be RISC (Sun Sparc, IBM P-series) for example (big endian). (where Intel and Itanium is little endian)Fortunately Oracle has some good answers for this kind of questions:- transportable tablespaces- DBMS_STREAMS_TABLESPACE_ADM.PULL_TABLESPACESTo a certain level physical migrations can be done 'on-line' (or at least with as less downtime as possible). Think of dataguard for example.As a last step; which solution you choose also depends on the infrastructure you are working in. Question you may ask yourself are:- Do I have staging space on the source system- Do I have network attached storage (NAS) available that I can use.- Can this NAS be connected to source and target ?- Are platform A en B actually on the same network anyway and- If on the same network: can I safely use that network to tranfer my terabytes of data (not hurting the performance of other systems)These are all realistic questions you have to deal with when you are talking about migrations.Logical migrationYou have to investigate into available solutions for logical migrations as well.Before I continue I should first explain what my definition of logical migration is.To me logical migration is 'exporting' the data from the source and 'importing' that into the target. Export can be any kind of tool such as: datapump, exp or even SQL*Loader or CTAS over a db-link. A 'logical' export can even be done 'on-line' if you would consider solutions like for example logical Standby database (same endian) or Oracle Streams (endian independent). 'Logical' migrations often need 'staging' (a location where you temporary dump you data). Note that datapump exports/imports can be done over the network without dumping anything to file.As said, if you set yourself the target to change the extent sizes to the recommended values, you automatically end up in a logical migration if you not already have extents of this size.There may be even more things to consider. For example.- you need to build more then one replica of the source database on Exadata.- you want to put as less load on the source system as possible because it's a highly critical production system.When dealing with these kind of questions always know that Exadata can do the job quicker, because it so powerful.Say for instance: you may realize exporting a 4TB database with datapump will cause to much load on the prodution system. But you still want to do a logical migration. For these kind of questions my answer would be to just transfer the datafiles to Exadata (and endian convert them if needed) and perform the export (expdp) and import (impdp) there. You probably don't even need to transfer the files, perhaps you can retrieve them from backup.Exporting data with datapump is something you prefer to do in a read consistent way. For that think of the expdp arguments: - flashback_time - undo_rententionAnd undo guarentee on tablespace level.Note 1. Note that exporting lobs in a read consistent way can not be accomplished only by tuning the undo retention of the database. For lobs undo is a property of the column. In order to make sure the retention of these objects is set well you may have to alter your table first. For more information on this see "The application developers guide - Large objects"Note 2. transferring or exporting indexes to Exadata seems rather useless to me. First, you have to consider yourself if the index is still needed in the first place. And if so, why not just recreate it on the target, since Exadata has probably got the muscles for it to recreate it in a small fraction of the time it took on your source database !SummaryMigrations to Exadata are not more complicated then normal migrations. When dealing with large amount of data one should always think of a strategy before htting the keybord. As production environments have their limitations it proofed to be helpful to workout alternative scenarios as well. With expdp, transportable tablespaces, transportable databases, Streams, physical, logical standby databases, Oracle has all thinkable options available to migrate to this wonderful piece of database !Helpful documents: - Technical White Paper: Best Practices for Migrating to HP Oracle Exadata Storage Server- Oracle Database High Availability Documentation - Features and Best Practices Rene KundersmaOracle Expert Services, The Netherlands

Intro Presuming you have to migrate your data to Exadata and you already did your capacity planning. Then, probably, your next step is plan how to actually move your existing database to Exadata / HP...

'Virtual Metal' Provisioning with Oracle VM and PXE

Basis for Bare Metal Provisioning (BMP) in EMGC 10.2.0.5 is as mentioned in an earlier blog entry "PXE boot". This blog entry describes how to setup PXE boot (TFTP and DHCP) for a para-virtualised guests. This allows you to automatically install virtualised guests by kickstart file.By the way, in this setup I am on OEl 5U2 x86, if you want to reproduce for say x86_64, you may need other packages.Below are my notes of the setup:- install dhcp-3.0.5-18.el5- install tftp-0.42-3.1.0.1 (we need this one later a required package for pypxeboot)- install tftp-server-0.42-3.1.0.1After installation of these packages, we begin with the configuration of dhcp in /etc/dhcpd.conf.As this is just a test I am not using all options for DHCP.Be care full if you test this, DHCP be working too good...## DHCP Server Configuration file.# see /usr/share/doc/dhcp*/dhcpd.conf.sample #ddns-update-style none;allow booting; allow bootp; subnet 192.168.200.0 netmask 255.255.255.0 { option routers 192.168.200.1; option subnet-mask 255.255.255.0; option nis-domain "nl.oracle.com"; option domain-name "nl.oracle.com"; option domain-name-servers 192.135.82.60; default-lease-time 60; max-lease-time 60; next-server 192.168.200.173; filename "/pxelinux.0"; host RK{ hardware ethernet 00:16:3e:62:39:d3; fixed-address 192.168.200.177; }}As you can see I specified subnet, netmask, domain-name and details for the host called "RK". Details are: name, mac and ip address.The purpose of the "next-server" is to specify the name (or ip) of the tftp-server. It makes sense to put DHCP and TFTP server on the same box. In order to (re)start dhcp: service dhcpd restart After setting up DHCP, TFTP needs to be setup. This is just a matter of enabling the service in inetd. Set disable = no in the file /etc/xinetd.d/tftp. After this, restart service xinetd.Pxeboot files need to be copied to /tftpboot on the tftp-server:cp /usr/lib/syslinux/pxelinux.0 /tftpboot/cp /usr/lib/syslinux/mboot.c32 /tftpboot/From your OEL distribution, copy the boot-installation files:cp $MOUNT_OEL_DISTR/images/xen/* /tftpboot/Create a PXE configuration file for the guest you want to start:[root@gridnode03 pxelinux.cfg]# gethostip -x 192.168.200.177C0A8C8B1So for a guest with ip-number 192.168.200.177 we need to put the details for the PV-PXE installation into /tftpboot/pxelinux.cfg/C0A8C8B1[root@gridnode03 ~]# cat /tftpboot/pxelinux.cfg/C0A8C8B1 default linuxprompt 1timeout 120label linux kernel vmlinuz append initrd=initrd.img lang=en_US keymap=us \ ks=nfs:192.168.200.200:/vol/vol1/distrib/linux32/workshop-ovs/oel/OEL5U2/ks.cfg \ ksdevice=eth0 ip=dhcpYou can see:- my OEL kickstart-file is on NFS (as my installation)- the ip number is obtained by ip using eth0I created my kickstart from an existing OEL installation.With the help of the command system-config-kickstart --generate I re-generated it.After this, I had to modify some bits about installation media (from cdrom to nfs).Specifics for my kickstart file hereSee the Redhat site for all options of kickstart.Before I could start a vm guest I also, had to:- install pypxeboot and- install udhcp-0.9.8-1usermac Then, created a vm configuration file:[root@nlhpblade07 pxe]# cat rk.cfg name = "RK"memory = "1024"disk = [ 'file:/OVS/running_pool/pxe/system.img,xvda,w',]vif = [ 'mac=00:16:3e:62:39:d3,bridge=xenbr0', '', ]vfb = ["type=vnc,vncunused=1,vnclisten=0.0.0.0"]#bootloader="/usr/bin/pygrub"bootloader="/usr/bin/pypxeboot"bootargs=vif[0]vcpus=1on_reboot = 'restart'on_crash = 'restart'Before I could start the VM, the 'disk' (image) had to be in place:[root@nlhpblade07 pxe]# dd if=/dev/zero of=system.img bs=1M count=80008000+0 records in8000+0 records out8388608000 bytes (8.4 GB) copied, 165.725 seconds, 50.6 MB/s[root@nlhpblade07 pxe]# So, after starting, remember that the third console of the installation enables you to see what is going on during the run of the anaconda installation procedure: After installation and before the reboot the vm-config file had to be modified and looks like this:[root@nlhpblade07 pxe]# cat rk.cfgname = "RK"memory = "1024"disk = [ 'file:/OVS/running_pool/pxe/system.img,xvda,w',]vif = [ 'mac=00:16:3e:62:39:d3,bridge=xenbr0', '', ]vfb = ["type=vnc,vncunused=1,vnclisten=0.0.0.0"]bootloader="/usr/bin/pygrub"vcpus=1on_reboot = 'restart'on_crash = 'restart'After a successful installation the OS is setup and ready to be used:Rene KundersmaOracle Expert Services, The Netherlands

Basis for Bare Metal Provisioning (BMP) in EMGC 10.2.0.5 is as mentioned in an earlier blog entry "PXE boot".This blog entry describes how to setup PXE boot (TFTP and DHCP) for a para-virtualised...

Enterprise Manager 10.2.0.5 has Oracle VM Manager onboard

Today, I have tested the new Oracle VM Management capabilities that Enterprise Manager Grid Control (EMGC) 10.2.0.5 has.The existing Oracle VM Manager, is rather quick and light weight, so you will probably think why do we need EMGC for that.Well, think about the following advantages:- All in one management console- EMGC runs also on Windows, so now you can manage Oracle VM from windows, which was not possible before- You can set thresholds- Use the Single Sign On that EMGC has- Use the security features EMGC has- Setting notifications and alerts so that admins can manage on exception basis.- Context based flows (even if manual currently). For example, change the vCPU on para-virtualized linux guests if database diagnostics (ADDM) suggests additional CPU- Specific EM features like Configuration compare, search, policies for host and hypervisor. The policies can span multiple tiers and can be user defined. Example, do not run mixed workloads (database and middleware) on the same Hypervisor.- Guest patching with integration with Unbreakable Linux Network (ULN)Some quick requirements had to be filled in before I could begin:- install a one-off patch to the OMS (patch 8244731)- install VnCViewer into $ORACLE_HOME/j2ee/OC4J_EM/applications/em/em- also had to be sure Oracle VM Agent was updated to version 2.2-70 or higher.This all went smoothly. Please note that it makes sense to install the patch and the VnCViewer at the same time when the OMS is down.Please read note 781879.1 for full details (as information may change and there is only one singe source of truth)First I had to create a server pool. After creating a server pool, it was just a matter of making the Oracle VM Server aware of the Vm images I already had on disk.And exactly for this was an option, which you can see in the capture below.This is the place where you can see all the VM Images discovered in your virtual server pool.After this, it is just a matter of selecting the images and "importing" them into your vm "admin" page.From here you can do the usual; start, stop, pause, suspend, clone and get to the console.To me this seems a new step taken for EMGC to be the complete management framework.Rene KundersmaOracle Expert Services, The Netherlands

Today, I have tested the new Oracle VM Management capabilities that Enterprise Manager Grid Control (EMGC) 10.2.0.5 has. The existing Oracle VM Manager, is rather quick and light weight, so you will...

Provisioning your GRID with Oracle VM Templates

Introduction (Chapter 1)Linux node installation and configuration (virtualized or not) for an Oracle Grid environment can be done on various ways. Of course, one could do this all manually, but for the larger environments this would of course be undo able. Also, you want to make sure each installation has the same specifications, and you want to be sure human errors that may occur during the installation are brought back to a minimum.This blog entry will have chapters in which all details of an automated Oracle VM cloning process will be described. The setup as described below is used to prepare education environments. It will also work for proof of concept envrionments and most parts of it may be even usable in your own Grid deployment strategy.The setup described allows you to setup an GRID environments that students can use to learn (for instance) how to install RAC, configure DataGuard, work with Enterprise Manager Grid Control. I can also be used to learn students how to work with Swingbench or FCF all within their own infrastructure. This virtualized solution help to quickly setup, repair, catch-up, restore and adapt the setup. It will save your IT department costs on hardware and storage and it will save you lots of time.The pictures on this page are best viewed with Firefox. Bare metal provisioningWithin the Oracle Grid, Oracle Enterprise Manager Grid Control release 10.2.0.4 with kickstart and PXE-boot is used more often these days as a way to do a so called "bare metal" installation of the OS:After this bare metal installation "post configuration scripts" took care of the node specific settings. Even with the use of Oracle Virtual Machines on top of such a node, the kickstart procedure can still be used; without too much effort a PXE-boot configuration for virtualized guests can be setup.This way of "bare metal installation" or better "virtual metal installation" by PXE-boot for VM Guests is a nice solution, which I will describe one day. But why would one do a complete installation for each VM while each VM differs only on a couple of configuration files ? This blog entry explains how to use an Oracle VM template to provision Virtual Guest Operating Systems for in a Grid situation.For educational purposes, where classes with a lot of students have to work each with their own Grid environment, a procedure is worked out to provision a blade system with operating systems and software, Grid ready, all based on Oracle templates. As said, more options are possible, this is how my solution works, it may work for you also.1. An example OS configuration is provided (node specific configuration files). From that template files a VM Guest specific configuration is generated automatically. This configuration describes settings for hostname, ipnumbers, etc. 2. A vm template (image) is provided.By automating the two steps above, one can easily and quickly setup Virtualized Oracle Linux Nodes, ready for RAC.The next chapter will be about the configuration templates and the cloning processThe process (Chapter 2)With this configuration templates as described earlier, "configuration clones" can be made.In this example I am using HP blade technology. On each blade six VMs will be running. For each blade and for each VM running on top of that the configuration files are generated. It makes sense to define configuration templates. With the use of scripts you could use these templates and generate configuration files for each specific vm.With a VM template in one hand, and an automatically generated set of configuration files in the other you can quickly build, or rebuild the infrastructure over and over again. Even if you need to make changes that reflect all vm's, they can be rolled out quite quickly.As said, this solution is extremely useful for education purposes, or situations where you have to provide lots of VM guests ready to be used instantly. Possible other uses are in proof of concept environments.In short the work flow of the cloning process looks like the following:1. A default virtual machine image is copied over2. Configuration files for the VM are generated, based upon the blade number and vm number and purpose of the VM3. The VM image is "mounted" and configuration files are overwritten with the generated configuration files. Also binaries (other programs) are put in place4. The VM image is unmounted and if needed "file based shared storage" is created. 5. The VM boots for the first time, ready to use immediately, totally pre-configuredThe concept itself can of course also be used for the Linux provisioning of your virtualized infrastructure as an alternative to bare metal provisioning.The next chapter will describe the hardware used and the chosen storage solutions for this example. Hardware used (Chapter 3)As discussed in the previous chapter, this project is build on HP blade technology.The solution described is of course independent of the hardware chosen.However, in order to describe the complete setup this chapter is here to describe the hardware used.This blade enclosure (C3000) has eight blades, each blade has: - two nics (broadcom) - two hba's (qlogic) - 16 GB of RAM - two quad core Intel Xeon processors Storage to the blades is made available by NFS and Fiber ChannelThe NFS share is used to provide the VM template that will be used as source.The same NFS share is also available to the VM guests in order to provide the guests the option to install software from a shared location. The SAN Storage comes from an HP MSA. This MSA devices are used for OCFS2. This is where the VM images files will be placedEach blade is available by a public network interface.Also a private network is setup as interconnect network for OCFS2 between the blades.For each blade the architecture be equal to the diagram below.VM distribution (Chapter 4)As said in an earlier chapter, each blade has 16GB RAM, so this is enough to run at least 6 VMs of 2GB RAM each. The purpose is to have: - 3 vms for Real Application Clusters (RAC) (11.1.0.7 CRS/ASM/RDBMS)- 1 vm for Dataguard (11.1.0.7 ASM/RDBMS)- 1 vm to run swingbench and demo applications- 1 vm to run Enterprise Manager grid Control (EMGC).This will look this way:As each blade has 146 GB local storage, there is room to have some VM's on local disks. Since, there is no intention to live migrate these nodes they can be put on a non-shared location.VM number six (EMGC) is too big to fit next to the other VMs on local storage. For reason a shared OCFS mount is made.Each VM uses the Oracle VM provided location for the VMs (/OVS/running_pool)With symbolic links the storage for the EMGC vm is brought to the OCFS2 shared disk: GRIDNODE09 -> /OVS_shared_large/running_pool/oemgc/nlhpblade07By default OCFS2 allows four nodes to concurrently mount OCFS2 filesystem. In order to mount the OCFS2 filesystem on all blades concurrently you have to specify the –N X argument with the execution of mkfs where X is the max. number of nodes that will concurrently mount the OCFS filesystem ever.mkfs.ocfs2 -b 4K -C 32K -N 8 -L ovmsdisk /dev/sdb1PV Templates (Chapter 5)Before doing any specific VM changes, first a template is chosen, in this case Oracle Enterprise Linux 5 update 2 (OEL5U2). This is an Oracle VM template downloaded from OTN.Our template is a para-virtualized template, based on a 32bit architecture.To remind you, this is how the para-virtualized architecture looks:b.t.w. para-virtualized kernels often work faster then hardware virtualized guests.Please see this link for more information on hardware v.s. para-virtualized guestsAs part of the procedure described, the template will be copied over six times to each blade. In order to use the VMs on a specific blade for a specific purpose configuration files must be made. The next chapter describes how this works.VM Specific files and clone procedure (Chapter 6)Each virtualized guest has a small set of configuration files that are specific for that OS.Typically these files exists outside of the guest (vm.cfg) and inside the guest.Specific files inside the vm: - /etc/sysconfig/network-scripts/ifcfg-eth*- /etc/sysconfig/network- ssh configuration filesSpecific files outside the vm:- vm.cfgFor VMs running on the same blade (and being part of the same 'grid') there are also files in common:- nsswitch.conf- resolv.conf- sudoers- sysctl.conf- hostsThe files mentioned above need to be changed. This is because of the fact each machine needs it's own NIC's with specific MAC Addresses and it's own ip-numbers.Of course, within a grid (on a blade) each VM has to have a unique name. In order to make sure unique MAC addresses will be generated, one has to setup standards.For the MAC addresses, the following formula is used: 00:16:3E:XD:0Y:0Z, where:X: the number of the bladeY: the number of the VM,Z: the number of the NIC within that VM.Host names will be used multiple times (but not within the same grid), the only thing that needs to change are the corresponding ip-numbers, these must be unique across the grids.For example, the MAC address for the second NIC on the third VM on blade 7 would look like: HWADDR=00:16:3E:7D:03:02The same strategy is used to determine the ip-numbers to be used: - For the public network 192.168.200.1XY is used.- For the internal network 10.0.0.1XY is used- For the vip 192.168.200.XY is used.Where:X: the number of the BladeY: the number of the VMFor example:- the public ip-number of node 3 on blade7 would be: 192.168.200.173- the private ip-number of node 3 on blade7 would be: 10.0.0.173- the virtual ip-number of node 3 on blade7 would be:192.168.200.73So, from here, as long as you know for which blade and for which VM you will be generating the configuration, you can script that:[root@nlhpblade07 tools]# ./clone_conf.sh nlhpblade01Copying config files from /OVS_shared_large/conf/nlhpblade07 to /OVS_shared_large/conf/nlhpblade01...Performing config changes specific to the blade and the VM...# nlhpblade01 - GRIDNODE01 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE01/ifcfg-eth0# nlhpblade01 - GRIDNODE01 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE01/ifcfg-eth1# nlhpblade01 - GRIDNODE01 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE01/network# nlhpblade01 - GRIDNODE01 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE01/vm.cfg# nlhpblade01 - GRIDNODE02 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE02/ifcfg-eth0# nlhpblade01 - GRIDNODE02 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE02/ifcfg-eth1# nlhpblade01 - GRIDNODE02 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE02/network# nlhpblade01 - GRIDNODE02 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE02/vm.cfg# nlhpblade01 - GRIDNODE03 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE03/ifcfg-eth0# nlhpblade01 - GRIDNODE03 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE03/ifcfg-eth1# nlhpblade01 - GRIDNODE03 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE03/network# nlhpblade01 - GRIDNODE03 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE03/vm.cfg# nlhpblade01 - GRIDNODE04 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE04/ifcfg-eth0# nlhpblade01 - GRIDNODE04 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE04/ifcfg-eth1# nlhpblade01 - GRIDNODE04 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE04/network# nlhpblade01 - GRIDNODE04 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE04/vm.cfg# nlhpblade01 - GRIDNODE05 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE05/ifcfg-eth0# nlhpblade01 - GRIDNODE05 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE05/ifcfg-eth1# nlhpblade01 - GRIDNODE05 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE05/network# nlhpblade01 - GRIDNODE05 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE05/vm.cfg# nlhpblade01 - GRIDNODE09 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE09/ifcfg-eth0# nlhpblade01 - GRIDNODE09 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE09/network# nlhpblade01 - GRIDNODE09 - /OVS_shared_large/conf/nlhpblade01/GRIDNODE09/vm.cfgPerforming node common changes for the configuration files...# nlhpblade01 - GRIDNODE09 - /OVS_shared_large/conf/nlhpblade01/common/cluster.conf# nlhpblade01 - GRIDNODE09 - /OVS_shared_large/conf/nlhpblade01/common/hosts[root@nlhpblade07 tools]# 'mounting a vm' (Chapter 7)Now that we generated the node specific configuration files and copied the basic template we are ready to modify the OS before even booting it. What will happen after 'mounting' the VM image file is that the generated configuration will be copied over into the VM.As said, at this moment the VM is an image file, for example /OVS/running_pool/GRIDNODE01/system.img. XEN will setup a loop in order to boot the OS from that image.We do kind of the same in order to change the OS before we boot it:First, the losetup command is used to associate a loop device with the file.A loop device, is a pseudo-device that makes a file accessible as a block device.[root@nlhpblade07 GRIDNODE03]# losetup /dev/loop9 system.imgNow we have mapped the image file to a block device, we want to see the partitions on that.For this we use the command kpartx. Kpartx creates device maps from partitioned tables.Kpart is part of device-mapper multipath[root@nlhpblade07 GRIDNODE03]# kpartx -a /dev/loop9So, lets see what partitions device-mapper has for us:[root@nlhpblade07 GRIDNODE03]# ls /dev/mapper/loop9*/dev/mapper/loop9p1 /dev/mapper/loop9p2 /dev/mapper/loop9p3kpartx found three partitions and told DM there are three partitions available. Let's see if we can identify the types:[root@nlhpblade07 GRIDNODE03]# file -s /dev/mapper/loop9p1/dev/mapper/loop9p1: Linux rev 1.0 ext3 filesystem dataThis is probably the /boot partition of the vm.[root@nlhpblade07 GRIDNODE03]# file -s /dev/mapper/loop9p2/dev/mapper/loop9p2: LVM2 (Linux Logical Volume Manager) , UUID: t2SAm03KoxfUcCOS3OYmsXf9ubqcy9qThis maybe the root or the swap partition[root@nlhpblade07 GRIDNODE03]# file -s /dev/mapper/loop9p3/dev/mapper/loop9p3: LVM2 (Linux Logical Volume Manager) , UUID: j2U7KUWen1ePjDvm4hTclZvA5YJyvl9[root@nlhpblade07 GRIDNODE03]# fdisk -l /dev/mapper/loop9p2This may also be the root or the swap partitionSo, in order to make a better guess in finding the root partition, let's see what the sizes are:[root@nlhpblade07 GRIDNODE03]# fdisk -l /dev/mapper/loop9p2Disk /dev/mapper/loop9p2: 13.8 GB, 13851371520 bytes255 heads, 63 sectors/track, 1684 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/mapper/loop9p2 doesn't contain a valid partition table[root@nlhpblade07 GRIDNODE03]# fdisk -l /dev/mapper/loop9p3Disk /dev/mapper/loop9p3: 5362 MB, 5362882560 bytes255 heads, 63 sectors/track, 652 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/mapper/loop9p3 doesn't contain a valid partition tableAs we can see, one partition is 5GB and the other is 13GB.Best guess would be, the 5GB partion is the swap and the 13GB partition the OS.With the command vgscan we can scan the newly 'discovered' 'disks' and search for volume groups on them:[root@nlhpblade07 GRIDNODE03]# vgscan Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2vgdisplay says we have one volume group (VolGroup00):[root@nlhpblade07 GRIDNODE03]# vgdisplay --- Volume group --- VG Name VolGroup00 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 5 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 17.84 GB PE Size 32.00 MB Total PE 571 Alloc PE / Size 571 / 17.84 GB Free PE / Size 0 / 0 VG UUID kmhYBm-Mpbv-usx2-vDur-rEVb-uP4i-kcP4fcWith the command, vgchange -a we can make logical volumes available to use for the kernel.[root@nlhpblade07 GRIDNODE03]# vgchange -a y VolGroup00 2 logical volume(s) in volume group "VolGroup00" now active[root@nlhpblade07 GRIDNODE03]# lvdisplaylvdisplay can be use to see to see the attributes of a logical volume:[root@nlhpblade07 GRIDNODE03]# lvdisplay --- Logical volume --- LV Name /dev/VolGroup00/LogVol00 VG Name VolGroup00 LV UUID B13hk3-f5qY-3gDY-Ackt-13gK-DZDc-cTWx3V LV Write Access read/write LV Status available # open 0 LV Size 14.72 GB Current LE 471 Segments 2 Allocation inherit Read ahead sectors 0 Block device 253:3 --- Logical volume --- LV Name /dev/VolGroup00/LogVol01 VG Name VolGroup00 LV UUID iEO4oG-XPMU-syWF-qupo-811i-G6Gg-QZEw5f LV Write Access read/write LV Status available # open 0 LV Size 3.12 GB Current LE 100 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:4So, now we found, (and made available to the logical volume) the root filesystem where the VM is on. Now we can mount that:[root@nlhpblade07 GRIDNODE03]# mkdir guest_local_LogVol00; [root@nlhpblade07 GRIDNODE03]# mount /dev/VolGroup00/LogVol00 guest_local_LogVol00 See the contents of the filesystem:[root@nlhpblade07 GRIDNODE03]# cd guest_local_LogVol00/[root@nlhpblade07 guest_local_LogVol00]# ls -latotal 224drwxr-xr-x 26 root root 4096 Jan 14 2009 .drwxr-xr-x 3 root root 4096 Oct 22 22:30 ..-rw-r--r-- 1 root root 0 Jul 24 05:02 .autorelabeldrwxr-xr-x 2 root root 4096 Dec 20 2008 bindrwxr-xr-x 2 root root 4096 Jun 6 11:26 bootdrwxr-xr-x 4 root root 4096 Jun 6 11:26 devdrwxr-xr-x 94 root root 12288 Jan 14 2009 etcdrwxr-xr-x 3 root root 4096 Jun 6 11:50 homedrwxr-xr-x 14 root root 4096 Dec 20 2008 libdrwx------ 2 root root 16384 Jun 6 11:26 lost+founddrwxr-xr-x 2 root root 4096 Apr 21 2008 mediadrwxr-xr-x 2 root root 4096 May 22 09:51 miscdrwxr-xr-x 3 root root 4096 Dec 20 2008 mntdr-xr-xr-x 2 root root 4096 Jun 10 11:11 netdrwxr-xr-x 3 root root 4096 Aug 21 04:11 opt-rw-r--r-- 1 root root 0 Jan 14 2009 poweroffdrwxr-xr-x 2 root root 4096 Jun 6 11:26 procdrwxr-x--- 17 root root 4096 Jan 13 2009 rootdrwxr-xr-x 2 root root 12288 Dec 20 2008 sbindrwxr-xr-x 4 500 500 4096 Jan 14 2009 scratchdrwxr-xr-x 2 root root 4096 Jun 6 11:26 selinuxdrwxr-xr-x 2 root root 4096 Apr 21 2008 srvdrwxr-xr-x 2 root root 4096 Jun 6 11:26 sysdrwxr-xr-x 3 root root 4096 Jun 6 11:33 tftpbootdrwxrwxrwt 9 root root 4096 Jan 14 2009 tmpdrwxr-xr-x 3 root root 4096 Dec 20 2008 u01drwxr-xr-x 14 root root 4096 Jun 6 11:31 usrdrwxr-xr-x 21 root root 4096 Jun 6 11:37 varAs this seems a rather easy way to mount a vm image file, it is still not something you will do very quickly for 40 VM images very quickly.For this reason, the described solution is scripted and called mount_vm.sh. This is how it works:[root@nlhpblade07 GRIDNODE05]# mount_vm.sh GRIDNODE05Starting mount...contents /etc/sysconfig/network -file of mounted node:NETWORKING=yesNETWORKING_IPV6=noHOSTNAME=gridnode05.nl.oracle.comGenerating unmount scriptTo unmount your image run /tmp/umount_GRIDNODE05.30992.sh as rootMounting finished...As you can see the images is mounted my a script and a script to unmount is automatically generated. In order to verify the right image file is mounted the contents of the file /etc/sysconfig/network is shown.'changing a vm' (Chapter 8)Now the vm image is mounted to the filesystem, we can go back to the generated config files.From here it is easy to copy over all specific configuration files to the vm.Better, would be to make a script available to do this, and that is done for this solution:[root@nlhpblade07 GRIDNODE05]# change_vm.sh GRIDNODE05If you are sure, hit Y or y to continueYContinuing...Starting config change for VM GRIDNODE05 on nlhpblade07...Copying swingbench...Changing ownership of swingbench files...Copying FCF-Java Demo...Changing ownership of FCF-Java Demo files...This vm requires pre-build file /OVS/sharedDisk/rdbms_home_11r1_01_ocfs.img as shared Oracle RDBMS HOMEFinished changing config...Now the VM is modified internally and still mounted, the unmount has to be done.This can be done by running the generated unmount script.This script was generated during the mount.[root@nlhpblade07 GRIDNODE05]# /tmp/umount_GRIDNODE05.30992.shUnmount finishedIf the unmount succeeded, you can remove this filerm: remove regular file `/tmp/umount_GRIDNODE05.30992.sh'? yAll Together (Chapter 9)In essence the procedure described above should be repeated for each VM you want to clone and change on each blade. This may already save you hours of work and reduces chances on mistakes, but still may seem a lot of steps. In order to repeat this for each blade, for each vm, from here it is just a matter of scripting.So, you could make a script, that for each blade, would do the following:Pseudo:for each blade in blade listdo Stop all vms first while vms still running do wait 10 seconds done Restore all machines (from NFS) Clone conf Mount and change all vms Start alldone For all 42 VM images the implemented version of the script above runs about four hours. After this a complete 42 node education environment is setup.Extra Options (Chapter 10)Besides changing only configuration settings on a VM as mentioned in the Change a VM chapter, other activities can also be done.For this solution the following options are also implemented.- configure vnc- configure ocfs2 within the guest, use a shared Oracle home to save space- copy and configure software (like an oracle home or swingbench)- create ASM disk files- create OCR and Voting disks- configure sudoers- provide software for EM Agent Deployment- configure sshRene KundersmaOracle Expert Services, The Netherlands

Introduction (Chapter 1)Linux node installation and configuration (virtualized or not) for an Oracle Grid environment can be done on various ways. Of course, one could do this all manually, but for...

Need shared storage fast ? use the Linux Target Framework

For all of us that need (shared) (iSCSI) storage for test or education purposes and don't want to install for example OpenFiler (which still is a great solution), there is now the Linux Target Framework (tgt). In short, tgt consists of a deamon and utilities that allow you to quickly setup (shared) storage.Tgt can be used for more, however my example is purely focused on setting up shared iSCSI storage.First, install the tgt software, this is available in Oracle Enterprise Linux 5.[root@gridnode05 tmp]# rpm -i scsi-target-utils-0.0-0.20070620snap.el5.i386.rpm Then, start the tgtd deamon.[root@gridnode05 tmp]# service tgtd startStarting SCSI target daemon: [ OK ]Export a new iSCSI target[root@gridnode05 tmp]# tgtadm --lld iscsi --op new \ --mode target --tid 2 -T 192.168.200.173:rkvolCreate storage to export from. Let's make it 100MB in size.This will be the actual storage that the initiator will see. In normal situation you should use a normal block or a lvm[root@gridnode05 tmp]# dd if=/dev/zero of=/scratch/rk.vol bs=1M count=100100+0 records in100+0 records out104857600 bytes (105 MB) copied, 0.367602 seconds, 285 MB/sAdd the "storage volume" to the target:[root@gridnode05 tmp]# tgtadm --lld iscsi --op new --mode logicalunit \--tid 2 --lun 1 -b /scratch/rk.vol Allow all initiator clients to use the target:[root@gridnode05 tmp]# tgtadm --lld iscsi --op bind --mode target --tid 2 -I ALLOn the client install the iSCSI initiator:[root@gridnode03 ~]# rpm -i /tmp/iscsi-initiator-utils-6.2.0.868-0.7.el5.i386.rpm After installation, start the service iscsi[root@gridnode03 ~]# chkconfig iscsi on[root@gridnode03 ~]# service iscsi startiscsid is stoppedTurning off network shutdown. Starting iSCSI daemon: [ OK ][ OK ]Setting up iSCSI targets: iscsiadm: No records found![ OK ]Discover the iscsi device:[root@gridnode03 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.200.175192.168.200.175:3260,1 192.168.200.173:rkvolRestart the iscsi service and notice the target coming in:[root@gridnode03 ~]# service iscsi restartStopping iSCSI daemon: /etc/init.d/iscsi: line 33: 29176 Killed /etc/init.d/iscsid stopiscsid dead but pid file existsTurning off network shutdown. Starting iSCSI daemon: [ OK ][ OK ]Setting up iSCSI targets: Logging in to [iface: default, target: 192.168.200.173:rkvol, portal: 192.168.200.175,3260]Login to [iface: default, target: 192.168.200.173:rkvol, portal: 192.168.200.175,3260]: successful[ OK ]See the block device coming in, in the messages file[root@gridnode03 ~]# tail -f /var/log/messagesMar 4 13:24:19 gridnode03 last message repeated 2 timesMar 4 13:24:19 gridnode03 iscsid: connection1:0 is operational nowMar 4 13:24:19 gridnode03 kernel: SCSI device sda: 204800 512-byte hdwr sectors (105 MB)Mar 4 13:24:19 gridnode03 kernel: sda: Write Protect is offMar 4 13:24:19 gridnode03 kernel: SCSI device sda: drive cache: write backMar 4 13:24:19 gridnode03 kernel: SCSI device sda: 204800 512-byte hdwr sectors (105 MB)Mar 4 13:24:19 gridnode03 kernel: sda: Write Protect is offMar 4 13:24:19 gridnode03 kernel: SCSI device sda: drive cache: write backMar 4 13:24:19 gridnode03 kernel: sda: unknown partition tableMar 4 13:24:19 gridnode03 kernel: sd 0:0:0:1: Attached scsi disk sdaVerify the size of the block device[root@gridnode03 ~]# fdisk -l /dev/sdaDisk /dev/sda: 104 MB, 104857600 bytes4 heads, 50 sectors/track, 1024 cylindersUnits = cylinders of 200 * 512 = 102400 bytesDisk /dev/sda doesn't contain a valid partition table[root@gridnode03 ~]# This seems a very neat utility to use in order to obtain shared storage for education, or testing purposes.Rene KundersmaOracle Expert Services, The Netherlands

For all of us that need (shared) (iSCSI) storage for test or education purposes and don't want to install for example OpenFiler (which still is a great solution), there is now the Linux...

Mass deployment for the Oracle Enterprise Manager Agent

In an Oracle Grid environment one of the first things you do once your nodes are stacked with an OS is installing the Oracle Enterprise Manager Grid Control Agent.Once this Agent is in place you can start deploying the Oracle Stack from Enterprise Manager.As always, there are multiple ways to do this. In this case I like to detail on a specific, quick and easy way: the Downloadable Agent.In short this is how it works:You have an already running EMGC (OMS) in place. This OMS provides an http url that will be used to download the Agent software. This downloading is done by a script called agentDownload.platform (where "platform" is the name of your OS, like linux). This script takes care of downloading the installations files, a "response" file by wget and automatically (silently) installing the agent.This all can be done within two minutes !The only thing one has to do is go to the OMS_HOME/sysman/agent_download directory. From here copy the script over to your node and run it. A good idea would be to have the script by default on your OS build.The script needs at least three arguments:1. the -m for for the location of the OMS (host name or ip-address)2. the -r for the port number of that OMS address3. the -b for the Oracle Base.From this point on, everything is done automatically.To give you some insight, some formatted output of one session is displayed below:[oracle@gridnode04 scratch]$ ./agentDownload -m nlgrid02 -r 4889 -b /u01/appagentDownload invoked on Tue Jan 27 19:45:25 CET 2009 with Arguments "-m nlgrid02 -r 4889 -b /u01/app"Platform=Linux.i686, OS=linuxGetPlatform:returned=0, and os is set to: linux, platform=Linux.i686Creating /scratch/agentDownload10.2.0.4.0Oui ...LogFile for this Download can be found at: "/scratch/agentDownload10.2.0.4.0Oui/agentDownload012709194525.log"Running on Selected Platform: Linux.i686Installer location: /scratch/agentDownload10.2.0.4.0OuiDownloading Agent install response file ...--19:45:25-- http://nlgrid02:4889/agent_download/10.2.0.4.0/agent_download.rspResolving nlgrid02... 192.168.200.179Connecting to nlgrid02|192.168.200.179|:4889... connected.HTTP request sent, awaiting response... 200 OKLength: 20066 (20K) [text/plain]Saving to: `agent_download.rsp'100%[===========================================================================================>] 20,066 --.-K/s in 0s 19:45:25 (126 MB/s) - `agent_download.rsp' saved [20066/20066]Finished Downloading with Status=0Downloaded response with status=0Provide the Agent Registration password so that the Management Agent can communicate with Secure Management Service.Note: You may proceed with the installation without supplying the password; however, Management Agent can be secured manually after the installation.If Oracle Management Service is not secured, agent will not be secured, so continue by pressing Enter Key.Enter Agent Registration Password: Downloading Oracle Installer ...--19:45:30-- http://nlgrid02:4889/agent_download/10.2.0.4.0/linux/oui/oui_linux.jarResolving nlgrid02... 192.168.200.179Connecting to nlgrid02|192.168.200.179|:4889... connected.HTTP request sent, awaiting response... 200 OKLength: 44236848 (42M) [application/java-archive]Saving to: `oui_linux.jar'100%[===========================================================================================>] 44,236,848 117M/s in 0.4s 19:45:31 (117 MB/s) - `oui_linux.jar' saved [44236848/44236848]Downloaded Oracle Installer with status=0Downloading Unzip Utility ...--19:45:31-- http://nlgrid02:4889/agent_download/10.2.0.4.0/linux/agent/install/unzipResolving nlgrid02... 192.168.200.179Connecting to nlgrid02|192.168.200.179|:4889... connected.HTTP request sent, awaiting response... 200 OKLength: 101448 (99K) [text/plain]Saving to: `unzip'100%[===========================================================================================>] 101,448 --.-K/s in 0.001s 19:45:31 (127 MB/s) - `unzip' saved [101448/101448]Adding execute permissions to unzip ...Downloaded UnzipUtility with status=0Verifying Installer jar ...Verified InstallerJar with status=0Unjarring Oracle Installer ...Archive: /scratch/agentDownload10.2.0.4.0Oui/oui_linux.jar creating: Disk1/stage/ creating: Disk1/stage/fastcopy/ inflating: Disk1/stage/fastcopy/setperms1.sh inflating: Disk1/stage/fastcopy/oracle.swd_Complete_1.xml inflating: Disk1/stage/fastcopy/oracle.swd_Complete_exp_1.xml Installation in progress (Tuesday, January 27, 2009 7:45:49 PM CET)............................................................... 35% Done................................................................ 70% Done.................... 81% Done.Install successfulLinking in progress (Tuesday, January 27, 2009 7:46:31 PM CET)Link successfulSetup in progress (Tuesday, January 27, 2009 7:46:47 PM CET)........... 100% Done.Setup successfulOf course, there are more options, and more to tell.For this I like to refer to OTN where excellent documentation is available.Rene KundersmaOracle Expert Services, The Netherlands

In an Oracle Grid environment one of the first things you do once your nodes are stacked with an OS is installing the Oracle Enterprise Manager Grid Control Agent. Once this Agent is in place you can...

Adding a block device online to an Oracle VM

In a previous blog posting an example has been given how to resize the root filesystem of your Oracle VM template. In a comment on this blog someone asked how to add a block device to a running Oracle VM. This block device could then be used to put the filesystem on.Here is the answer. My Oracle VM Server indicates there are 4 guests running (without Dom0):[root@ovms1 ]# xm listName ID Mem VCPUs State Time(s)Domain-0 0 667 4 r----- 1905.1oel-repos 1 1400 2 -b---- 62.1openfiler1 2 256 1 -b---- 122.5vdb11a 10 1400 1 -b---- 24.3vsupp1 6 1500 2 -b---- 984.3So, for the vdb11a we want to add a block device online. First we create a file which we will use as 'block device".[root@ovms1 ]# dd if=/dev/zero of=extradisk01.img bs=1M count=100100+0 records in100+0 records out104857600 bytes (105 MB) copied, 0.194952 seconds, 538 MB/sThen, we attach this file as a block to the VM with ID 10 (vdb11a):[root@ovms1 ]# xm block-attach 10 file:`pwd`/extradisk01.img /dev/xvdc1 wThe 'w' indicates the disk is read/write available, but other options are available, this is the syntax:xm block-attach domain-id be-dev fe-dev mode [bedomain-id]Create a new virtual block device. This will trigger a hotplug event for the guest.OPTIONS domain-id The domain id of the guest domain that the device will be attached to. be-dev The device in the backend domain (usually domain 0) to be exported. This can be specified as a physical partition (phy:sda7) or as a file mounted as loopback (file://path/to/loop.iso). fe-dev How the device should be presented to the guest domain. It can be specified as either a symbolic name, such as /dev/hdc, for common devices, or by device id, such as 0x1400 (/dev/hdc device id in hex). mode The access mode for the device from the guest domain. Supported modes are w (read/write) or r (read-only). w! will make the hypervisor think the vbd attached to multiple domains to be ok. bedomain-id The back end domain hosting the device. This defaults to domain 0.In the syslog of the VM guest you see the disk coming in:Jan 10 20:52:58 vdb11a kernel: Registering block device major 202Let's double check if this is our disk, it should be about 100M[root@vdb11a ~]# fdisk -l /dev/xvdc1 Disk /dev/xvdc1: 104 MB, 104857600 bytes255 heads, 63 sectors/track, 12 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/xvdc1 doesn't contain a valid partition table[root@vdb11a ~]# There is also an option to detach the block devicePlease note also to add the file to the vm.cfg because if you forget, the next time you wonder where the device is.Rene KundersmaOracle Expert Services, The Netherlands

In a previous blog posting an example has been given how to resize the root filesystem of your Oracle VM template. In a comment on this blog someone asked how to add a block device to a running Oracle...

Running Oracle VM Server into an Oracle VM Server

Believe it or not, there can be a valid reason to run an Oracle VM Server into an Oracle VM Server and run a Virtual Machine into that. In search of a solution people asked if this was something they could do. They also realized "could do" != "advised to do"One thing going through my mind immediately was, even if this would work, then this will probably only for education purposes etc. And that was exactly the goal...Step 1. Logged on to a Dell Precision 3400, with a Quad core cpu and 8GB RAM. This should be enough to do the job. Then booted up Oracle VM Server 2.1.2. After that, started Oracle VM Manager in one of the domains and imported the Oracle VM Server as an iso image as a resource.Step 2. Created a virtual machine with the following specs:- Ram 3GB (planning to run only one guest into this guest.)- Diskspace (enough to hold the Oracle VM Server only since I would run my iso's from the shared NFS storage)- machine name: OVSServerStep 3. After creating the VM, the installation of Oracle Virtual Server was as easy as in a normal situation. On my first Oracle VM Server this was now the status:[root@ovms1 ~]# xm listName ID Mem VCPUs State Time(s)Domain-0 0 667 4 r----- 1207.692_OVSServer 9 3000 1 r----- 2800.0oel-repos 1 1400 2 -b---- 9.1openfiler1 2 256 1 -b---- 72.2As you can see the first Oracle VM Server was named: ovms1. Step 4. Now that I have a Oracle VM Server running into an Oracle VM Server, half of the proof is done. A complete test would of course be to run a VM guest into that.So, the /OVS was then mounted on the existing NFS share that hold the "running_pool" images, so we were able to use an existing para virtualized image I often use for RAC things. [root@ovsvirt ~]# xm listName ID Mem VCPUs State Time(s)Domain-0 0 542 1 r----- 770.9vdb11a 1 1400 1 ------ 2584.5As shown above, the second Oracle VM Server hostname is called ovsvirt.Step 5. I noticed, that I could not use Hardware Virtualization Support. So until now I only tried to run para virtualized. Whit that information I was able to start the virtual guest "vdb11a" .Step 6: See if I could get CRS running on that node:[root@vdb11a bin]# /crs/bin//crs_stat -tName Type Target State Host ------------------------------------------------------------ora....C21.srv application ONLINE ONLINE vdb11a ora....C22.srv application ONLINE OFFLINE ora....NGC2.cs application ONLINE ONLINE vdb11a ora....C21.srv application ONLINE ONLINE vdb11a ora....C22.srv application ONLINE OFFLINE ora....INGX.cs application ONLINE ONLINE vdb11a ora....21.inst application ONLINE ONLINE vdb11a ora....22.inst application ONLINE OFFLINE ora....B1C2.db application ONLINE ONLINE vdb11a ora....SM1.asm application ONLINE ONLINE vdb11a ora....1A.lsnr application ONLINE ONLINE vdb11a ora.vdb11a.gsd application ONLINE ONLINE vdb11a ora.vdb11a.ons application ONLINE ONLINE vdb11a ora.vdb11a.vip application ONLINE ONLINE vdb11a ora....SM2.asm application ONLINE OFFLINE ora....1B.lsnr application ONLINE OFFLINE ora.vdb11b.gsd application ONLINE OFFLINE ora.vdb11b.ons application ONLINE OFFLINE ora.vdb11b.vip application ONLINE ONLINE vdb11a Summary. Test succeeded. We are able to run an oracle VM Server as a guest into an Oracle VM Server. And even into that Oracle VM Server, you could run Oracle RAC as a virtual guest (unsupported) without effort.Rene KundersmaOracle Expert Services, The Netherlands

Believe it or not, there can be a valid reason to run an Oracle VM Server into an Oracle VM Server and run a Virtual Machine into that. In search of a solution people asked if this was something they...

Resizing the filesystem of your Oracle VM template

To be quick on some testing you download one of the handy Oracle VM templates.But after using the template for a while you notice the root filesystem of the VM got full, what do you do ?Since this question came along for a couple of times now, one way to handle this is described below.First, find out what the current size of the root filesystem is:[root@vsupp1 ~]# df -mFilesystem 1M-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00 3844 2885 761 80% //dev/hda1 99 12 82 13% /boottmpfs 502 0 502 0% /dev/shmSo, we add a second 'disk' to the VM. In this case it is discovered as /dev/hdb. In order to use it, we put a partition on it with the label of LVM (logical volume manager): [root@vsupp1 ~]# fdisk /dev/hdbDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuilding a new DOS disklabel. Changes will remain in memory only,until you decide to write them. After that, of course, the previouscontent won't be recoverable.Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)Command (m for help): pDisk /dev/hdb: 4194 MB, 4194304000 bytes255 heads, 63 sectors/track, 509 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id SystemCommand (m for help): nCommand action e extended p primary partition (1-4)pPartition number (1-4): 1First cylinder (1-509, default 1): Using default value 1Last cylinder or +size or +sizeM or +sizeK (1-509, default 509): Using default value 509Command (m for help): tSelected partition 1Hex code (type L to list codes): 8eChanged system type of partition 1 to 8e (Linux LVM)Command (m for help): wThe partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks. Now the partition is on it, we create a physical volume on top of it:[root@vsupp1 ~]# pvcreate /dev/hdb1Physical volume "/dev/hdb1" successfully created Now, let's take a look the current volume groups we use: [root@vsupp1 ~]# vgdisplay --- Volume group --- VG Name VolGroup00 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 5.88 GB PE Size 32.00 MB Total PE 188 Alloc PE / Size 188 / 5.88 GB Free PE / Size 0 / 0 VG UUID o0hAPH-zDE1-dmlH-Ol3e-01yf-Y9fX-grxs1GSo, let's extend this volume group with the new physical volume:[root@vsupp1 ~]# vgextend VolGroup00 /dev/hdb1 Volume group "VolGroup00" successfully extendedLet's find out if the VG size changed:[root@vsupp1 ~]# vgdisplay --- Volume group --- VG Name VolGroup00 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 2 Act PV 2 VG Size 9.75 GB PE Size 32.00 MB Total PE 312 Alloc PE / Size 188 / 5.88 GB Free PE / Size 124 / 3.88 GB VG UUID o0hAPH-zDE1-dmlH-Ol3e-01yf-Y9fX-grxs1GThe VG size changed, now let's take a look at the Logical Volume (LV) size: [root@vsupp1 ~]# lvdisplay --- Logical volume --- LV Name /dev/VolGroup00/LogVol00 VG Name VolGroup00 LV UUID DAbIh5-2rnI-Scah-xhcI-BM9a-lGIL-drQQXl LV Write Access read/write LV Status available # open 1 LV Size 3.88 GB Current LE 124 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:0 --- Logical volume --- LV Name /dev/VolGroup00/LogVol01 VG Name VolGroup00 LV UUID 04Is4L-1miY-0t5W-3Fth-3EuT-eF5M-eE4DU4 LV Write Access read/write LV Status available # open 1 LV Size 2.00 GB Current LE 64 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:1So the LV (VolGroup00/LogVol00) we use is 3.88 GB. Here is the resize: [root@vsupp1 ~]# lvextend -L 7.6G /dev/VolGroup00/LogVol00 Rounding up size to full physical extent 7.62 GB Extending logical volume LogVol00 to 7.62 GB Logical volume LogVol00 successfully resizedCool, the LV is extended. Now the thing left to extend is the filesystem.With Oracle Enterprise Linux we can do this online as we go: [root@vsupp1 ~]# resize2fs /dev/VolGroup00/LogVol00 7Gresize2fs 1.39 (29-May-2006)Filesystem at /dev/VolGroup00/LogVol00 is mounted on /; on-line resizing requiredPerforming an on-line resize of /dev/VolGroup00/LogVol00 to 1835008 (4k) blocks.The filesystem on /dev/VolGroup00/LogVol00 is now 1835008 blocks long.And see, the filesystem is resized online.[root@vsupp1 ~]# df -mFilesystem 1M-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00 6944 2886 3700 44% //dev/hda1 99 12 82 13% /boottmpfs 502 0 502 0% /dev/shmRene KundersmaOracle Expert Services, The Netherlands

To be quick on some testing you download one of the handy Oracle VM templates. But after using the template for a while you notice the root filesystem of the VM got full, what do you do ?Since this...

Oracle

Integrated Cloud Applications & Platform Services