Tuesday Apr 14, 2015

Upcoming Webinar on April 15th about Node-oracledb driver for Node.js

Update:
   Watch the recording on the Oracle Database Development Web Series YouTube channel

Tomorrow I'll be giving a webinar covering node-oracledb, the Node.js driver for Oracle database.

Date: Wednesday, April 15th
Time: 9am (San Francisco time)
Webex - No need to register. Session will be recorded.
US Toll Free Audio (1-866-682-4770), with international numbers available (Meeting ID: 8232385# & PIN: 123456#)
Speaker: Christopher Jones
Topic:

Introduction to node-oracledb: the new Node.js driver for Oracle Database

Want to write highly scalable, event driven applications? Node.js lets you do just that.

After a quick introduction to Node.js, this session dives into the features of node-oracledb, the new, open source Node.js driver for Oracle Database, which is under active development.

To join the webinar, go to Webex

The webinar is part of an ongoing weekly series of developer sessions on a variety of topics Oracle Database Development Web Series - it's worth keeping an eye on the schedule.

A quick link to the node-oracledb homepage is http://ora.cl/wHu

Saturday Mar 28, 2015

node-oracledb 0.4.2 is on NPM (Node.js driver for Oracle Database)

The 0.4.2 version of the Node.js driver for Oracle Database is out.

  • Node-oracledb is now officially on the npmjs.com repository. This simplifies the Install instructions by removing the need to manually clone or download from GitHub. Thanks to Tim Branyen for setting this up and handing over stewardship to us.

  • Metadata support was added. Column names are now provided in the execute() callback result object. See the doc example.

  • We saw a few people try to use strangely old versions of Node 0.10. I've bumped up the lower limit requirement a bit. It won't force you to use the latest Node.js 0.10 patch set but you really should keep up to date with security fixes.

    If you want to build with Node 0.12, there is a community contributed patch from Richard Natal that can be found here. This patch also allows node-oracledb to work with io.js.

  • The default Instant Client directory on AIX was changed from /opt/oracle/instantclient_12_1 to /opt/oracle/instantclient. This now matches the default of other platforms.

  • One other small change was some improvements to the Windows install documentation.

Yes, work is continuing behind the scenes on other features.

Friday Mar 13, 2015

node-oracledb 0.4.1 is on GitHub (Node.js driver for Oracle Database)

Just a few small changes in this update of the Node.js driver for Oracle Database.

  • Support for External authentication was added This closes Issue #15.

  • The isAutoCommit flags now works with query execution. This is useful in cases where multiple DML statements are executed followed by a SELECT statement. This can be used to avoid a round trip to the database that an explicit call to commit() would add.

  • Added AIX build support to package.json. Thanks to Hannes Prirschl for submitting a pull request.

  • Errors messages when using properties that are out of range have been improved

  • Numerous API doc updates. Thanks to Greg Huang, BuiltInParris, mello151, and others for instigating some of these changes.

  • Fixed a bug: When terminate() of a connection pool fails because connections have not yet been closed, subsequent use of release() to close those connections no longer gives an error "ORA-24550: Signal Received".

  • Thanks go to krishnanm86 for a code cleanup pull request.

Bigger features (e.g. LOBS and RETURNING INTO) that I know you really, really want are still being worked on - have patience!

Monday Feb 16, 2015

node-oracledb 0.3.1 is on GitHub (Node.js driver for Oracle Database)

On behalf of the development team, I have merged some new features and fixes to node-oracledb

Updates for node-oracledb 0.3.1

  • Added Windows build configuration. See Node-oracledb Installation on Windows. Thanks to Rinie Kervel for submitting a pull request, and thanks to all those that commented and tested.
  • Added Database Resident Connection Pooling (DRCP) support. See API Documentation for the Oracle Database Node.js Driver

    "Database Resident Connection Pooling enables database resource sharing for applications that run in multiple client processes or run on multiple middle-tier application servers. DRCP reduces the overall number of connections that a database must handle. DRCP is distinct from node-oracledb's local connection pool. The two pools can be used separately, or together.
  • Made an explicit connection release() do a rollback, to be consistent with the implicit release behavior.

  • Made install on Linux look for Oracle libraries in a search order:

    • Using install-time environment variables $OCI_LIB_DIR and $OCI_INC_DIR
    • In the highest version Instant Client RPMs installed
    • In $ORACLE_HOME
    • In /opt/oracle/instantclient
  • Added RPATH support on Linux, so LD_LIBRARY_PATH doesn't always need to be set. See Advanced installation on Linux

  • The directory name used by the installer for the final attempt at locating an Instant Client directory is now /opt/oracle/instantclient or C:\oracle\instantclient. This path may be used if OCI_DIR_LIB and OCI_INC_LIB are not set and the installer has to guess where the libraries are.

  • Added a compile error message "Oracle 11.2 or later client libraries are required for building" if attempting to build with older Oracle client libraries. This helps developers self-diagnose this class of build problem.

  • Fixed setting the isAutoCommit property.

  • Fixed a crash using pooled connections on Windows.

  • Fixed a crash querying object types.

  • Fixed a crash doing a release after a failed terminate. (The Pool is still unusable - this will be fixed later)

  • Clarified documentation that terminate() doesn't release connections. Doing an explicit release() of each pooled connection that is no longer needed is recommended to avoid resource leaks and maximize pool usage.

  • Updated version to 0.3.1 (surprise!)

Thursday Jan 22, 2015

A Personal Victory: Oracle Database Sample Schemas are on GitHub

For anyone who ever deleted a row from a table in Oracle's Sample HR schema and wanted it back, help is nearby. You no longer have to download the full "Oracle Database 12c Release 1 Examples" zip (499,228,127 bytes worth for the Linux bundle) and run the Oracle installer. Now you can clone our GitHub db-sample-schema repository and run the creation SQL scripts in SQL*Plus.

This new repository installs these six sample schemas:

  • HR: Human Resources
  • OE: Order Entry
  • PM: Product Media
  • IX: Information Exchange
  • SH: Sales History
  • BI: Business Intelligence

Because of the widespread use of these schemas, we did minimal changes to the bundle. The install, as it is given, installs all schemas and needs to be done on a database server since file system access is needed from the database.

But now, if you want, you can fork the repo and modify it to install just the HR schema from a client machine. Or change your fork to install the HR schema into an arbitrary user name of your choice so multiple people can test the same data set. And what about modifying the script to do DROP TRIGGER SECURE_EMPLOYEES getting rid of that annoying time-based trigger which yells 'You may only make changes during normal office hours' if you try to make changes after 6pm or on weekends? This may be a great teaching tool about triggers but not useful when you are configuring demonstrations for big conferences late into the night!

And why is this a personal victory? Because as a client tool person, how to find these schema creation scripts has irked me in the past. The HR schema replaced SCOTT/TIGER in the Oracle documentation a long time ago but was not easily available to use. I've written a lot of examples using HR but never had a good way to instruct how to install the schema. I'm glad to have helped (being partially modest here about the legal and administrative things it required) getting this small set of scripts out on GitHub. If it makes it easier for someone to talk about features or issues by reference to a common data set, then my job is done. Having the scripts readily available is also a reminder to the Oracle community to share information and knowledge efficiently. Even as we head to a world of cloneable databases and snapshots, sometimes it is just easier to run a SQL script.

This repo is a piece of a jigsaw, and it can be used where it fits. The schemas could be now considered "traditional". In future, Oracle Database teams will continue to create fresh data sets to show off newer and upcoming database features, such as these analytical-sql-examples that you might be interested in.

Tuesday Jan 20, 2015

Introducing node-oracledb - a Node.js driver for Oracle Database

When we announced we were working on a Node.js driver for Oracle Database, it was to a large, enthusiastic audience at Oracle OpenWorld. I'm excited to let you know we have just pushed node-oracledb to GitHub.

Node.js is an upcoming, open source, cross-platform runtime environment for writing server-side and networking applications in JavaScript.

What is node-oracledb?

The node-oracledb driver connects to Oracle Database to make Node.js applications fast and functional. The node-oracledb 0.2 driver preview release already supports:

Node-oracledb is an open source project maintained by Oracle under the Apache 2.0 license. Contributions can be made under The Oracle Contributor Agreement

Where can I get node-oracledb?

To install node-oracledb, clone the repo, install the free Oracle Instant Client (or use a local Oracle Database such as the free Oracle Express Edition), and use npm to install from the cloned directory. The full instructions show how to create a local Node.js instance and covers more details. The driver is currently only on GitHub.

The driver uses Oracle client libraries - the same way Oracle Database drivers for other C based scripting languages do. This means it can take advantage of the significant feature set, engineering, and testing invested in those libraries. It also allows Node.js applications to talk to multiple different versions of the database with Oracle's standard client-to-server version compatibility.

The node-oracledb driver will build on Oracle Linux, OS X and Solaris. Our current testing has focused on Oracle Linux 6. Windows support shouldn't be difficult to add - sorry it's not there yet.

How do I use node-oracledb?

A simple query example is:

var oracledb = require('oracledb');

oracledb.getConnection(
  {
    user          : "hr",
    password      : "welcome",
    connectString : "localhost/XE"
  },
  function(err, connection)
  {
    if (err) { console.error(err); return; }
    connection.execute(
      "SELECT department_id, department_name "
    + "FROM departments "
    + "WHERE department_id < 70 "
    + "ORDER BY department_id",
      function(err, result)
      {
        if (err) { console.error(err); return; }
        console.log(result.rows);
      });
  });

The output, with Oracle's HR schema, is:

$ node select.js
[ [ 10, 'Administration' ],
  [ 20, 'Marketing' ],
  [ 30, 'Purchasing' ],
  [ 40, 'Human Resources' ],
  [ 50, 'Shipping' ],
  [ 60, 'IT' ] ]

The examples directory and API documentation contain more examples.

What is the future for node-oracledb?

We are actively working on adding features to node-oracledb including Windows platform support, LOB support, batch fetching / streaming of large query result sets, and DRCP support. We are also working on things like making it available on npmjs.com for standard npm installation, and making our test suite publishable.

We look forward to your constructive feedback so we can incorporate any fixes and "must-haves" into a 1.0 release soon.

I'm looking forward to sharing more with you as the driver progresses. My colleague Dan McGhan will also be evangelizing JavaScript and Oracle at jsao.io.

node-oracledb Links

Monday Jan 19, 2015

The Easiest Way to Enable Oracle Database Application Development on Mac OS X

tl;dr Your OS X applications can connect to Oracle Database as if it was running natively on OS X if you simply run Oracle Database in a VirtualBox VM with port forwarding enabled (easy).

To work backwards through the installation process: in the Network window of the VirtualBox GUI, I enable a NAT Network adapter.

Then, under Networking -> Advanced -> Port Forwarding, I create a TCP rule with Host IP 127.0.0.1 and both the Host and Guest Port fields set to the port number that the Oracle network listener in the VM is using: the Oracle default is 1521. I leave the Guest IP field blank.

I click OK twice, dismissing the "The current port forwarding rules are not valid. None of the host or guest port values may be set to zero." dialog

Start the VM. If the DB is set to start during OS boot, you don't even need to log in.

Update: If you have Oracle Database 12c you can use the new EM Express console for DB management and monitoring. Just enable port forwarding for port 5500 and then browse (from OS X) to https://localhost:5500/em. See this article by Gerald Venzl.

The Client

Connecting to the database from OS X tools and clients uses the same forms of Oracle connect string as if the database was native on OS X. For example, using the "Easy Connect" hostname/service_name form, I can simply use 'localhost' as if the database was on OS X:

cjones@mac:~$ sqlplus cj/welcome@localhost/orcl

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 19 09:20:38 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> 

Here I was using SQL*Plus from the free, easy to install Instant Client bundle. Download the Oracle Instant Client Basic, SDK and SQL*Plus ZIP files. Unzip them, create the two symbolic links given in the install instructions, and set DYLD_LIBRARY_PATH to the Instant Client directory.

The Instant Client can be used, among other things, for building PHP, Python, Node.js and other language drivers. With each of these you would use exactly the same connect string to connect to the database.

Prebuilt VMs

There are prebuilt VMs with Oracle Database already installed, such as the Database App Development VM. Download and import it into VirtualBox.

The Database

It's easy to install your own Oracle Database "XE" Express Edition database for development. After the XE RPM is installed on Oracle Linux 5 and 6, run its simple configuration script to set up the administration passwords, the listener port (default is 1521), and whether to autostart during boot. Install XE by following the ten steps in Chapter 4 of The Underground PHP and Oracle Manual or check the official instructions.

Alternately you could install the Enterprise Edition for free ("only for the purpose of developing, testing, prototyping, and demonstrating" - read the click-through license). There are various scripts on the web to autostart at machine boot.

The OS

If you need an OS, Oracle Linux is free from our public yum server. Download the ISO, create a new VirtualBox VM and tell it where to find the ISO. Boot and follow the install prompts.

VirtualBox

VirtualBox is free from VirtualBox.org and runs on OS X, Windows, Linux and Solaris.

The only "trick" to using VirtualBox is to remember the magic cursor-releasing key, e.g. the Left Command Key on OS X. This key is useful if/when you haven't installed the VirtualBox "Guest Additions" into the VM and clicking into the VM window captures the cursor. The specific key combination is shown as a reminder on the bottom right of the containing VirtualBox VM window. Pressing it returns cursor control to the host OS. Luckily, once you install the "Guest Additions" the cursor is automatically released when you mouse out of the VM window.

How I Enable Autostarting of Oracle Database for Demonstrations and Development

Here are the steps I have been using to enable autostarting of Oracle Datbase Enterprise Edition 10g, 11g or 12c during boot time of Oracle Linux 5 and 6. I also use these steps with Oracle Linux 7. These steps are useful for the kinds of demonstration and development setups that I typically need.

These steps are not needed for Oracle XE, since its install will prompt whether to autostart the DB and will configure the system automatically.

Create a new service script

Create a file /etc/init.d/dbora using dborc

Set permissions on the script

# chmod 750 /etc/init.d/dbora

Tell Linux to autostart/stop the service

# chkconfig --add dbora
# chkconfig dbora on

Edit /etc/oratab

In /etc/oratab, change the autostart field from N to Y for any databases that you want autostarted.

Starting / Stopping the DB

The DB will start and stop at machine boot and shutdown.

Or it can be manually controlled with:

# service dbora start

and

# service dbora stop

Starting Oracle Database 12c Multitenant PDBs

To also start all pluggable databases when the container database starts, you can do:

connect / as sysdba

create or replace trigger sys.after_startup
   after startup on database
begin
   execute immediate 'alter pluggable database all open';
end after_startup;
/

Monday Nov 17, 2014

Configuring Python cx_Oracle and mod_wsgi on Oracle Linux

The Web Server Gateway Interface (WSGI) is a standardized interface between web servers and Python web frameworks or applications. Many frameworks including Django support WSGI.

This post is a brief how-to about configuring Apache's mod_wsgi with Python's cx_Oracle driver for Oracle Database. The steps are for Oracle Linux.

  1. Download Instant Client Basic & SDK ZIP files from OTN. For cx_Oracle 5.1, use the ZIPs, not the RPMs.

  2. As root, unzip the files to the same directory, e.g. /opt/oracle/instantclient_12_1:

    mkdir /opt/oracle
    cd /opt/oracle
    unzip /tmp/instantclient-basic-linux.x64-12.1.0.2.0.zip
    unzip /tmp/instantclient-sdk-linux.x64-12.1.0.2.0.zip
    
  3. Configure Instant Client:

    cd /opt/oracle/instantclient_12_1
    ln -s libclntsh.so.12.1 libclntsh.so
    
  4. Install the pip package management tool for Python by following pip.readthedocs.org/en/latest/installing.html and downloading get-pip.py. Then run:

    python get-pip.py
    
  5. Install cx_Oracle:

    export LD_RUN_PATH=/opt/oracle/instantclient_12_1
    export ORACLE_HOME=/opt/oracle/instantclient_12_1
    pip install cx_Oracle
    

    The key here is the use of LD_RUN_PATH. This obviates the need to later set LD_LIBRARY_PATH or configure ldconfig for cx_Oracle to find the Instant Client libraries. There is a cx_Oracle-specific variable FORCE_RPATH which has the same effect.

    Note the cx_Oracle installer overloads the meaning of ORACLE_HOME. This variable is not normally used with Instant Client.

    Neither ORACLE_HOME or LD_RUN_PATH need to be set at runtime.

    If you don't use LD_RUN_PATH or FORCE_RPATH during installation, you will need to make LD_LIBRARY_PATH available to the Apache process or use ldconfig to add Instant Client to the system wide library search path.

    Configuring ldconfig is an effective and commonly used solution. However it has a potential problem that if multiple Oracle products exist, with possibly differing versions of Oracle libraries on the same machine, then there might be library clashes. If you wish to use it, create a file /etc/ld.conf.so.d/oracle-instant-client.conf containing:

    /opt/oracle/instantclient_12_1
    

    Then update the linker cache by running:

    ldconfig

    Alternatively set LD_LIBRARY_PATH in Apache's environment file, /etc/sysconfig/httpd. In Oracle Linux 6 use:

    export LD_LIBRARY_PATH=/opt/oracle/instantclient_12_1
    

    In Oracle Linux 7 use:

    LD_LIBRARY_PATH=/opt/oracle/instantclient_12_1
    

    In Oracle Linux 7, don't reference variables on the right-hand side of the equals sign since they won't be expanded.

    [The Apache environment configuration file location varies between Linux distributions. On OpenSUSE see /etc/sysconfig/apache2. On Debian-based distributions look at /etc/apache2/envvars].

  6. Set any other Oracle environment variables in the Apache environment configuration file /etc/sysconfig/httpd. For example:

    NLS_LANG=GERMAN_GERMANY.AL32UTF8
    

    (Prefix any variable setting with export in Oracle Linux 6)

  7. Install mod_wsgi:

    yum install mod_wsgi
    
  8. Add this line to /etc/httpd/conf/httpd.conf:

    WSGIScriptAlias /wsgi_test /var/www/html/wsgi_test.py
    
  9. On Oracle Linux 6, start the web server with:

    service httpd start
    

    On Oracle Linux 7 use:

    systemctl start httpd.service
    
  10. Create a test file /var/www/html/wsgi_test.py that connects to your database:

    #-*- coding: utf-8 -*-
    
    def query():
        import cx_Oracle
        db = cx_Oracle.connect("hr", "welcome", "localhost/orcl")
        cursor = db.cursor()
        cursor.execute("select city from locations where location_id = 2200")
        return cursor.fetchone()[0]
    
    def wsgi_test(environ, start_response):
        output = query()
    
        status = '200 OK'
        headers = [('Content-type', 'text/plain'),
    	       ('Content-Length', str(len(output)))]
        start_response(status, headers)
        yield output
    
    application = wsgi_test
    
  11. Load http://localhost/wsgi_test in a browser. The city of the queried location id will be displayed.

That's it. Let me know how it works for you.

Information on cx_Oracle can be found here.

Information on Oracle Linux can be found here.

Information on Oracle Database can be found here.

Wednesday Oct 29, 2014

"PL/SQL: The Scripting Language Liberator" - video recording now available

Oracle University has released a video from Oracle OpenWorld of a great session by Steven Feuerstein and myself. We walked through a PHP application, showed some application tuning techniques for Oracle Database, and then looked at improving the use of Oracle Database features to aid performance and scalability, and also easily add features to the application.

The official blurb was:

PL/SQL: The Scripting Language Liberator: While scripting languages go in and out of favor, Oracle Database and PL/SQL persist, managing data and implementing business logic. This session walks through a web application to show how PL/SQL can be integrated for better logic encapsulation and performance; how Oracle's supplied packages can be used to enhance application functionality and reduce application complexity; and how to efficiently use scripting language connection and statement handling features to get better performance and scalability. Techniques shown in this session are applicable to mobile, web, or midtier applications written in languages such as JavaScript, Python, PHP, Perl, or Ruby on Rails. Using the right tool for the right job can be liberating.

The video is free for everyone. Lots of the other good content in the Oracle Learning Streams is available via subscription, if you're interested.

Friday Sep 26, 2014

Oracle OpenWorld is Upon Us! Application Developers Unite!

It's a super, extra, enormously busy time of the year at Oracle because our big Oracle OpenWorld Conference is next week. We have all been working on new products and new presentations.

I have a full mix of events, meetings, presentations and booth duty. If you're at the conference drop by to say Hi. I'll be at the Application Development booth (Left - SLD-163) in Moscone South on Monday morning, Tuesday lunchtime and Wednesday morning (plus other odd times).

Also lookout for me onstage with Tom Kyte on Monday and with Steven Feuerstein on Wednesday.

The session details, and other sessions you might find interesting are listed on my Focus On Application Development page.

If you are a developer, I recommend coming to the Programming and Scripting "Meet the Expert" session at 6pm Tuesday evening (Moscone South - 307). I cannot stress enough how valuable it is to have a dialog and relationship with the developers who create the software and APIs you use.

Thursday Apr 24, 2014

Installing PHP on Oracle HTTP Server 12c

OTN have posted an updated guide for installing PHP on Oracle HTTP Server. Check it out!

Using PHP-FPM is a great way to integrate with Oracle Fusion Middleware.

Monday Dec 02, 2013

PHP Examples in New "Oracle Linux 6 DTrace Tutorial"

My colleague, Gavin Bowe, has released a new Oracle Linux 6 DTrace Tutorial. It is available in HTML, PDF and ePub from http://docs.oracle.com/cd/E37670_01/index.html.

Chapter 3 on "Tracing User-Space Applications" has some PHP examples.

Tuesday Nov 05, 2013

Tracing Silex from PHP to the OS with DTrace

In this blog post I show the full stack tracing of Brendan Gregg's php_syscolors.d script in the DTrace Toolkit. The Toolkit contains a dozen very useful PHP DTrace scripts and many more scripts for other languages and the OS.

For this example, I'll trace the PHP micro framework Silex, which was the topic of the second of two talks by Dustin Whittle at a recent SF PHP Meetup. His slides are at Silex: From Micro to Full Stack.

Installing DTrace and PHP

The php_syscolors.d script uses some static PHP probes and some kernel probes. For Oracle Linux I discussed installing DTrace and PHP in DTrace PHP Using Oracle Linux 'playground' Pre-Built Packages. On other platforms with DTrace support, follow your standard procedures to enable DTrace and load the correct providers. The sdt and systrace providers are required in addition to fasttrap.

On Oracle Linux, I loaded the DTrace modules like:

# modprobe fasttrap
# modprobe sdt
# modprobe systrace
# chmod 666 /dev/dtrace/helper

Installing the DTrace Toolkit

I download DTraceToolkit-0.99.tar.gz and extracted it:

$ tar -zxf DTraceToolkit-0.99.tar.gz

The PHP scripts are in the Php directory and examples in the Examples directory.

Installing Silex

I downloaded the "fat" Silex .tgz file from the download page and extracted it:

$ tar -zxf silex_fat.tgz

I changed the demonstration silex/web/index.php so I could use the PHP development web server:

<?php

// web/index.php

$filename = __DIR__.preg_replace('#(\?.*)$#', '', $_SERVER['REQUEST_URI']);
if (php_sapi_name() === 'cli-server' && is_file($filename)) {
    return false;
}

require_once __DIR__.'/../vendor/autoload.php';

$app = new Silex\Application();

//$app['debug'] = true;

$app->get('/hello', function() {
        return 'Hello!';
    });

$app->run();

?>

Running DTrace

The php_syscolors.d script uses the -Z option to dtrace, so it can be started before PHP, i.e. when there are zero of the requested probes available to be traced. I ran DTrace like:

# cd DTraceToolkit-0.99/Php
# ./php_syscolors.d

Next, I started the PHP developer web server in a second terminal:

$ cd silex
$ php -S localhost:8080 -t web web/index.php

At this point, the web server is idle, waiting for requests. DTrace is idle, waiting for the probes in php_syscolors.d to be fired, at which time the action associated with each probe will run.

I then loaded the demonstration page in a browser:

http://localhost:8080/hello

When the request was fulfilled and the simple output of "Hello" was displayed, I ^C'd php and dtrace in their terminals to stop them.

DTrace output over a thousand lines long had been generated. Here is one snippet from when run() was invoked:

C    PID/TID   DELTA(us)              FILE:LINE TYPE     -- NAME
...
1   4765/4765         21   Application.php:487  func     -> run
1   4765/4765         29   ClassLoader.php:182  func       -> loadClass
1   4765/4765         17   ClassLoader.php:198  func         -> findFile
1   4765/4765         31                 ":-    syscall        -> access
1   4765/4765         26                 ":-    syscall        <- access
1   4765/4765         16   ClassLoader.php:198  func         <- findFile
1   4765/4765         25                 ":-    syscall      -> newlstat
1   4765/4765         15                 ":-    syscall      <- newlstat
1   4765/4765         13                 ":-    syscall      -> newlstat
1   4765/4765         13                 ":-    syscall      <- newlstat
1   4765/4765         22                 ":-    syscall      -> newlstat
1   4765/4765         14                 ":-    syscall      <- newlstat
1   4765/4765         15                 ":-    syscall      -> newlstat
1   4765/4765         60                 ":-    syscall      <- newlstat
1   4765/4765         13                 ":-    syscall      -> newlstat
1   4765/4765         13                 ":-    syscall      <- newlstat
1   4765/4765         20                 ":-    syscall      -> open
1   4765/4765         16                 ":-    syscall      <- open
1   4765/4765         26                 ":-    syscall      -> newfstat
1   4765/4765         12                 ":-    syscall      <- newfstat
1   4765/4765         17                 ":-    syscall      -> newfstat
1   4765/4765         12                 ":-    syscall      <- newfstat
1   4765/4765         12                 ":-    syscall      -> newfstat
1   4765/4765         12                 ":-    syscall      <- newfstat
1   4765/4765         20                 ":-    syscall      -> mmap
1   4765/4765         14                 ":-    syscall      <- mmap
1   4765/4765       3201                 ":-    syscall      -> mmap
1   4765/4765         27                 ":-    syscall      <- mmap
1   4765/4765       1233                 ":-    syscall      -> munmap
1   4765/4765         53                 ":-    syscall      <- munmap
1   4765/4765         15                 ":-    syscall      -> close
1   4765/4765         13                 ":-    syscall      <- close
1   4765/4765         34       Request.php:32   func         -> main
1   4765/4765         22       Request.php:32   func         <- main
1   4765/4765         31   ClassLoader.php:182  func       <- loadClass
1   4765/4765         33       Request.php:249  func       -> createFromGlobals
1   4765/4765         29       Request.php:198  func         -> __construct
1   4765/4765         24       Request.php:218  func           -> initialize
1   4765/4765         26   ClassLoader.php:182  func             -> loadClass
1   4765/4765         89   ClassLoader.php:198  func               -> findFile
1   4765/4765         43                 ":-    syscall              -> access
...

The output shows PHP functions being called and returning (and where they are located) and which system calls the PHP functions in turn invoked. The time each line took from the previous one is displayed in the third column.

The first column is the CPU number. In this example, the process was always on CPU 1 so the output is naturally ordered without requiring post-processing, or the D script requiring to be modified to display a time stamp.

On a terminal, the output of php_syscolors.d is color-coded according to whether each function is a PHP or system one, hence the file name.

Summary

With one tool, I was able to trace the interaction of a user application with the operating system. I was able to do this to an application running "live" in a web context.

The DTrace Toolkit provides a very handy repository of DTrace information. Even though the PHP scripts were created in the time frame of the original PHP DTrace PECL extension, which only had PHP function entry and return probes, the scripts provide core examples for custom investigation and resolution scripts. You can easily adapt the ideas and create scripts using the other PHP static probes, which are listed in the PHP Manual.

Because DTrace is "always on", you can take advantage of it to resolve development questions or fix production situations.

Friday Nov 01, 2013

DTracing a PHPUnit Test: Looking at Functional Programming

Here's a quick example of using DTrace Dynamic Tracing to work out what a PHP code base does.

I was reading the article Functional Programming in PHP by Patkos Csaba and wondering how efficient this type of programming is. I thought this would be a good time to fire up DTrace and see what is going on. Since DTrace is "always available" even in production machines (once PHP is compiled with --enable-dtrace), this was easy to do.

I have Oracle Linux with the UEK3 kernel and PHP 5.5 with DTrace static probes enabled, as described in DTrace PHP Using Oracle Linux 'playground' Pre-Built Packages

I installed the Functional Programming sample code and Sebastian Bergmann's PHPUnit. Although PHPUnit is included in the Functional Programming example, I found it easier to separately download and use its phar file:

cd ~/Desktop
wget -O master.zip \
  https://github.com/tutsplus/functional-programming-in-php/archive/master.zip
wget https://phar.phpunit.de/phpunit.phar
unzip master.zip

I created a DTrace D script functree.d:

#pragma D option quiet

self int indent;

BEGIN
{
  topfunc = $1;
}

php$target:::function-entry
/copyinstr(arg0) == topfunc/
{
  self->follow = 1;
}

php$target:::function-entry
/self->follow/
{
  self->indent += 2;
  printf("%*s %s%s%s\n", self->indent, "->", arg3?copyinstr(arg3):"", 
                            arg4?copyinstr(arg4):"", copyinstr(arg0));
}

php$target:::function-return
/self->follow/
{
  printf("%*s %s%s%s\n", self->indent, "<-", arg3?copyinstr(arg3):"", 
                            arg4?copyinstr(arg4):"", copyinstr(arg0));
  self->indent -= 2;
}

php$target:::function-return
/copyinstr(arg0) == topfunc/
{
  self->follow = 0;
}

This prints a PHP script function call tree starting from a given PHP function name. This name is passed as a parameter to DTrace, and assigned to the variable topfunc when the DTrace script starts. With this D script, choose a PHP function that isn't recursive, or modify the script to set self->follow = 0 only when all calls to that function have unwound.

From looking at the sample FunSets.php code and its PHPUnit test driver FunSetsTest.php, I settled on one test function to trace:

function testUnionContainsAllElements() {
  ...
}

I invoked DTrace to trace function calls invoked by this test with

# dtrace -s ./functree.d -c 'php phpunit.phar \
 functional-programming-in-php-master/FunSets/Tests/FunSetsTest.php' \
 '"testUnionContainsAllElements"'

The core of this command is a call to PHP to run PHPUnit on the FunSetsTest.php script. Outside that, DTrace is called and the PID of PHP is passed to the D script $target variable so the probes fire just for this invocation of PHP.

Note the quoting around the PHP function name passed to DTrace. The parameter must have double quotes included so DTrace knows it is a string.

The output is:

PHPUnit 3.7.28 by Sebastian Bergmann.

......-> FunSetsTest::testUnionContainsAllElements
  -> FunSets::singletonSet
  <- FunSets::singletonSet
  -> FunSets::singletonSet
  <- FunSets::singletonSet
  -> FunSets::union
  <- FunSets::union
  -> FunSets::contains
    -> FunSets::{closure}
      -> FunSets::contains
        -> FunSets::{closure}
        <- FunSets::{closure}
      <- FunSets::contains
    <- FunSets::{closure}
  <- FunSets::contains
  -> PHPUnit_Framework_Assert::assertTrue
    -> PHPUnit_Framework_Assert::isTrue
    <- PHPUnit_Framework_Assert::isTrue
    -> PHPUnit_Framework_Assert::assertThat
      -> PHPUnit_Framework_Constraint::count
      <- PHPUnit_Framework_Constraint::count
      -> PHPUnit_Framework_Constraint::evaluate
        -> PHPUnit_Framework_Constraint_IsTrue::matches
        <- PHPUnit_Framework_Constraint_IsTrue::matches
      <- PHPUnit_Framework_Constraint::evaluate
    <- PHPUnit_Framework_Assert::assertThat
  <- PHPUnit_Framework_Assert::assertTrue
  -> FunSets::contains
    -> FunSets::{closure}
      -> FunSets::contains
        -> FunSets::{closure}
        <- FunSets::{closure}
      <- FunSets::contains
      -> FunSets::contains
        -> FunSets::{closure}
        <- FunSets::{closure}
      <- FunSets::contains
    <- FunSets::{closure}
  <- FunSets::contains
  -> PHPUnit_Framework_Assert::assertTrue
    -> PHPUnit_Framework_Assert::isTrue
    <- PHPUnit_Framework_Assert::isTrue
    -> PHPUnit_Framework_Assert::assertThat
      -> PHPUnit_Framework_Constraint::count
      <- PHPUnit_Framework_Constraint::count
      -> PHPUnit_Framework_Constraint::evaluate
        -> PHPUnit_Framework_Constraint_IsTrue::matches
        <- PHPUnit_Framework_Constraint_IsTrue::matches
      <- PHPUnit_Framework_Constraint::evaluate
    <- PHPUnit_Framework_Assert::assertThat
  <- PHPUnit_Framework_Assert::assertTrue
  -> FunSets::contains
    -> FunSets::{closure}
      -> FunSets::contains
        -> FunSets::{closure}
        <- FunSets::{closure}
      <- FunSets::contains
      -> FunSets::contains
        -> FunSets::{closure}
        <- FunSets::{closure}
      <- FunSets::contains
    <- FunSets::{closure}
  <- FunSets::contains
  -> PHPUnit_Framework_Assert::assertFalse
    -> PHPUnit_Framework_Assert::isFalse
      -> {closure}
        -> main
        <- main
      <- {closure}
    <- PHPUnit_Framework_Assert::isFalse
    -> PHPUnit_Framework_Assert::assertThat
      -> PHPUnit_Framework_Constraint::count
      <- PHPUnit_Framework_Constraint::count
      -> PHPUnit_Framework_Constraint::evaluate
        -> PHPUnit_Framework_Constraint_IsFalse::matches
        <- PHPUnit_Framework_Constraint_IsFalse::matches
      <- PHPUnit_Framework_Constraint::evaluate
    <- PHPUnit_Framework_Assert::assertThat
  <- PHPUnit_Framework_Assert::assertFalse
<- FunSetsTest::testUnionContainsAllElements
...

Time: 1.85 seconds, Memory: 3.75Mb
OK (9 tests, 23 assertions)

The periods correspond to the successful tests before and after (and from) the test I was tracing.

You can see the function entry ("->") and return ("<-") points. Cross checking with the testUnionContainsAllElements() source code confirms the two singletonSet() calls, one union() call, two assertTrue() calls and finally an assertFalse() call. These assertions have a contains() call as a parameter, so contains() is called before the PHPUnit assertion functions are run. You can see contains() being called recursively, and how the closures are invoked.

If you want to focus on the application logic and suppress the PHPUnit function trace, you could turn off tracing when assertions are being checked by adding D clauses checking the entry and exit of assertFalse() and assertTrue().

But if you want to see all of PHPUnit's code flow, you can modify the functree.d code that sets and unsets self->follow, and instead change it to toggle the variable in request-startup and request-shutdown probes:

php$target:::request-startup
{
  self->follow = 1
}

php$target:::request-shutdown
{
  self->follow = 0
}

Be prepared for a large amount of output!

About

Tourists looking out over an Opal mine
I'm a Product Manager in Server Technologies, working on scripting languages and developer-access.
Email: christopher.jones@oracle.com
Twitter: http://twitter.com/ghrd
Links: OTN Node.js Developer Center
OTN PHP Developer Center
Book: Free PHP Oracle book

Search

Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today