Measuring Usability with Common Industry Format (CIF) Usability Tests

Sean Rice, Manager, Applications User Experience


A User-centered Research and Design Process

The Oracle Fusion Applications user experience was five years in the making. The development of this suite included an extensive and comprehensive user experience design process: ethnographic research, low-fidelity workflow prototyping, high fidelity user interface (UI) prototyping, iterative formative usability testing, development feedback and iteration, and sales and customer evaluation throughout the design cycle. However, this process does not stop when our products are released. We conduct summative usability testing using the ISO 25062 Common Industry Format (CIF) for usability test reports as an organizational framework. CIF tests allow us to measure the overall usability of our released products.  These studies provide benchmarks that allow for comparisons of a specific product release against previous versions of our product and against other products in the marketplace.

What Is a CIF Usability Test?

CIF refers to the internationally standardized method for reporting usability test findings used by the software industry. The CIF is based on a formal, lab-based test that is used to benchmark the usability of a product in terms of human performance and subjective data. The CIF was developed and is endorsed by more than 375 software customer and vendor organizations led by the National Institute for Standards and Technology (NIST), a US government entity. NIST sponsored the CIF through the American National Standards Institute (ANSI) and International Organization for Standardization (ISO) standards-making processes.

Oracle played a key role in developing the CIF. The CIF report format and metrics are consistent with the ISO 9241-11 definition of usability: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” Our goal in conducting CIF tests is to measure performance and satisfaction of a representative sample of users on a set of core tasks and to help predict how usable a product will be with the larger population of customers.

Why Do We Perform CIF Testing?

The overarching purpose of the CIF for usability test reports is to promote incorporation of usability as part of the procurement decision-making process for interactive products. CIF provides a common format for vendors to report the methods and results of usability tests to customer organizations, and enables customers to compare the usability of our software to that of other suppliers. CIF also enables us to compare our current software with previous versions of our software.

CIF Testing for Fusion Applications

Oracle Fusion Applications comprises more than 100 modules in seven different product families. These modules encompass more than 400 task flows and 400 user roles. Due to resource constraints, we cannot perform comprehensive CIF testing across the entire product suite. Therefore, we had to develop meaningful inclusion criteria and work with other stakeholders across the applications development organization to prioritize product areas for testing. Ultimately, we want to test the product areas for which customers might be most interested in seeing CIF data. We also want to build credibility with customers; we need to be able to make the case to current and prospective customers that the product areas tested are representative of the product suite as a whole. Our goal is to test the top use cases for each product.

The primary activity in the scoping process was to work with the individual product teams to identify the key products and business process task flows in each product to test. We prioritized these products and flows through a series of negotiations among the user experience managers, product strategy, and product management directors for each of the primary product families within the Oracle Fusion Applications suite (Human Capital Management, Supply Chain Management, Customer Relationship Management, Financials, Projects, and Procurement). The end result of the scoping exercise was a list of 47 proposed CIF tests for the Fusion Applications product suite.

 Figure 1. A participant completes tasks during a usability test in Oracle’s Usability Labs

Fusion Supplier Portal CIF Test

The first Fusion CIF test was completed on the Supplier Portal application in July of 2011.  Fusion Supplier Portal is part of an integrated suite of Procurement applications that helps supplier companies manage orders, schedules, shipments, invoices, negotiations and payments. The user roles targeted for the usability study were Supplier Account Receivables Specialists and Supplier Sales Representatives, including both experienced and inexperienced users across a wide demographic range.  The test specifically focused on the following functionality and features:

  • Manage payments – view payments
  • Manage invoices – view invoice status and create invoices
  • Manage account information – create new contact, review bank account information
  • Manage agreements – find and view agreement, upload agreement lines, confirm status of agreement lines upload
  • Manage purchase orders (PO) – view history of PO, request change to PO, find orders
  • Manage negotiations – respond to request for a quote, check the status of a negotiation response

These product areas were selected to represent the most important subset of features and functionality of the flow, in terms of frequency and criticality of use by customers.

A total of 20 users participated in the usability study. The results of the Supplier Portal evaluation were favorable and exceeded our expectations.

Figure 2. Fusion Supplier Portal

Next Studies

We plan to conduct two Fusion CIF usability studies per product family over the next nine months. The next product to be tested will be Self-service Procurement. End users are currently being recruited to participate in this usability study, and the test sessions are scheduled to begin during the last week of November.


I'm trying to determine the best way for measuring the PeopleSoft User Experience. We are embarking on an upgrade and I would like to capture baseline metrics for my population on our current version first. If anyone can direct me or provide me with some questions or surveys I could use rather than re-inventing the wheel, that would be appreciated.

Posted by Steve Martin on February 20, 2015 at 09:29 AM PST #

Thanks for reaching out Steve. Check out what UX Direct offers on this (under "Deploy" for example) - if you need further guidance on PeopleSoft UX then contact the UX Direct folks.

Posted by Ultan O'Broin (Oracle Applications User Experience) on February 20, 2015 at 10:49 AM PST #

Wonderful article. I am about to embark on standing up an Enterprise UX practice at my company and found this article to be invaluable towards additional proof of the value and priority of CIF. I was wondering if you could point me to some other case studies to use as additional support to move this forward and to get the funding needed to get this off the ground. Thank you! -- Scott

Posted by Scott F on January 23, 2016 at 05:12 AM PST #

You might send these links of interest:

Emphasis on Practical Usability Research at HFES Annual Meeting in Chicago


Common Industry Format for Usability Test Reports

Lots of other resources on the Internet too. Sorry I cannot provide more insight on this.

Posted by Ultan O'Broin (Apps UX) on January 25, 2016 at 12:09 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed

The Oracle Applications User Experience (OAUX) Usable Apps in the Cloud blog.

Oracle Applictions User Experience logo

Twitter Usable Apps on Twitter

Main OAUX Website:

Usable Apps