Friday Oct 18, 2013

Defining Discovery: Core Concepts

Discovery tools have had a referencable working definition since at least 2001, when Ben Shneiderman published 'Inventing Discovery Tools: Combining Information Visualization with Data Mining'.  Dr. Shneiderman suggested the combination of the two distinct fields of data mining and information visualization could manifest as new category of tools for discovery, an understanding that remains essentially unaltered over ten years later.  An industry analyst report titled Visual Discovery Tools: Market Segmentation and Product Positioning from March of this year, for example, reads, "Visual discovery tools are designed for visual data exploration, analysis and lightweight data mining."

Tools should follow from the activities people undertake (a foundational tenet of activity centered design), however, and Dr. Shneiderman does not in fact describe or define discovery activity or capability. As I read it, discovery is assumed to be the implied sum of the separate fields of visualization and data mining as they were then understood.  As a working definition that catalyzes a field of product prototyping, it's adequate in the short term.  In the long term, it makes the boundaries of discovery both derived and temporary, and leaves a substantial gap in the landscape of core concepts around discovery, making consensus on the nature of most aspects of discovery difficult or impossible to reach.  I think this definitional gap is a major reason that discovery is still an ambiguous product landscape.

To help close that gap, I'm suggesting a few definitions of four core aspects of discovery.  These come out of our sustained research into discovery needs and practices, and have the goal of clarifying the relationship between discvoery and other analytical categories.  They are suggested, but should be internally coherent and consistent.  

Discovery activity is: "Purposeful sense making activity that intends to arrive at new insights and understanding through exploration and analysis (and for these we have specific defintions as well) of all types and sources of data."

Discovery capability is: "The ability of people and organizations to purposefully realize valuable insights that address the full spectrum of business questions and problems by engaging effectively with all types and sources of data."

Discovery tools: "Enhance individual and organizational ability to realize novel insights by augmenting and accelerating human sense making to allow engagement with all types of data at all useful scales."

Discovery environments: "Enable organizations to undertake effective discovery efforts for all business purposes and perspectives, in an empirical and cooperative fashion."

Note: applicability to a world of Big data is assumed - thus the refs to all scales / types / sources - rather than stated explicitly.  I like that Big Data doesn't have to be written into this core set of definitions, b/c I think it's a transitional label - the new version of Web 2.0 - and goes away over time.

References and Resources:

Tuesday Aug 20, 2013

Discovery and the Age of Insight

Several weeks ago, I was invited to speak to an audience of IT and business leaders at Walmart about the Language of Discovery.   Every presentation is a feedback opportunity as much as a chance to broadcast our latest thinking (musicians call it trying out new material), so I make a point to share evolving ideas and synthesize what we've learned since the last instance of public dialog.  

For the audience at Walmart, as part of the broader framing for the Age of Insight, I took the opportunity to share findings from some of the recent research we've done on Data Science (that's right, we're studying data science).  We've engaged consistently with data science practitioners for several years now (some of the field's leaders are alumni of Endeca), as part of our ongoing effort to understand the changing nature of analytical and sense making activities, the people undertaking them, and the contexts in which they take place.  We've seen the discipline emerge from an esoteric specialty into full mainstream visibility for the business community.  Interpreting what we've learned about data science through a structural and historic perspective lead me to draw a broad parallel between data science now and natural philosophy at its early stages of evolution.

We also shared some exciting new models for enterprise information engagement; crafting scenarios using language of discovery to describe discovery needs and activity at the level of discovery architecture, IT portfolio planning,  and knowledge management (which correspond to UX, technology, and business perspectives) - demonstrating the versatility of the language as a source of linkage across separate disciplines.

We continue to identify new frontiers for the language of discovery - I'm looking forward to sharing some of this work soon.

Friday May 24, 2013

Big Data Is Not the Insight: Presenting the Language of Discovery in London

Last week, in a presentation titled "Big Data Is Not the Insight: The Language of Discovery" I had the opportunity to share our evolving perspective on discovery and its relationship to big data with the audience at the Enterprise Search Europe conference in London.  Our point of view is rooted in our (ongoing) deep research into discovery needs and activities in both enterprise and consumer domains, and it is always exciting to share our latest understanding and insights. 

We've published the slides and materials shared at the conference, and welcome dialog about everything we've shared; the big ideas and fundamental concepts, the detailed findings, the implications for people active in the discovery and business analytics space, our recommended best practices for creators of discovery tools and solutions, etc.

I've included the description of the presentation from the conference program to complement the slides.

Designing Effective Search and Discovery Experiences for the Enterprise, Using the Language of Discovery

The oncoming tidal wave of Big Data, with its rapidly evolving ecosystem of multi-channel information saturated environments and services, brings profound challenges and opportunities for the design of effective user experiences that UX practitioners are just beginning to engage with in a meaningful fashion. In this coming Age of Insight, 'discovery' is not only the purview of specialized Data Scientists who create exotic visualizations of massive data sets, it is a fundamental category of human activity that is essential to everyday interactions between people, resources, and environments. Search is the gateway to discovery, and thus is indispensable as a capability.

To provide architects and designers with an effective starting point for creating satisfying search and discovery experiences this session presents a simple analytical and generative vocabulary for understanding how people conduct the broad range of discovery activities necessary in the information-permeated enterprise, and defining the search experiences they need.

 Specifically, this session will present:

  • A simple, research-derived language for describing search and discovery needs and activities that spans domains, environments, media, and user types
  • Observed and reusable patterns of discovery activities in individual and collaborative settings
  • A practical model that defines actionable patterns of information engagement throughout the enterprise
  • Examples of the architecture of successful discovery experiences at small and large scales
  • A vocabulary and perspective for discovery as a critical individual and organizational capability
  • Guidance on using this vocabulary to drive large scale IT portfolio management as well as the design of individual search solutions


Exploring the emerging space of discovery interactions, analytics, and sensemaking.


« June 2016