Data Mining: The Next Generation

Abstract

Data Mining (DM) has enjoyed great popularity in recent years, with advances in both research and commercialization.  The first generation of DM research and development has yielded several commercially available systems, both stand-alone and integrated with database systems; produced scalable versions of algorithms for many classical DM problems;  and introduced novel pattern discovery problems.

In recent years, research has tended to be fragmented into several distinct pockets without a comprehensive framework.  Researchers have continued to work largely within the parameters of their parent disciplines, building upon existing and  distinct research methodologies. Even when they address a common problem (for example, how to cluster a dataset) they  apply different techniques, different perspectives on what the important issues are, and different evaluation criteria.  While different approaches can be complementary, and such a diversity is ultimately a strength of the field, better  communication across disciplines is required if DM is to forge a distinct identity with a core set of principles,  perspectives, and challenges that differentiate it from each of the parent disciplines. 
Further, while the amount and  complexity of data continues to grow rapidly, and the task of distilling useful insight continues to be central, serious  concerns have emerged about social implications of DM. Addressing these concerns will require advances in our  theoretical understanding of the principles that underlie DM algorithms, as well as an integrated approach to security and  privacy in all phases of data management and analysis.    

Researchers from a variety of backgrounds assembled at Dagstuhl to re-assess the current directions of the field, to  identify critical problems that require attention, and to discuss ways to increase the flow of ideas across the different  disciplines that DM has brought together. The workshop did not seek to draw up an agenda for the field of DM. Rather,  it offers the participants’ perspective on two technical directions – compositionality and privacy – and describes some  important application challenges that drove the discussion. Both of these directions illustrate the opportunities for crossdisciplinary research, and there was broad agreement that they represent important and timely areas for further work; of course, the choice of these directions as topics for discussion also reflects the personal interests and biases of the  workshop participants.