Home > Research > Publications & Outputs > Empirical data analysis

Electronic data

  • 1008

    Rights statement: ©2016 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

    Accepted author manuscript, 1 MB, PDF document

    Available under license: CC BY-NC: Creative Commons Attribution-NonCommercial 4.0 International License

Links

Text available via DOI:

View graph of relations

Empirical data analysis: a new tool for data analytics

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paper

Published
Publication date9/10/2016
Host publication2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
PublisherIEEE
Pages52-59
Number of pages8
ISBN (Electronic)9781509018970
ISBN (Print)9781509018987
Original languageEnglish
EventIEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS - , Hungary
Duration: 9/10/2016 → …

Conference

ConferenceIEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS
CountryHungary
Period9/10/16 → …

Conference

ConferenceIEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS
CountryHungary
Period9/10/16 → …

Abstract

In this paper, a novel empirical data analysis approach (abbreviated as EDA) is introduced which is entirely data-driven and free from restricting assumptions and predefined problem- or user-specific parameters and thresholds. It is well known that the traditional probability theory is restricted by strong prior assumptions which are often impractical and do not hold in real problems. Machine learning methods, on the other hand, are closer to the real problems but they usually rely on problem- or user-specific parameters or thresholds making it rather art than science. In this paper we introduce a theoretically sound yet practically unrestricted and widely applicable approach that is based on the density in the data space. Since the data may have exactly the same value multiple times we distinguish between the data points and unique locations in the
data space. The number of data points, k is larger or equal to the number of unique locations, l and at least one data point occupies each unique location. The number of different data points that
have exactly the same location in the data space (equal value), f can be seen as frequency. Through the combination of the spatial density and the frequency of occurrence of discrete data points, a
new concept called multimodal typicality, τ MM is proposed in this paper. It offers a closed analytical form that represents ensemble properties derived entirely from the empirical observations of data. Moreover, it is very close (yet different) from the histograms, from the probability density function (pdf) as well as from fuzzy set membership functions. Remarkably, there is no need to perform complicated pre-processing like clustering to get the multimodal representation. Moreover, the closed form for the case of Euclidean, Mahalanobis type of distance as well as some other forms (e.g. cosine-based dissimilarity) can be expressed recursively making it applicable to data streams and online algorithms. Inference/estimation of the typicality of data points that were not present in the data so far can be made. This new concept allows to rethink the very foundations of statistical and machine learning as well as to develop a series of anomaly
detection, clustering, classification, prediction, control and other algorithms.

Bibliographic note

©2016 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.