A fast information-theoretic approximation of joint mutual information feature selection

Heng Liu, Gregory Ditzler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Feature selection is an important step in data analysis to address the curse of dimensionality. Such dimensionality reduction techniques are particularly important when if a classification is required and the model scales in polynomial time with the size of the feature (e.g., some applications include genomics, life sciences, cyber-security, etc.). Feature selection is the process of finding the minimum subset of features that allows for the maximum predictive power. Many of the state-of-the-art information-theoretic feature selection approaches use a greedy forward search; however, there are concerns with the search in regards to the efficiency and optimality. A unified framework was recently presented for information-theoretic feature selection that tied together many of the works in over the past twenty years. The work showed that joint mutual information maximization (JMI) is generally the best options; however, the complexity of greedy search for JMI scales quadratically and it is infeasible on high dimensional datasets. In this contribution, we propose a fast approximation of JMI based on information theory. Our approach takes advantage of decomposing the calculations within JMI to speed up a typical greedy search. We benchmarked the proposed approach against JMI on several UCI datasets, and we demonstrate that the proposed approach returns feature sets that are highly consistent with JMI, while decreasing the run time required to perform feature selection.

Original languageEnglish (US)
Title of host publication2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages4610-4617
Number of pages8
ISBN (Electronic)9781509061815
DOIs
Publication statusPublished - Jun 30 2017
Event2017 International Joint Conference on Neural Networks, IJCNN 2017 - Anchorage, United States
Duration: May 14 2017May 19 2017

Publication series

NameProceedings of the International Joint Conference on Neural Networks
Volume2017-May

Other

Other2017 International Joint Conference on Neural Networks, IJCNN 2017
CountryUnited States
CityAnchorage
Period5/14/175/19/17

    Fingerprint

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Cite this

Liu, H., & Ditzler, G. (2017). A fast information-theoretic approximation of joint mutual information feature selection. In 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings (pp. 4610-4617). [7966441] (Proceedings of the International Joint Conference on Neural Networks; Vol. 2017-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IJCNN.2017.7966441