A comparative study of fairness-enhancing interventions in machine learning

Sorelle A. Friedler, Sonam Choudhary, Carlos Eduardo Scheidegger, Evan P. Hamilton, Suresh Venkatasubramanian, Derek Roth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

19 Scopus citations

Abstract

Computers are increasingly used to make decisions that have significant impact on people's lives. Often, these predictions can affect different population subgroups disproportionately. As a result, the issue of fairness has received much recent interest, and a number of fairness-enhanced classifiers have appeared in the literature. This paper seeks to study the following questions: how do these different techniques fundamentally compare to one another, and what accounts for the differences? Specifically, we seek to bring attention to many under-appreciated aspects of such fairness-enhancing interventions that require investigation for these algorithms to receive broad adoption. We present the results of an open benchmark we have developed that lets us compare a number of different algorithms under a variety of fairness measures and existing datasets. We find that although different algorithms tend to prefer specific formulations of fairness preservations, many of these measures strongly correlate with one another. In addition, we find that fairness-preserving algorithms tend to be sensitive to fluctuations in dataset composition (simulated in our benchmark by varying training-test splits) and to different forms of preprocessing, indicating that fairness interventions might be more brittle than previously thought.

Original languageEnglish (US)
Title of host publicationFAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency
PublisherAssociation for Computing Machinery, Inc
Pages329-338
Number of pages10
ISBN (Electronic)9781450361255
DOIs
StatePublished - Jan 29 2019
Event2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019 - Atlanta, United States
Duration: Jan 29 2019Jan 31 2019

Publication series

NameFAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency

Conference

Conference2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019
CountryUnited States
CityAtlanta
Period1/29/191/31/19

Keywords

  • Benchmarks
  • Fairness-aware machine learning

ASJC Scopus subject areas

  • Business, Management and Accounting(all)
  • Engineering(all)

Fingerprint Dive into the research topics of 'A comparative study of fairness-enhancing interventions in machine learning'. Together they form a unique fingerprint.

  • Cite this

    Friedler, S. A., Choudhary, S., Scheidegger, C. E., Hamilton, E. P., Venkatasubramanian, S., & Roth, D. (2019). A comparative study of fairness-enhancing interventions in machine learning. In FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (pp. 329-338). (FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency). Association for Computing Machinery, Inc. https://doi.org/10.1145/3287560.3287589