Binary Black-Box Attacks against Static Malware Detectors with Reinforcement Learning in Discrete Action Spaces

Mohammadreza Ebrahimi, Jason Pacheco, Weifeng Li, James Lee Hu, Hsinchun Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent machine learning- and deep learning-based static malware detectors have shown breakthrough performance in identifying unseen malware variants. As a result, they are increasingly being adopted to lower the cost of dynamic malware analysis and manual signature identification. Despite their success, studies have shown that they can be vulnerable to adversarial malware attacks, in which an adversary modifies a known malware executable subtly to fool the malware detector into recognizing it as a benign file. Recent studies have shown that automatically crafting these adversarial malware variants at scale is beneficial to improve the robustness of malware detectors. For conciseness, we refer to this process as Adversarial Malware example Generation (AMG). Most AMG methods rely on prior knowledge about the architecture or parameters of the detector, which is not often available in practice. Moreover, the majority of these methods are restricted to additive modifications that append contents to the malware executable without modifying its original content. In this study, we offer a novel Reinforcement Learning (RL) method, AMG-VAC, which extends Variational Actor-Critic (VAC) to non-continuous action spaces where modifications are inherently discrete. We evaluate the evasion performance of the proposed AMG-VAC on two reputable machine learning-based malware detectors. While the proposed method outperforms extant non-RL and RL-based AMG methods by statistically significant margins, we show that the obtained evasive action sequences are useful in shedding light on malware detectors' vulnerabilities.

Original languageEnglish (US)
Title of host publicationProceedings - 2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages85-91
Number of pages7
ISBN (Electronic)9781728189345
DOIs
StatePublished - May 2021
Event2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021 - Virtual, Online
Duration: May 27 2021 → …

Publication series

NameProceedings - 2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021

Conference

Conference2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021
CityVirtual, Online
Period5/27/21 → …

Keywords

  • adversarial malware generation
  • approximate sampling
  • binary black-box attack
  • reinforcement learning
  • static malware detection
  • variational actor-critic

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Binary Black-Box Attacks against Static Malware Detectors with Reinforcement Learning in Discrete Action Spaces'. Together they form a unique fingerprint.

Cite this