Code obfuscation against symbolic execution attacks

Sebastian Banescu, Christian S Collberg, Vijay Ganesh, Zack Newsham, Alexander Pretschner

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Citations (Scopus)

Abstract

Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

Original languageEnglish (US)
Title of host publicationProceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016
PublisherAssociation for Computing Machinery
Pages189-200
Number of pages12
Volume5-9-December-2016
ISBN (Electronic)9781450347716
DOIs
StatePublished - Dec 5 2016
Event32nd Annual Computer Security Applications Conference, ACSAC 2016 - Los Angeles, United States
Duration: Dec 5 2016Dec 9 2016

Other

Other32nd Annual Computer Security Applications Conference, ACSAC 2016
CountryUnited States
CityLos Angeles
Period12/5/1612/9/16

Fingerprint

Intellectual property
Semantics
Virtualization
Malware

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Cite this

Banescu, S., Collberg, C. S., Ganesh, V., Newsham, Z., & Pretschner, A. (2016). Code obfuscation against symbolic execution attacks. In Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016 (Vol. 5-9-December-2016, pp. 189-200). Association for Computing Machinery. https://doi.org/10.1145/2991079.2991114

Code obfuscation against symbolic execution attacks. / Banescu, Sebastian; Collberg, Christian S; Ganesh, Vijay; Newsham, Zack; Pretschner, Alexander.

Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016. Vol. 5-9-December-2016 Association for Computing Machinery, 2016. p. 189-200.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Banescu, S, Collberg, CS, Ganesh, V, Newsham, Z & Pretschner, A 2016, Code obfuscation against symbolic execution attacks. in Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016. vol. 5-9-December-2016, Association for Computing Machinery, pp. 189-200, 32nd Annual Computer Security Applications Conference, ACSAC 2016, Los Angeles, United States, 12/5/16. https://doi.org/10.1145/2991079.2991114
Banescu S, Collberg CS, Ganesh V, Newsham Z, Pretschner A. Code obfuscation against symbolic execution attacks. In Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016. Vol. 5-9-December-2016. Association for Computing Machinery. 2016. p. 189-200 https://doi.org/10.1145/2991079.2991114
Banescu, Sebastian ; Collberg, Christian S ; Ganesh, Vijay ; Newsham, Zack ; Pretschner, Alexander. / Code obfuscation against symbolic execution attacks. Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016. Vol. 5-9-December-2016 Association for Computing Machinery, 2016. pp. 189-200
@inproceedings{9749cec6b4924a1b8aec3bf6f18e5b95,
title = "Code obfuscation against symbolic execution attacks",
abstract = "Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.",
author = "Sebastian Banescu and Collberg, {Christian S} and Vijay Ganesh and Zack Newsham and Alexander Pretschner",
year = "2016",
month = "12",
day = "5",
doi = "10.1145/2991079.2991114",
language = "English (US)",
volume = "5-9-December-2016",
pages = "189--200",
booktitle = "Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016",
publisher = "Association for Computing Machinery",

}

TY - GEN

T1 - Code obfuscation against symbolic execution attacks

AU - Banescu, Sebastian

AU - Collberg, Christian S

AU - Ganesh, Vijay

AU - Newsham, Zack

AU - Pretschner, Alexander

PY - 2016/12/5

Y1 - 2016/12/5

N2 - Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

AB - Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

UR - http://www.scopus.com/inward/record.url?scp=85007524382&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85007524382&partnerID=8YFLogxK

U2 - 10.1145/2991079.2991114

DO - 10.1145/2991079.2991114

M3 - Conference contribution

AN - SCOPUS:85007524382

VL - 5-9-December-2016

SP - 189

EP - 200

BT - Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016

PB - Association for Computing Machinery

ER -