Code obfuscation against symbolic execution attacks

Sebastian Banescu, Christian S Collberg, Vijay Ganesh, Zack Newsham, Alexander Pretschner

Research output: Chapter in Book/Report/Conference proceedingConference contribution

44 Scopus citations

Abstract

Code obfuscation is widely used by software developers to protect intellectual property, and malware writers to hamper program analysis. However, there seems to be little work on systematic evaluations of effectiveness of obfuscation techniques against automated program analysis. The result is that we have no methodical way of knowing what kinds of automated analyses an obfuscation method can withstand. This paper addresses the problem of characterizing the resilience of code obfuscation transformations against automated symbolic execution attacks, complementing existing works that measure the potency of obfuscation transformations against human-assisted attacks through user studies. We evaluated our approach over 5000 different C programs, which have each been obfuscated using existing implementations of obfuscation transformations. The results show that many existing obfuscation transformations, such as virtualization, stand little chance of withstanding symbolicexecution based deobfuscation. A crucial and perhaps surprising observation we make is that symbolic-execution based deobfuscators can easily deobfuscate transformations that preserve program semantics. On the other hand, we present new obfuscation transformations that change program behavior in subtle yet acceptable ways, and show that they can render symbolic-execution based deobfuscation analysis ineffective in practice.

Original languageEnglish (US)
Title of host publicationProceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016
PublisherAssociation for Computing Machinery
Pages189-200
Number of pages12
Volume5-9-December-2016
ISBN (Electronic)9781450347716
DOIs
StatePublished - Dec 5 2016
Event32nd Annual Computer Security Applications Conference, ACSAC 2016 - Los Angeles, United States
Duration: Dec 5 2016Dec 9 2016

Other

Other32nd Annual Computer Security Applications Conference, ACSAC 2016
CountryUnited States
CityLos Angeles
Period12/5/1612/9/16

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint Dive into the research topics of 'Code obfuscation against symbolic execution attacks'. Together they form a unique fingerprint.

  • Cite this

    Banescu, S., Collberg, C. S., Ganesh, V., Newsham, Z., & Pretschner, A. (2016). Code obfuscation against symbolic execution attacks. In Proceedings - 32nd Annual Computer Security Applications Conference, ACSAC 2016 (Vol. 5-9-December-2016, pp. 189-200). Association for Computing Machinery. https://doi.org/10.1145/2991079.2991114