Deobfuscation of virtualization-obfuscated software: A semantics-based approach

Kevin Coogan, Gen Lu, Saumya Debray

Research output: Chapter in Book/Report/Conference proceedingConference contribution

62 Scopus citations

Abstract

When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.

Original languageEnglish (US)
Title of host publicationCCS'11 - Proceedings of the 18th ACM Conference on Computer and Communications Security
Pages275-284
Number of pages10
DOIs
StatePublished - Nov 14 2011
Event18th ACM Conference on Computer and Communications Security, CCS'11 - Chicago, IL, United States
Duration: Oct 17 2011Oct 21 2011

Publication series

NameProceedings of the ACM Conference on Computer and Communications Security
ISSN (Print)1543-7221

Other

Other18th ACM Conference on Computer and Communications Security, CCS'11
CountryUnited States
CityChicago, IL
Period10/17/1110/21/11

    Fingerprint

Keywords

  • Deobfuscation
  • Dynamic analysis
  • Virtualization

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this

Coogan, K., Lu, G., & Debray, S. (2011). Deobfuscation of virtualization-obfuscated software: A semantics-based approach. In CCS'11 - Proceedings of the 18th ACM Conference on Computer and Communications Security (pp. 275-284). (Proceedings of the ACM Conference on Computer and Communications Security). https://doi.org/10.1145/2046707.2046739