Deobfuscation of virtualization-obfuscated software

A semantics-based approach

Kevin Coogan, Gen Lu, Saumya K Debray

Research output: Chapter in Book/Report/Conference proceedingConference contribution

52 Citations (Scopus)

Abstract

When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.

Original languageEnglish (US)
Title of host publicationProceedings of the ACM Conference on Computer and Communications Security
Pages275-284
Number of pages10
DOIs
StatePublished - 2011
Event18th ACM Conference on Computer and Communications Security, CCS'11 - Chicago, IL, United States
Duration: Oct 17 2011Oct 21 2011

Other

Other18th ACM Conference on Computer and Communications Security, CCS'11
CountryUnited States
CityChicago, IL
Period10/17/1110/21/11

Fingerprint

Semantics
Engineers
Static analysis
Dynamic analysis
Virtualization
Malware

Keywords

  • Deobfuscation
  • Dynamic analysis
  • Virtualization

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this

Coogan, K., Lu, G., & Debray, S. K. (2011). Deobfuscation of virtualization-obfuscated software: A semantics-based approach. In Proceedings of the ACM Conference on Computer and Communications Security (pp. 275-284) https://doi.org/10.1145/2046707.2046739

Deobfuscation of virtualization-obfuscated software : A semantics-based approach. / Coogan, Kevin; Lu, Gen; Debray, Saumya K.

Proceedings of the ACM Conference on Computer and Communications Security. 2011. p. 275-284.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Coogan, K, Lu, G & Debray, SK 2011, Deobfuscation of virtualization-obfuscated software: A semantics-based approach. in Proceedings of the ACM Conference on Computer and Communications Security. pp. 275-284, 18th ACM Conference on Computer and Communications Security, CCS'11, Chicago, IL, United States, 10/17/11. https://doi.org/10.1145/2046707.2046739
Coogan K, Lu G, Debray SK. Deobfuscation of virtualization-obfuscated software: A semantics-based approach. In Proceedings of the ACM Conference on Computer and Communications Security. 2011. p. 275-284 https://doi.org/10.1145/2046707.2046739
Coogan, Kevin ; Lu, Gen ; Debray, Saumya K. / Deobfuscation of virtualization-obfuscated software : A semantics-based approach. Proceedings of the ACM Conference on Computer and Communications Security. 2011. pp. 275-284
@inproceedings{8518e03155e843a49168f0d1e2595f81,
title = "Deobfuscation of virtualization-obfuscated software: A semantics-based approach",
abstract = "When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.",
keywords = "Deobfuscation, Dynamic analysis, Virtualization",
author = "Kevin Coogan and Gen Lu and Debray, {Saumya K}",
year = "2011",
doi = "10.1145/2046707.2046739",
language = "English (US)",
isbn = "9781450310758",
pages = "275--284",
booktitle = "Proceedings of the ACM Conference on Computer and Communications Security",

}

TY - GEN

T1 - Deobfuscation of virtualization-obfuscated software

T2 - A semantics-based approach

AU - Coogan, Kevin

AU - Lu, Gen

AU - Debray, Saumya K

PY - 2011

Y1 - 2011

N2 - When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.

AB - When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.

KW - Deobfuscation

KW - Dynamic analysis

KW - Virtualization

UR - http://www.scopus.com/inward/record.url?scp=80755169494&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80755169494&partnerID=8YFLogxK

U2 - 10.1145/2046707.2046739

DO - 10.1145/2046707.2046739

M3 - Conference contribution

SN - 9781450310758

SP - 275

EP - 284

BT - Proceedings of the ACM Conference on Computer and Communications Security

ER -