Reverse engineering self-modifying code: Unpacker extraction

Saumya Debray, Jay Patel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

21 Scopus citations


An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or "packed") form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.

Original languageEnglish (US)
Title of host publicationProceedings - 17th Working Conference on Reverse Engineering, WCRE 2010
Number of pages10
StatePublished - 2010
Event17th Working Conference on Reverse Engineering, WCRE 2010 - Beverly, MA, United States
Duration: Oct 13 2010Oct 16 2010

Publication series

NameProceedings - Working Conference on Reverse Engineering, WCRE
ISSN (Print)1095-1350


Other17th Working Conference on Reverse Engineering, WCRE 2010
Country/TerritoryUnited States
CityBeverly, MA


  • Binary analysis
  • Malware analysis
  • Reverse engineering
  • Self-modifying code

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'Reverse engineering self-modifying code: Unpacker extraction'. Together they form a unique fingerprint.

Cite this