Reverse engineering self-modifying code: Unpacker extraction

Saumya K Debray, Jay Patel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or "packed") form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.

Original languageEnglish (US)
Title of host publicationProceedings - Working Conference on Reverse Engineering, WCRE
Pages131-140
Number of pages10
DOIs
StatePublished - 2010
Event17th Working Conference on Reverse Engineering, WCRE 2010 - Beverly, MA, United States
Duration: Oct 13 2010Oct 16 2010

Other

Other17th Working Conference on Reverse Engineering, WCRE 2010
CountryUnited States
CityBeverly, MA
Period10/13/1010/16/10

Fingerprint

Reverse engineering
Malware

Keywords

  • Binary analysis
  • Malware analysis
  • Reverse engineering
  • Self-modifying code

ASJC Scopus subject areas

  • Software

Cite this

Debray, S. K., & Patel, J. (2010). Reverse engineering self-modifying code: Unpacker extraction. In Proceedings - Working Conference on Reverse Engineering, WCRE (pp. 131-140). [5645447] https://doi.org/10.1109/WCRE.2010.22

Reverse engineering self-modifying code : Unpacker extraction. / Debray, Saumya K; Patel, Jay.

Proceedings - Working Conference on Reverse Engineering, WCRE. 2010. p. 131-140 5645447.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Debray, SK & Patel, J 2010, Reverse engineering self-modifying code: Unpacker extraction. in Proceedings - Working Conference on Reverse Engineering, WCRE., 5645447, pp. 131-140, 17th Working Conference on Reverse Engineering, WCRE 2010, Beverly, MA, United States, 10/13/10. https://doi.org/10.1109/WCRE.2010.22
Debray SK, Patel J. Reverse engineering self-modifying code: Unpacker extraction. In Proceedings - Working Conference on Reverse Engineering, WCRE. 2010. p. 131-140. 5645447 https://doi.org/10.1109/WCRE.2010.22
Debray, Saumya K ; Patel, Jay. / Reverse engineering self-modifying code : Unpacker extraction. Proceedings - Working Conference on Reverse Engineering, WCRE. 2010. pp. 131-140
@inproceedings{40999aba51b842cf876e7a80d810a1b7,
title = "Reverse engineering self-modifying code: Unpacker extraction",
abstract = "An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or {"}packed{"}) form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.",
keywords = "Binary analysis, Malware analysis, Reverse engineering, Self-modifying code",
author = "Debray, {Saumya K} and Jay Patel",
year = "2010",
doi = "10.1109/WCRE.2010.22",
language = "English (US)",
isbn = "9780769541235",
pages = "131--140",
booktitle = "Proceedings - Working Conference on Reverse Engineering, WCRE",

}

TY - GEN

T1 - Reverse engineering self-modifying code

T2 - Unpacker extraction

AU - Debray, Saumya K

AU - Patel, Jay

PY - 2010

Y1 - 2010

N2 - An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or "packed") form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.

AB - An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or "packed") form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.

KW - Binary analysis

KW - Malware analysis

KW - Reverse engineering

KW - Self-modifying code

UR - http://www.scopus.com/inward/record.url?scp=78650642358&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78650642358&partnerID=8YFLogxK

U2 - 10.1109/WCRE.2010.22

DO - 10.1109/WCRE.2010.22

M3 - Conference contribution

AN - SCOPUS:78650642358

SN - 9780769541235

SP - 131

EP - 140

BT - Proceedings - Working Conference on Reverse Engineering, WCRE

ER -