On the Infeasibility of Modeling Polymorphic Shellcode

Yingbo Song; Michael E. Locasto; Angelos Stavrou; Angelos D. Keromytis; Salvatore Stolfo

On the Infeasibility of Modeling Polymorphic Shellcode
Song, Yingbo
Locasto, Michael E.
Stavrou, Angelos
Keromytis, Angelos D.
Stolfo, Salvatore
Computer Science
Persistent URL:
CCS '07: proceedings of the 14th ACM Conference on Computer and Communications Security: Alexandria, Virginia, USA, October 29-November 2, 2007 (New York : Association for Computing Machinery, 2007), pp. 541-551.
Polymorphic malcode remains a troubling threat. The ability formal code to automatically transform into semantically equivalent variants frustrates attempts to rapidly construct a single, simple, easily verifiable representation. We present a quantitative analysis of the strengths and limitations of shellcode polymorphism and consider its impact on current intrusion detection practice. We focus on the nature of shellcode decoding routines. The empirical evidence we gather helps show that modeling the class of self-modifying code is likely intractable by known methods, including both statistical constructs and string signatures. In addition, we develop and present measures that provide insight into the capabilities, strengths, and weaknesses of polymorphic engines. In order to explore countermeasures to future polymorphic threats, we show how to improve polymorphic techniques and create a proof-of-concept engine expressing these improvements. Our results indicate that the class of polymorphic behavior is too greatly spread and varied to model effectively. Our analysis also supplies a novel way to understand the limitations of current signature-based techniques. We conclude that modeling normal content is ultimately a more promising defense mechanism than modeling malicious or abnormal content.
Computer science
Digital signatures
Publisher DOI:
Item views
text | xml
Suggested Citation:
Yingbo Song, Michael E. Locasto, Angelos Stavrou, Angelos D. Keromytis, Salvatore Stolfo, , On the Infeasibility of Modeling Polymorphic Shellcode, Columbia University Academic Commons, .

Columbia University Libraries | Policies | FAQ