A new generic taxonomy of malware behavioural detection and removal techniques

Lee Ling Chuan, Mahamod Ismail, Chan Lee Yee, Kasmiran Jumari

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Modern malware has become a major threat to today's Internet communications. The threat can infiltrate hosts using a variety of methods, such as attacks against known software vulnerabilities, hidden functionality in regular programs, drive-by download from unsafe web sites, and so forth. Matching a file stream against a known virus pattern is a fundamental technique for detecting viruses. With the popularity and variety of malware attack over the Internet, computer virus protection companies need to constantly update new virus signatures in their virus definition databases. However, the increasing size of the signature database can only detect known virus but cannot defend against new variants of malware. In this paper, we present an overview of the detection of modern malware focuses on suspect behavioural patterns. Contrary to classical heuristic engines which focus on the detection of encrypted malware samples, we integrate a known packer detector as well as unpacking routines to circumvent the protection techniques used by most of the modern malware. We believe that many obfuscated techniques used by malware authors are available on the Internet. More precisely, the use of known packer removals would strip out the packer protection with our dedicated decryption routines. Our apprehensive program is based on the integration of both static heuristic and emulator approaches; however, they do not necessarily have to serve as a complement for each other. Static heuristic scanner involves static extraction, which is relying on byte signature to identify a dedicated viral signature. Emulator can execute the arbitrary code from the instance and would trace the instance's body code in a virtual environment. It can be used to combat any protection code, regardless of the complexity of the protection algorithm. Fragments of virus body could be detected while the execution is in a decrypted virus body. Lastly, we present experimental results that indicate our proposed technique can provide good performance against obfuscated malware. Through this study, we hope to help security researchers understand our defence approach and give some directions for future research.

Original languageEnglish
Pages (from-to)260-270
Number of pages11
JournalJournal of Theoretical and Applied Information Technology
Volume42
Issue number2
Publication statusPublished - 2012

Fingerprint

Malware
Taxonomies
Taxonomy
Virus
Computer viruses
Packers
Signature
Internet
Heuristics
Attack
Computer Virus
Virtual Environments
Scanner
Vulnerability
Virtual reality
Strip
Websites
Fragment
Engine
Complement

Keywords

  • Dynamic Analysis
  • Emulator
  • Heuristic
  • Malware
  • Static Analysis

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

A new generic taxonomy of malware behavioural detection and removal techniques. / Chuan, Lee Ling; Ismail, Mahamod; Yee, Chan Lee; Jumari, Kasmiran.

In: Journal of Theoretical and Applied Information Technology, Vol. 42, No. 2, 2012, p. 260-270.

Research output: Contribution to journalArticle

Chuan, Lee Ling ; Ismail, Mahamod ; Yee, Chan Lee ; Jumari, Kasmiran. / A new generic taxonomy of malware behavioural detection and removal techniques. In: Journal of Theoretical and Applied Information Technology. 2012 ; Vol. 42, No. 2. pp. 260-270.
@article{5736346f4c8b4044abb8a5b175918f7d,
title = "A new generic taxonomy of malware behavioural detection and removal techniques",
abstract = "Modern malware has become a major threat to today's Internet communications. The threat can infiltrate hosts using a variety of methods, such as attacks against known software vulnerabilities, hidden functionality in regular programs, drive-by download from unsafe web sites, and so forth. Matching a file stream against a known virus pattern is a fundamental technique for detecting viruses. With the popularity and variety of malware attack over the Internet, computer virus protection companies need to constantly update new virus signatures in their virus definition databases. However, the increasing size of the signature database can only detect known virus but cannot defend against new variants of malware. In this paper, we present an overview of the detection of modern malware focuses on suspect behavioural patterns. Contrary to classical heuristic engines which focus on the detection of encrypted malware samples, we integrate a known packer detector as well as unpacking routines to circumvent the protection techniques used by most of the modern malware. We believe that many obfuscated techniques used by malware authors are available on the Internet. More precisely, the use of known packer removals would strip out the packer protection with our dedicated decryption routines. Our apprehensive program is based on the integration of both static heuristic and emulator approaches; however, they do not necessarily have to serve as a complement for each other. Static heuristic scanner involves static extraction, which is relying on byte signature to identify a dedicated viral signature. Emulator can execute the arbitrary code from the instance and would trace the instance's body code in a virtual environment. It can be used to combat any protection code, regardless of the complexity of the protection algorithm. Fragments of virus body could be detected while the execution is in a decrypted virus body. Lastly, we present experimental results that indicate our proposed technique can provide good performance against obfuscated malware. Through this study, we hope to help security researchers understand our defence approach and give some directions for future research.",
keywords = "Dynamic Analysis, Emulator, Heuristic, Malware, Static Analysis",
author = "Chuan, {Lee Ling} and Mahamod Ismail and Yee, {Chan Lee} and Kasmiran Jumari",
year = "2012",
language = "English",
volume = "42",
pages = "260--270",
journal = "Journal of Theoretical and Applied Information Technology",
issn = "1992-8645",
publisher = "Asian Research Publishing Network (ARPN)",
number = "2",

}

TY - JOUR

T1 - A new generic taxonomy of malware behavioural detection and removal techniques

AU - Chuan, Lee Ling

AU - Ismail, Mahamod

AU - Yee, Chan Lee

AU - Jumari, Kasmiran

PY - 2012

Y1 - 2012

N2 - Modern malware has become a major threat to today's Internet communications. The threat can infiltrate hosts using a variety of methods, such as attacks against known software vulnerabilities, hidden functionality in regular programs, drive-by download from unsafe web sites, and so forth. Matching a file stream against a known virus pattern is a fundamental technique for detecting viruses. With the popularity and variety of malware attack over the Internet, computer virus protection companies need to constantly update new virus signatures in their virus definition databases. However, the increasing size of the signature database can only detect known virus but cannot defend against new variants of malware. In this paper, we present an overview of the detection of modern malware focuses on suspect behavioural patterns. Contrary to classical heuristic engines which focus on the detection of encrypted malware samples, we integrate a known packer detector as well as unpacking routines to circumvent the protection techniques used by most of the modern malware. We believe that many obfuscated techniques used by malware authors are available on the Internet. More precisely, the use of known packer removals would strip out the packer protection with our dedicated decryption routines. Our apprehensive program is based on the integration of both static heuristic and emulator approaches; however, they do not necessarily have to serve as a complement for each other. Static heuristic scanner involves static extraction, which is relying on byte signature to identify a dedicated viral signature. Emulator can execute the arbitrary code from the instance and would trace the instance's body code in a virtual environment. It can be used to combat any protection code, regardless of the complexity of the protection algorithm. Fragments of virus body could be detected while the execution is in a decrypted virus body. Lastly, we present experimental results that indicate our proposed technique can provide good performance against obfuscated malware. Through this study, we hope to help security researchers understand our defence approach and give some directions for future research.

AB - Modern malware has become a major threat to today's Internet communications. The threat can infiltrate hosts using a variety of methods, such as attacks against known software vulnerabilities, hidden functionality in regular programs, drive-by download from unsafe web sites, and so forth. Matching a file stream against a known virus pattern is a fundamental technique for detecting viruses. With the popularity and variety of malware attack over the Internet, computer virus protection companies need to constantly update new virus signatures in their virus definition databases. However, the increasing size of the signature database can only detect known virus but cannot defend against new variants of malware. In this paper, we present an overview of the detection of modern malware focuses on suspect behavioural patterns. Contrary to classical heuristic engines which focus on the detection of encrypted malware samples, we integrate a known packer detector as well as unpacking routines to circumvent the protection techniques used by most of the modern malware. We believe that many obfuscated techniques used by malware authors are available on the Internet. More precisely, the use of known packer removals would strip out the packer protection with our dedicated decryption routines. Our apprehensive program is based on the integration of both static heuristic and emulator approaches; however, they do not necessarily have to serve as a complement for each other. Static heuristic scanner involves static extraction, which is relying on byte signature to identify a dedicated viral signature. Emulator can execute the arbitrary code from the instance and would trace the instance's body code in a virtual environment. It can be used to combat any protection code, regardless of the complexity of the protection algorithm. Fragments of virus body could be detected while the execution is in a decrypted virus body. Lastly, we present experimental results that indicate our proposed technique can provide good performance against obfuscated malware. Through this study, we hope to help security researchers understand our defence approach and give some directions for future research.

KW - Dynamic Analysis

KW - Emulator

KW - Heuristic

KW - Malware

KW - Static Analysis

UR - http://www.scopus.com/inward/record.url?scp=84866637307&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84866637307&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84866637307

VL - 42

SP - 260

EP - 270

JO - Journal of Theoretical and Applied Information Technology

JF - Journal of Theoretical and Applied Information Technology

SN - 1992-8645

IS - 2

ER -