Abstract

Today, many modern malware developers is taking the advantage of Application Programming Interface (API) hook technique to take the control of the victim computer which making it difficult to detect their presence. Because of the sophistication of rootkit tools, a remote attacker can use native API to compromise any computer which can later be used for many illegal activities such as sniffing network lines, capturing passwords, sending spam and DDoS attack, etc. Thus to protect end-system by identifying and preventing native API malicious code hooking is a challenging problem to the defenders. Today, many different malware-analysis tools incur specific features against malwares but manual and error-prone. In this study, we proposed a behavior-based monitoring detection system to effectively deal native API hooks in user-mode. Unlike other malware identification techniques, our approach involved dynamically analyzing the behavior of native API call hooking malwares. Comparing our experimental evaluation results with existing tools show better performance with no false positive.

Keywords:

API hook , dynamic analysis , malicious code , rootkit , user-mode,

References

1. Battistoni, R., E. Gabrielli and L.V. Mancini, 2004. A host intrusion prevention system for windows operating systems. Proceeding of 9th European Symposium on Research in Computer Security (ESORICS ’04), pp: 352-368.
   CrossRef    
2. Deng, Z., D. Xu, X. Zhang and X. Jiang, 2012. IntroLib: Efficient and transparent library calls introspection for malware forensics. Digit. Invest., 9: S13-S23.
   CrossRef    
3. Hejazi, S.M., C. Talhi and M. Debbai, 2009. Extraction of forensically sensitive information from windows physical memory. Digit. Invest., 6: S121-S131.
   CrossRef    
4. Kumar, E.U., 2010. User-mode Memory Scanning on 32-bit & 64-bit windows. J. Comput. Virol., 6(2): 123-141.
   CrossRef    
5. Liu, L., Z. Yin, S. Yuli, H. Lin and H. Wang, 2012. Research and design of rootkit detection method. Phys. Proc., 33: 852-857.
   CrossRef    
6. Ma, W., P. Duan, S. Liu, G. Gu and J.C. Liu, 2012. Automatically evading system-call-behavior based Malware detection. J. Comput. Virol., 8: 1-13.
   CrossRef    
7. Mansoori, M., O. Zakaria and A. Gani, 2012. Improving exposure of intrusion deception system through implementation of hybrid honeypot. Int. Arab J. Inf. Techn., 9(5).
   
8. Rabek, J.C., R.I. Khazan, S.M. Lewandowski and R.K. Cunningham, 2003. Detection of Injected, dynamically generated, and obfuscated malicious code. Proceeding of the 2003 ACM Workshop on Rapid Malcode, pp: 76-82.
   CrossRef    
9. Wagner, D. and P. Soto, 2002. Mimicry attacks on host-based intrusion detection systems. Proceeding of 9th ACM Conference on Computer and Communications Security, pp: 255-264.
   CrossRef    
10. Wang, M., C. Zhang and J. Yu, 2006. Native API based windows anomaly intrusion detection method using SVM. Proceeding of IEEE International Conference on Sensor Networks, Ubiquitous and Trustworthy Computing (SUTC’06), 1: 514-519.
    CrossRef    PMCid:PMC1861497    
11. White, A., B. Schatz and E. Foo, 2012. Surveying the user space through user allocations. Digit. Invest., 9: S3-S12.
    CrossRef    
12. Ye, Y., D. Wang, T. Li and D. Ye, 2007. IMDS: Intelligent malware detection system. Proceeding of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp: 1043-1047.
    CrossRef    
13. Yi, X., H. Da-Rong and S. Jun, 2010. Analysis of windows rootkits stealth and detection technologies. Proceeding of the 2nd International Conference on Applied Robotics for Power Industry, 2010.
    

Competing interests

The authors have no competing interests.

Open Access Policy

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Copyright

The authors have no competing interests.