COMP7905A - Reverse engineering and malware analysis

Semester 1, 2017-18

Mr. Frankie F.K. Li
Teaching assistants
Mr. Wing Cheong Virchow Chan
Mr. Ka Kin Ken Ma
Syllabus This course provides students a foundational knowledge about reverse engineering and malware analysis, through the study of various cases and hand-on analysis of malware samples. It covers fundamental concepts in malware investigations so as to equip the students with enough background knowledge in handling malicious software attacks. Various malware incidents will be covered, such as cases in Ransomware, banking-trojan, state-sponsored and APT attacks, cases in Stuxnet and malicious software attacks on Industrial Control System and IoT devices. With the experience of studying these cases and analyzing selected samples, the students will be able to understand the global cyber security landscape and its future impact. Hands-on exercises and in-depth discussion will be provided to enable students to acquire the required knowledge and skill set for defending and protecting an enterprise network environment.
Introduction by Instructor Cyber Security has become the top priority for any organizations in protecting their digital assets or online activities. Modern sophisticated adversaries can easily find vulnerabilities and make exploits to launch attacks for financial gains or as a way to achieve specific objectives. Cases of denial-of-service attacks to business servers, leakage of business and customer private information, advance malware attacks, fraudulent and malicious websites are found very common in daily news. This module provides you basic knowledge in malware analysis processes and their complexities as well as illustrate on how to build an analytical capability best suited enterprises environment. The tools and techniques presented in this course are intended for students to analyze selected samples in virtualization test-bed, so they can think like a blackhat to understand the TTP of malware authors.
Learning Outcomes
Course Learning Outcomes Relevant Programme Learning Outcomes
CLO1. Be able to understand the cyber security challenges raised from malicious software attacks PLO.1, 4, 6, 7, 8, 9, 14, 15
CLO2. Be able to analyze the security risks, threats and potential vulnerabilities on enterprise networks environment. PLO.3, 5, 6, 7, 8, 9, 10, 11, 12, 13
CLO3. Be able to carry out independent analysis of modern malware samples using behavioral, code analysis and memory forensic techniques PLO2, 3, 4, 5, 6, 7, 8, 9, 11
CLO4. Be able to apply the learned techniques to protect, reduce the security risks and avoid malicious software attacks on computer systems or networks. PLO.3, 5, 6, 7, 8, 9, 16
CLO5. Be able to research independently and use learned skills and tools to investigate malicious software attacks and implement or update a cyber protection plan PLO.1, 2, 3, 14, 15, 16
View Programme Learning Outcomes
Pre-requisites Students should have programming/development skills (Assembly, C, C++, Python) and knowledge in Operating System and computer network.
Compatibility Mutually exclusive with: COMP7804 E-commerce security cases and technologies.
Topics covered
Course Content No. of Hours Course Learning Outcomes
1. Modern malware 3 CLO1, CLO2
2. OS internals 4.5 CLO1, CLO2, CLO4
3. Sandbox virtualization and behavioral analysis 4.5 CLO2, CLO3, CLO4, CLO5
4. Reverse engineering and code analysis 9 CLO2, CLO3, CLO4, CLO5
5. Identify malware through memory forensics 3 CLO1, CLO2, CLO3, CLO4, CLO5
6. Cyber threat intelligence and IOCs 3 CLO1, CLO2, CLO5
7. Incident response 3 CLO2, CLO5
Description Type Weighting * Examination Period ^ Course Learning Outcomes
Individual project (malware attack incident identification) Continuous Assessment 10% - CLO2, CLO3, CL04
Individual project (analysis malware in a virtualization test-bed) Continuous Assessment 20% - CLO2, CLO3, CL04
Group or Individual project (identification of malicious functions of sample malware) Continuous Assessment 20% - CLO2, CLO3, CL04
Group or Individual
(identification of malware iocs and development of yara rules)
Continuous Assessment 10% - CL01, CL02, CL05
Written exam covers all taught content in the course. Written Examination 40% Dec 8 to 23, 2017 CL01, CL05
* The weighting of coursework and examination marks is subject to approval
^ The exact examination date uses to be released when all enrolments are confirmed after add/drop period by the Examinations Office.  Students must oblige to the examination schedule. Students should NOT enrol in the course if they are not certain that they will be in Hong Kong during the examination period.  Absent from examination may result in failure in the course. There is no supplementary examination for all MSc curriculums in the Faculty of Engineering.

Course materials Recommended readings:
  • Practical Malware Analysis: The Hands-On Guide to Dissection Malicious Software by Michael Sikorski and Andrew Honig, (Mar 3, 2012)
  •  The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory by Michael Hale Ligh, Andrew Case, Jamie Levy, AAron Walters, 1st Edition, 2014.
Session dates
Date Time Venue Remark
Session 1 6 Sep 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 2 13 Sep 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 3 20 Sep 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 4 27 Sep 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 5 4 Oct 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 6 11 Oct 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 7 25 Oct 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 8 1 Nov 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 9 8 Nov 2017 (Wed) 7:00pm - 10:00pm LE-2  
Session 10 15 Nov 2017 (Wed) 7:00pm - 10:00pm LE-2  
LE - Library Extension Building
Add/drop 1 September, 2017 - 14 September, 2017
Quota 100