Measuring virtual machine detection in malware using DAD tracer

Measuring virtual machine detection in malware using DSD tracerBoris Lau, Vanja Svajcer Sophoslabs, Journal in Computer Virology, 2008 SophosSophosvirtual environment detectionDSD Tracera dynamic and static tracing systemmalicious filespacker codevirtual machine detection methodsDSD tracer20071OutlineIntroductionVirtual machine detection methodsMethodology of our study with DSD-TracerResultsConclusion2Introduction#1Virtual machine technology is first implemented by IBMMore attention from virus writers & computer security researchersIf in VMmalware will behave like a normal programIf the proportion is > 0.1%developing an environment to successfully analyze VM-aware malware is important

3VMmalwaremalwaremalwareVMreal host3Introduction#2The most common security use cases with VMSoftware vulnerability researchMalware analysisHoneypots

4OSOScreate send unexpected application input black box analysisSystem debugger to trace error condition

1. virusOS real host

1. VMhoney potvirushoney potvirusVMhost

4Virtual machine detection methods#1If VM is detected, the malware will stop its execution orlaunch a specially crafted payloadZlob TrojansIRC botsExecutable packers

5Zlob TrojansIRC botsThe main bot function wont be exhibited and terminate its execution5Virtual machine detection methods#2Detection of running under MS virtual PC using VPC communication channelCommunication between guest OS & VMMExceptions due to opcode0x0f, 0x3f / 0x0f, 0xc7, 0xc8 Call different VMM services 0x07, 0x0B60x0f, 0x3f undefined OPcode0x0f, 0xc7, 0xc8illegal encoding of an existing opcode. 6Invalid instruction VPC communication channel detection

7070BIsRunningInsideVirtualMachine() API7Virtual machine detection methods#3Detection of running under VMware using VMWare control APIVMWare backdoor communicationguest host communication IN instructionport 0x5658eax0x564D5868VMXhebx function number

8hostuser modeIN / OUT priviledge instructionexception89

if VMware is running, no exception is generated. Instead, the EBX register is altered to contain 'VMXh(magic number)the ECX register is also altered to contain the VMware product ID

9Anti-VMWare prevention virtual machine initialization settings10

settingsVM configuration filebug10Virtual machine detection methods#4Redpillusing SIDT, SGDT or SLDTSxxT x86 instructionReturn the contests of the sensitive registerIDT in VMWare is 0xffXXXXXXIDT in Virtual PC is 0xe8XXXXXXCompare with 0xd0Invalid in multi processor system11SxxT x86 instructionVMx86 instruction setExSIDTIDTRmemoryVMVMIDTRhosts IDTRSIDThostvalue11Redpill12

Checks the first byte of the IDTR and compare with 0xd012Virtual machine detection methods#5SMSW VMWare detectionStore Machine Specific Word instructionReturn 16-bit result32 bits register16-bit undefined + 16-bit resultIn VMWare, the top 16-bits doesnt change13definemagic numberregisterregistertop 16-bitmajic numbertop 16-bit13SMSW VMWare detection code14

system services in VMWare, Virtual network car MACSystem Bios14Methodology of our study with DSD-Tracer#1DSD-Traceridentify obfuscation packersdynamic & static analysis

15DSD-Tracer is a malware analysis framework that integrates dynamic & static analysis15Methodology of our study with DSD-Tracer#216

Methodology of our study with DSD-Tracer#3Dynamic componentInstructions decoded before its executionAll CPU registersReads / writes to virtual / physical memoryInterrupts / exceptions generatedInstrumented virtual machineLow-level information17Dynamic componentdecodeCPU register statememory stateinterrupt/exceptionsInstrumented virtual machinedetect VM low-level information17Methodology of our study with DSD-Tracer#4Static componentC++ interfacePython ScriptMatch known techniques for detecting VMAutomatic replication harnessWeb-based automatic replication harness18Methodology of our study with DSD-Tracer#5Case studyDSD-Tracer on ThemidaAnalyzing Themida by traditional debugger/static technique is troublesomerecording memory-iodump sample in static environment19Dsddump page-level anti-dumping techniquesDump virtual addressCPU tickVM-aware technique19Methodology of our study with DSD-Tracer#6Justification for using DSD-TracerCoverage of packed samplesLow-level accuracyCircumventing armour techniquesMitigating factors in using DSD-TracerNo Bochs detect techniques in any sample4 samples/hour, 5 samples from each set of packed file85% of Themida samples with VM-aware techniques

20samplespack20%static analysis techniquepacked samplesunpackingembedded VM-aware detectionlow level assembly informationtoolsNorman Sandbox Analyzerreal time analysisida-x86emuunpackingDSD-Tracer kernel mode debugger ring0debugeereal OS or CPU stateBochsBuchsopen source software Buchss debugger is written by other userspachdebuggerVmware, Virtual PCBochs Software Emulator x86 VMware Virtual PC x86 Virtual PC MS Widows 20Methodology of our study with DSD-Tracer#7Proof of concept experiment for DSD-Tracer on VMwareCross-verified multiple dynamic analysisImplemented on VMware Workstation 6Invisible breakpointGDB script for printing the assembly execution trace in user mode

21Bochs v.s Vmwaredetect VMX backdoorVmware21Results#1VM detection in packers193 different packers, 400 packed samplesOverall VM detection rate is 1.15%Themida accounting for 1.03%ExeCryptor accounting for 0.15%EncPkcustom packers22VM-aware detectionmalicious file obfuscation methods

ExeCryptor0.15%samplesvirtual and real environmentcreate ExeCrytor executablesExeCryptorEncPkSophos generic custom packer detectioncustom packersmalware22Results#2VM detection in malware familiesStatic analysis rules disassemblyDynamic analysis rules Sophos virus engine emulation2 million known malicious filesA large set of knows clean filesVM-aware samples < 1%Method breakdownTable 1.Family breakdown Table 2.Dial/FlashL

23rulesnon-malicious filesDial/FlashLVM23Results#324

Results#4VMWare backdoor detection method 50% VPC illegal instruction detection methodVPC illegal instruction detection method 93% VMWare backdoor detection method

25Results#5Fig. 7 VMWare backdoor detection in 2007

26

Results#6Fig. 8 VPC backdoor detections in 200727

ConclusionCombination of dynamic and static analysis is better2.13% VM-aware samples28Static analysis methods isnt reliableobfuscated & encrypted codeDynamic analysis is slowVM-aware techniques VMVM-based automated analysis system

28 Q & A29AppendixVMWare backdoor I/O portOn the Cutting Edge:Thwarting Virtual MachineDetectionTrapping worm in a virtual netVMVirtual PCBochshttp://hi.baidu.com/%CC%FA%D0%AC%B9%C3%C4%EF/blog/item/085cc609b215f3226b60fba5.html http://www.osnews.com/story/1054

30backdoor I/P portcommand numberVM Detectionpaperdetection methodsdiscuss the use of VMware to create a test environment for malicious code30Thanks ~31