Key Malware Detection Techniques

February 17, 2022

Last Updated: January 15, 2024

Share on:

Threat actors employ malicious software, known as malware, to perform certain malicious activities. There are numerous types of malware, each deployed to achieve different objectives. For example, spyware aims to gather information from computing devices, trojans aim to gain a persistent hold on a computer system, and ransomware aims to encrypt information and extort its owners.

Malware detection involves using techniques and tools to identify, block, alert, and respond to malware threats. Basic malware detection techniques can help identify and restrict known threats and include signature-based detection, checksumming, and application allowlisting. Advanced malware detection tools employ artificial intelligence (AI) and machine learning to proactively look for and identify new and unknown malware threats.

This is part of an extensive series of guides about malware protection.

Malware Detection Techniques

Signature-Based Detection

Signature-based detection uses the unique digital footprint, known as a signature, of software programs running on a protected system. Antivirus programs scan software, identifies their signature and compares it to signatures of known malware.

Antivirus products use a large database of known malware signatures, typically maintained by a security research team operated by the antivirus vendor. This database is frequently updated and the latest version is synchronized with protected devices.

When an antivirus program identifies software that meets a known signature, it stops the process and either quarantines or deletes it. This is a simple and effective approach to malware detection and is important as the first line of defense. However, as attackers become more sophisticated, the signature-based approach cannot detect a wide variety of newer threats.

Checksumming

This method is a type of signature analysis that involves calculating cyclic redundancy check (CRC) checksums. Checksumming helps verify that files are uncorrupted. The main drawback of signature-based detection is creating a massive database generating false positives, which checksumming aims to address.

Hackers often use polymorphic malicious advertisements to avoid detection by signature-based identification methods. Polymorphic viruses can change themselves when replicating, eliminating consistent search strings – usually, the hacker encrypts random decryption command sets in the form of non-constant keys in the virus code.

Thus, when the security team identifies a malicious signature, the malware no longer contains the code fragment and cannot be found. The absence of a detectable signature in the variable code requires other malicious code detection techniques, such as:

Statistical analysis – analyzes the frequency of processor commands to determine if a file is infected.
Cryptanalysis – known-plaintext cryptanalysis decodes encrypted viruses using an equation system (like the classic cryptographic technique of decoding text without a decryption key). The cryptanalysis system reconstructs the decryption program’s algorithm and keys, applying the algorithm to encoded fragments to decode the overall body of the encrypted virus.
Heuristics – a malware detection team scans and analyses behavioral data to identify anomalous activity. The team must search for malicious code associated with suspicious behavior, such as a code served to thousands of users within a few minutes. The security team can then prioritize and further investigate suspicious incidents.

Reduced masks – the malware detection team can use elements within the encrypted virus body to circumvent the need for an encryption key when obtaining static code. The static code produced can reveal the malware’s signature or mask.

Application Allowlisting

Application allowlists (aka whitelists) are the opposite of the attack signature approach. Instead of defining which software the antivirus program should block, it maintains a list of approved applications and blocks everything else.

This solution is not perfect but can be highly effective, especially in high-security environments. It is quite common for legitimate applications to have security vulnerabilities, or introduce unneeded features that increase the attack surface. In some cases, the application itself is benign, but its use could expose the device to threats – for example, in some environments, there may be a need to block web browsing and email.

Application allow listing works best with devices that are strictly task-focused, such as web servers and internet of things (IoT) devices.

Machine Learning Behavioral Analysis

The above techniques are known as “static” detection techniques because they rely on binary rules that either match or do not match a process running in the environment. Static malware detection cannot learn, it can only add more rules or fine-tune its rules over time to increase coverage.

By contrast, new dynamic techniques, based on artificial intelligence and machine learning (AI/ML), can help security tools learn to differentiate between legitimate and malicious files and processes, even if they do not match any known pattern or signature. They do this by observing file behavior, network traffic, frequency of processes, deployment patterns, and more. Over time, these algorithms can learn what “bad” files look like, making it possible to detect new and unknown malware.

AI/ML malware detection is known as “behavioral” detection because it is based on an analysis of the behavior of suspect processes. These algorithms have a threshold for malicious behavior, and if a file or process exhibits unusual behavior that crosses the threshold, they determine it to be malicious.

Behavioral analysis is powerful, but can sometimes miss malicious processes or incorrectly classify legitimate processes as malicious. In addition, attackers can manipulate AI/ML training processes. In several cases, attackers fed specially-crafted artifacts to a behavioral analysis mechanism, to train it to recognize malicious software as safe.

Related content: Read our guide to advanced malware detection (coming soon)

Advanced Malware Detection Technologies

While many organizations rely on legacy antivirus as their malware detection strategy, mature security organizations typically use two types of advanced solutions to defend against malware – endpoint protection platforms and endpoint detection and response solutions.

Endpoint Protection Platforms (EPP)

EPPs are deployed on endpoints such as employee workstations, servers, and cloud-based resources. They serve as the first line of defense that can identify threats and block them before they cause damage to sensitive assets.

EPPs use multiple techniques to detect and block malware:

Static analysis – EPPs leverage traditional static analysis methods to identify known malware strains and allow/deny applications flagged by administrators.
Behavioral analysis – EPPs add behavioral analysis to detect unknown threats or known malware that uses evasion tactics like mutation or encryption.
Sandboxed inspection – EPPs can run suspicious content in a sandbox, isolated from the main operating system. This makes it possible to “detonate” a file, observe its behavior and confirm if it is really malicious or not.
Content Disarm and Reconstruction (CDR) – EPPs make it possible to remove malicious elements of legitimate content, and allow the user to access the content itself. For example, if a Word document has a malicious macro, CDR can remove the macro and allow the user to access the file, instead of blocking it entirely.

In addition to these techniques, once the malware is detected, EPPs can actively protect the environment, for example by isolating the endpoint from the network.

Endpoint Detection and Response (EDR)

EDR solutions complement EPP solutions by allowing security teams to identify and respond to attacks on endpoint devices. If EPP failed to contain a threat, EDR makes it possible to:

Triage and investigate alerts – EDR provides rich data from endpoints that allows security analysts to identify signs of an attack, and investigate them to confirm a security incident.
Threat hunting – EDR makes it possible to proactively search endpoints and explore relevant data for signs of a breach.

When an analyst confirms a threat on an endpoint, they can use the EDR platform for incident response. For example, analysts can quarantine all devices affected by malware, wipe and reimage infected endpoints, and run automated security playbooks. Security playbooks can be used to coordinate a response to a malware threat across multiple security tools – including firewalls and network segmentation, intrusion prevention systems (IPS), and email security. Many EDR solutions incorporate EPP capabilities.

Advanced Malware Protection with Cynet

The Cynet 360 Advanced Threat Detection and Response platform provides protection against threats including zero-day attacks, advanced persistent threats (APT), advanced malware, and trojans that can evade traditional signature-based security measures.

Block exploit-like behavior

Cynet monitors endpoints memory to discover behavioral patterns that are typical to exploit such as an unusual process handling request. These patterns are common to the vast majority of exploits, whether known or new, and provide effective protection even from zero-day exploits.

Block exploit-derived malware

Cynet employs multi-layered malware protection that includes ML-based static analysis, sandboxing and process behavior monitoring. In addition, they provide fuzzy hashing and threat intelligence. This ensures that even if a successful zero-day exploit establishes a connection with the attacker and downloads additional malware, Cynet will prevent this malware from running so no harm can be done.

Uncover hidden threats

Cynet uses an adversary-centric methodology to accurately detect threats throughout the attack chain. Cynet thinks like an adversary, detecting behaviors and indicators across endpoints, files, users, and networks. They provide a holistic account of the operation of an attack, irrespective of where the attack may try to penetrate.

Accurate and precise

Cynet uses a powerful correlation engine and provides its attack findings with near-zero false positives and is free from excessive noise. This simplifies the response for security teams so they can react to important incidents.

You can carry out automatic or manual remediation, so your security teams have a highly effective yet straightforward way to detect, disrupt, and respond to advanced threats before they have a chance to do damage.

Learn more about Cynet’s Next-Generation Antivirus (NGAV) Solution.

See Our Additional Guides on Key Cybersecurity Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of Cybersecurity