The threat of advanced targeted attacks, also known as advanced persistent threats, or APTs, has spawned a wave of innovation in the security market.

According to a document share by Lawrence Orans, Jeremy D’Hoinne, analysts at Gartner, a company that provides security managers with a framework to select and deploy the most-effective threat defense technologies, part of the key findings show that traditional defense-in-depth components are still necessary, but are no longer sufficient in protecting against advanced targeted attacks and advanced malware.

According to Lawrence Orans, Jeremy D’Hoinne, “Today’s threats require an updated layered defense model that utilizes “lean forward” technologies at three levels: network, payload (executables, files and Web objects) and endpoint; combining two or all three layers offers highly effective protection against today’s threat environment and many vendors fit squarely in one style, but also have some characteristics of adjacent styles. The trend will be for vendors to “bleed through” multiple styles as solutions mature”.

Analysis of the Key Findings

Traditional defense tools are failing to protect enterprises from advanced targeted attacks and the broader problem of advanced malware.

In 2013, enterprises will spend more than $13 billion on firewalls, intrusion prevention systems (IPSs), endpoint protection platforms and secure Web gateways (see Note 1).

Yet, advanced targeted attacks (ATAs) and advanced malware continue to plague enterprises. ATAs are considered advanced because of their ability to bypass traditional security mechanisms.

Five Styles of Advanced Threat Defense: Strengths & Weaknesses

Style 1 — Network Traffic Analysis

This style includes a broad range of techniques for Network Traffic Analysis. For example, anomalous DNS traffic patterns are a strong indication of botnet activity.

NetFlow records (and other flow record types) provide the ability to establish baselines of normal traffic patterns and to highlight anomalous patterns that represent a compromised environment.

Some tools combine protocol analysis and content analysis. The sample vendors we list all use internally developed signatureless techniques that have been effective in detecting advanced threats.

The strengths include, real-time detection; includes signatureless and signature-based techniques and endpoint agents are not required.

But the challenges include, it requires careful tuning and knowledgeable staff to avoid false positives; limited ability to block attacks (applies to out-of-band tools) and does not monitor traffic from off-network mobile endpoints.

Style 2 — Network Forensics

Network Forensics tools provide full-packet capture and storage of network traffic, and provide analytics and reporting tools for supporting incident response, investigative and advanced threat analysis need

The ability of these tools to extract and retain metadata differentiates these security-focused solutions from the packet capture tools aimed at the network operations buyer.

The strengths: Can deliver a high ROI (due to reducing incident response time and personnel); can reconstruct and replay flows and events over days or weeks, due to high-capacity storage options (for example, 200TB) and detailed reports can be used to help meet regulatory requirements, such as e-discovery or Payment Card Industry, for in-depth analysis and continuous monitoring of network traffic.

However, the challenges are the tools are complex, and skilled personnel are required to operate them; costs rise with the amount of data and the retention time; reports that analyze large amounts of data are time-intensive and may need to be run off-hours and does not capture traffic from off-network mobile endpoints.

Style 3 — Payload Analysis

Using a sandbox environment, the Payload Analysis technique is used to detect malware and targeted attacks on a near-real-time basis.

Payload Analysis solutions provide detailed reports about malware behavior, but they do not enable a postcompromise ability to track endpoint behavior over a period of days, weeks or months. Enterprises that seek that capability will need to use the incident response features of the solutions in Style 5 (Endpoint Forensics). The sandbox environment can reside on-premises or in the cloud.

Cloud-based Payload Analysis is a valid approach, but it is also a low barrier to entry for vendors.

Off-the-shelf hypervisors and virtualization technology, vendors can easily create sandbox environments and label them as Payload Analysis solutions.

Feedback from Gartner clients indicates that there is a wide range in the ability of these cloud-based Payload Analysis solutions to accurately detect malware.

As per the strengths, it is very effective in detecting malware that successfully bypasses signature-based solution; detailed reports highlight registry changes, API calls, process behavior and other information about the behavior of the malware (these reports are helpful for postcompromise analysis, but are not a substitute for the in-depth tools outlined in Style 2 and Style 5).

Also, there is an optional blocking capability for outbound callback to command and control centers for those on-premises-based (noncloud) solutions that can be placed in the line of traffic.

The challenges abound because behavioral analysis can take several seconds or minutes to complete, previously undetected malware is allowed to pass through, potentially compromising one or more endpoints; some evasion techniques can defeat the behavioral analysis technique.

For example, sleep timers, in which the malware code executes on a delayed basis (hours or days), may result in the malware going undetected during the time it is resident in the sandbox. Some vendors have techniques for detecting and thwarting sleep timer evasions.

It does not provide validation that the malware executed on endpoints. Just because the malware behaved a certain way in a simulated environment does not guarantee that it will behave that way on real endpoints.

Some malware does not install and execute as expected on every endpoint.

Many solutions only support a limited range of payloads. Some support executables (.exe files) only.

Most solutions only support Microsoft Windows, although some cloud-based approaches support Android. At the time of this writing, none support Apple Mac OS X.

Privacy and data protection concerns may prevent some enterprises from implementing cloud-based sandboxes.

Style 4 — Endpoint Behavior Analysis

There is more than one approach to Endpoint Behavior Analysis to defend against targeted attacks.

Several vendors focus on the concept of application containment to protect endpoints by isolating applications and files in virtual containers.

Other innovations in this style include system configuration, memory and process monitoring to block attacks, and techniques to assist with real-time incident response.

An entirely different strategy for ATA defense is to restrict application execution to only known good applications, also known as “whitelisting” (see “How to Successfully Deploy Application Control”).

The application containment approach allows malware to execute, but it does so in a contained environment where it cannot access content and information outside of its container. For example, the containers intercept kernel system calls and block malicious activity such as thread injection attacks.

By isolating Web browsing sessions, this approach protects users from malicious websites, including drive-by download sites and “watering holes.”

These solutions require an agent on every endpoint. (For more information, see “Technology Overview for Virtualization and Containment Solutions for Advanced Targeted Attacks.”).

The strengths are: blocks zero-day attacks and previously unseen malware (applies to application containment solutions); protects systems whether they are on or off the corporate network; provides basic forensic capabilities through analysis of blocked malware.

While the challenges are: deploying and managing endpoint agents can be operationally intensive and creates challenges in bring your own device (BYOD) environments; endpoint agents have varying restrictions for supporting operating systems, file types, applications and browsers and containment solutions utilize additional CPU and memory resources.

The more containers in use, the greater the impact, according to Gartner.

Style 5 — Endpoint Forensics

Endpoint Forensics serves as a tool for incident response teams. Endpoint agents collect data from the hosts they monitor.

These solutions are helpful for pinpointing which computers have been compromised by malware, and highlighting specific behavior of the malware.

Some solutions use various indicators of compromise (IOCs) to detect malicious behavior on the endpoint.

Examples of IOCs include suspicious Microsoft Windows registry key creation, DNS requests or installed binaries.

The strengths are, it helps automate the time-consuming task of incident response; monitors activity on hosts when they are on or off corporate networks; some agents provide limited containment features (for example, preventing previously detected malware from running again or on other endpoints in the company) and limited remediation capabilities.

The challenges include, a lack of ability to block zero-day attacks in real time; deploying and managing endpoint agents can be operationally intensive; at the time of this writing, support for non-Windows endpoints is limited and events are not always prioritized and typically require knowledgeable staff to investigate.