Background Information
WithSecure participated in the 2023 MITRE ATT&CK Evaluation (“Turla”). This was the fifth time we have taken part in the evaluations, run annually by MITRE.
MITRE Turla evaluation results can be found here
What is MITRE Turla
From MITRE:
“Turla is a Russian-based threat group that has infected victims in over 45 countries, spanning a range of industries including government, embassies, military, education, research and pharmaceutical companies since 2004. Heightened activity was seen in mid-2015. Turla is known for conducting watering hole and spearphishing campaigns and leveraging in-house tools and malware. Turla’s espionage platform is mainly used against Windows machines, but has also been seen used against macOS and Linux machines.”
WithSecure's high-level results
MITRE executed 143 tests across 19 steps in 2 scenarios and WithSecure has visibility within each step. This means that Elements EDR successfully detected activity across the kill chain.
WithSecure analysis and opinion
MITRE themselves list some “key things” about evaluations on their website
ATT&CK Evaluations are a starting point
We believe that customers should always take time to look deeper into the evaluation results, and make informed decisions of their choice of EDR solution. They should always look into the operational aspects of the solution, as well as any “group test” results. If the customer is swamped with false positives, it will actually slow down their workflow as they need to work out which is real and which is not.
There are no winners
MITRE make it explicitly clear that there are no winners or rankings from the evaluations. Several other vendors may claim “#1 in detection” or similar in their marketing activities, but this is misleading at best. It is always better to fully evaluate, and ask more questions from the vendor if needed.
Counting has limitations
Just because there are some numeric “scores” for each vendor solution in the evaluation, this does not paint the full picture. MITRE recommend taking these alongside your own needs as an organization.
Not all techniques are created equal
A technique for detecting one threat may differ to the technique used for detecting a different threat. There is no single “right way” to detect things, and potential customers are always recommended to dive into the details before making any decision.
Not all procedures are created equal
In some cases, a process starting may be detected from the command line being executed, and in other cases it might be detected from an API call. Not all tests, and not all vendors, use the same methods. As long as the detection with context is made, the actual method does not matter that much.
Participation in the evaluations
MITRE themselves set some fairly strict guidelines for participation in the evaluation process.
Standard products and configurations
Vendors taking part are expected to join with their “off the shelf” product, the exact same one that they sell to their customers. WithSecure is proud to say that it has always used the same product that our customers use, with the exact same configuration.
Protection Capabilities
Protection capabilities, such as those used by WithSecure Elements Endpoint Protection (EPP), which is usually used alongside WithSecure Elements Endpoint Detection and Response (EDR), are required to operate in a non-blocking manner. It is okay for them to detect malicious actions and TTPs (Tactics, Techniques and Procedures), but they should not prevent them from executing in the tests. This is NOT how customers use the software in the real world! Out of the box, Elements EPP would have blocked some of these long before EDR even needed to evaluate it, and this registers in the MITRE tests as a missed detection. We disagree with this from a real-world perspective.
Configuration Changes
Vendor configuration changes made during the tests are highlighted in the results. This is intended to signify changes made in response to earlier test phases, and in some cases we believe that some vendors indeed used this to improve their specific detections for MITRE tests. For example, the results show that one vendor had significant improvements in detection in MITRE-specific tests between the initial test and the final test.
WithSecure actually made one configuration change during the tests, which was highlighted, but that configuration change was not MITRE-specific. It affected other aspects of our detections, and was actually needed in non-MITRE scenarios too. We constantly look at our configurations to improve them, for the benefit of all of our customers.
MITRE expectations
MITRE expects the visibility of a detection to be recorded in a specific way. Vendors who make the detection visible in a different way are penalized. Our belief is that, as long as there is visibility, the way it is presented is actually somewhat irrelevant. For example, if we made everything visible the way MITRE expect, we would introduce a lot of “noise” into the reporting, reducing the value to the customer. A customer who has to wade through pages of noise will miss the important information, leading to less effective use of the solution. That information IS still available for advanced analysis of the situation, so we’re not hiding it away.
During our analysis of the comparative results between vendors, we noticed that for one of our missed detections, we had actually detected it but reported in a way not aligned with MITRE’s expectations, and we were down-marked. For the same test, another vendor actually did things in a very similar way to WithSecure, but was not penalized. In this respect, we consider that the analysis between vendors was sometimes inconsistent.
Conclusion and key findings
The MITRE evaluations test how many ways a product can detect a single threat. Let's put that into context:
- MITRE lists 14 tactics that attackers can use
- There are at least 200 adversary groups each with their own flavour of Tactics, Techniques and Procedures (TTP)
- There are nearly 4000 new attacks each day.
- The MITRE evaluations only cover a small proportion if scenarios
Having said that, we are happy to participate in the MITRE evaluations, they are the most objective and authoritative ones available at this time, even if the scope is limited.
We are not unhappy with the results. If we got 100% across the board to satisfy these tests, this would mean having 20 ways to detect a threat all across the kill chain. This is great for MITRE results, but it creates hell for your SOC analysts because they may have 20 or more alerts to process. Maybe they have some SOAR capability to collapse them into a smaller number but this will involve development work to create it.
There is more to Detection and Response than just detecting threats:
- Keeping false positive rates low
- If and how you respond as a vendor
- Providing security insights that enable clients to improve their security posture
- Ensuring skilled experts are investigating, containing and remediating attacks
- Just being a supportive partner.
With our 30+ years of working with partners and customers, WithSecure has an unparalleled experience of working with, and co-securing our customers.
Remember, the quality of the chef’s chopping knives has no bearing on the nutritional value of the meal. That comes from the right ingredients, combined in the right way. We have the right chefs.
What will WithSecure do with the evaluation results?
WithSecure is always aiming to improve our software and solutions, and we take learnings from evaluations such as these.
We did have visibility in most areas of the tests, however not all of these were taken into account in the evaluation due to the way the MITRE tests work.
Some of the findings do show that we need to improve in some areas, and we take “action items” to make these improvements. Some will come sooner than others, this is entirely normal within the software industry, not just for security software.
Any changes we make will be for the benefit of our customers. If, as a side-effect, they help with MITRE tests, that is okay, but we will not optimize our software just to satisfy MITRE evaluation criteria.
Final words
We believe that our WithSecure Elements Endpoint Detection and Response is a robust solution, providing high visibility into the threats detected, together with solid response capabilities.
In the real world, WithSecure Elements keeps you safe!
The views and opinions expressed in this article are those of WithSecure and do not necessarily reflect the views or positions of any entities they represent.