How to find potential vendors

“How do I find a vendor” is one of the first, and most surprisingly difficult, problems when building a security assessment program from scratch.

Some trends from the survey:

In some cases, the trouble is figuring out where to start to explore the industry. In others, it is separating the wheat from the chaff. At this point, security consulting is a big business and there are hundreds of options in the U.S. alone. In this section we’ll present a variety of approaches to help you find a strong partner for your assessment program or one off test.

Network recommendations

“Ask your network” is likely the first piece of advice you’ll receive. There is a good reason for this: a recommendation from a trusted party, based on their personal experience with a vendor, is an exceptionally high signal. But, when seeking network recommendations, be sure to keep in mind both the quality of your network and their ability to accurately assess the quality of their experience.

Be sure to take recommendations from non-clients of the vendor with a grain of salt, as there is more potential bias. Often, organizations can have successful engagements and be happy with a clean report – or maybe a flashy finding. This is not a clear indication of a good assessment in all cases. It’s possible that the finding was actually low-hanging fruit, or the clean report a sign of poor coverage.

Follow-the-leader (industry endorsements)

Security assessments are common practice at most of the largest tech companies, including all the Big Five (FAANG, or is it MANGA?). These companies also tend to be large enough to support marketplaces or integrations - and many require a security assessment for inclusion inside these walled gardens. In such cases, they often advertise a specific set of approved vendors. Those vendors are validated and vetted by a sophisticated security program, and can be considered a safe bet. However, be wary that your needs (and budget!) may be different from those of these large organizations.

A couple examples of these programs include Facebook Workplace ⁸ and the Google OAuth API verification ⁹ .

Research

Conference speakers

Security conferences are a rich hunting ground for vendors. Unfortunately, it can be hard to distinguish between high quality talks and vendor pitches, especially if you’re only referencing a conference program. Be wary of pay-to-play schemes and sponsored sessions. Despite that caveat, it can be productive to review the speakers and trainers, especially at “tier one” conferences (like BlackHat, Defcon, and CCC). Keep an eye out for talks and workshops specific to your technologies and problem space.

Published research

Like conference speakers, published research can also indicate the appropriate quality vendor. Security assessment is partially a creative pursuit. Research can reflect curiosity and innovation. Additionally, research can indicate a vendor’s technical depth on the bench. You may not need an expert in that specific topic, but if you’re looking for adjacent work, there is a correlated signal. Again, like with talks and trainings, this requires confidence separating marketing fluff and FUD from real research. Competitions like Pwn2Own can also demonstrate the proficiency of a vendor’s engineers.

Prominent staff

Like research, you can try and select a vendor based on their prominent staff or leadership. This is only valuable for vendors small enough that you could reasonably expect the direct engagement of that person. This is irrelevant for large vendors. They are more than happy to sell you on a big name, but with hundreds of consultants are likely to keep their notable employees for their most prestigious or challenging engagements, or research and presentations. If you’re considering hiring a former security researcher, bug hunter, or independent consultant, you should be vetting them individually.

Public reports

Most security vendors will either have public reports available, or will provide an example upon request. Reviewing these examples can help indicate the quality of the deliverable you’ll receive. They may contain evidence of the vendor’s methodology and reasoning around risk within a business context. Assume that these reports are of a higher quality than average for the vendor. Any deficiencies should be a red flag.

A generic repository of reports has been collated over on GitHub. The CNCF has also commissioned a set of high quality reports as part of graduating projects, for example:

• The Envoy audit, by Cure53

• The etcd audit, by Trail of Bits

If you are seeking a public report, expect a substantial surcharge. Public reports should always be objective and comprehensive. Reputable vendors should not suppress or remove findings from a report based on customer demand

Compliance approved

If one of your motivations for a security assessment is meeting compliance obligations, ensuring your provider satisfies the relevant requirements is obviously essential. For example, the Payment Card Industry (PCI) standard requires that organizations contract an Approved Scanning Vendor (ASV) for certain levels of certification. Another example would be CREST accreditation. Even if you’re not motivated by compliance, these can still be valuable broad indexes of possible vendors, although inclusion in an arbitrary scheme does not provide a strong signal.

Assessment standards work

There have been numerous efforts to standardize penetration testing and security assessments¹⁰ . While none have achieved universal adoption, their contributors can be a useful reference list. Not contributing to these efforts isn’t a negative signal, but participants in major standardization efforts, such as the Penetration Testing Execution Standard, show a focus on methodology, consistency in delivery, and alignment with industry best practices.

Certifications

Certifications are controversial in the security industry. A few maintain some cachet, such as lab-based certs like those from GIAC and Offensive Security for penetration testers. There are regional certification schemes with fairly broad adoption, like CREST in the UK. You should value the certification of an individual delivering your assessment more than a vendor carrying an organizational certification. Know that not all certification schemes are equal. Be dubious of non-technical certification, as well as overly broad ones

Analyst recommendations

Industry analysts have a mixed reputation, with the Gartner Magic Quadrant a meme in many circles. That being said, if you’re in the type of organization for which analysts hold a lot of sway, turning to Gartner, Forrester, or another advisory is a reasonable option. You’ll need to pay for access to these reports. Based on those I’ve reviewed, they at least won’t steer you terribly wrong. More importantly, just like “nobody ever got fired for buying IBM,” if your company respects these rankings you benefit from following suit. However, you should accept that in using these recommendations you will be directed at the top of the market. The only companies in most of these analyst reports are those that are large enough to afford the inefficiency of directly courting inclusion.

Survey results

Respondents were asked to stack-rank their preferred methods of finding an assessment vendor. As expected, network recommendations are considered best. I personally dispute the value of published research. It’s less likely to be directly correlated to real world assessment competency, but it is certainly flashy.