How to request and review proposals

Running an RFP

Large enterprises frequently conduct structured RFPs. However, in small and medium organizations procurement much more closely resembles a few vendor solicitations and a comparison of the proposals.

How many vendors should you consider?

Gartner, and my anecdotal experience, recommend between three and five providers for a shortlist. ¹³ Remember to make informed decisions with regards to the cost of running this process in terms of internal resourcing. Any more than five vendors and you’re making an outsized investment just in vendor selection, which invites decision paralysis. If you are planning to procure a five day assessment, the RFP process can easily become a large component of your total costs. For less mature organizations, it can be more effective to start with a (formal or informal) RFI (Request For Information). This is a more casual introduction that can help you narrow your specific requirements for a vendor.

Fielding initial sales calls

All but the smallest firms have dedicated sales, who will be your point-of-contact during the process. Small vendors may have managing or principal consultants participate directly, which provides an advantage for your ability to directly vet the technical resources. Make sure that before the call you get a good understanding of who you’re speaking to: sales, sales and delivery, or the actual person who will deliver your assessment?

No matter the vendor, do not sign a contract before getting a chance to speak with a technical resource! That’s ultimately who you’re paying for.

What to ask prospective vendorsPermalink

In order to make an informed decision, you need data. For less experienced clients, it can be unclear what to ask or what questions are valuable. Here’s a cheat sheet of key questions:

How soon would you be able to staff this engagement?

Security services vendors can frequently book weeks or months out. If you have a tight timeline it can be efficient to lead with that constraint. You also can use this opportunity to discuss staffing models. There are benefits and drawbacks to staffing multiple people on an engagement. It is easier to staff and quicker to deliver an engagement when multiple testers can be used. These collaborative engagements can also benefit from thought partnership and complementary skill sets. They also take less chronological time, which makes them easier for you to support as a client and present shorter requirements for maintaining the test environment or change freeze. However, in a solo engagement the consultant is more likely to develop domain- and target-specific knowledge - which can have outsized impact if you have complicated business logic. If you have rigorous Assessor Requirements, onboarding multiple consultants can be more challenging.

What experience do you have with organizations like ours?

Be aware of any specifics of your client profile - industry, technology, or maturity. Identify a vendor that has evidence of success with such clients or at least is able to clearly identify how those will impact the assessment. Ask for a relevant reference or case study as evidence of this competency.

What is your engagement model?

This is an open ended question, but you should be sure to dig into:

1. Their supported collaboration models
   

   Do they provide real time access and feedback via Slack (or other chat)? Do they prefer to operate over email or phone calls? What do they include in project updates and on what cadence?

2. How they assign people to projects
   

   You should expect the vendor to align resourcing based on vertical, technology stack, or level of experience. You should ask whether you will be able to review consultant biographies, or whether you are guaranteed a level of experience or expertise.

   
   Despite this callout, it is not uncommon or unreasonable for vendors to only agree to general requirements. Substantial contracts (think ~$500k+) provide more leeway for client specification.

3. Their project management approach
   

   This has an outsized impact on large engagements and for enterprise clients. There are a variety of common models: from informal project management, to identification of a lead consultant with responsibility for the project, to dedicated project management staff - a PM or a TPM, or simply client-managed projects with no vendor-side coordination.

4. The deliverable timeline and quality assurance process
   

   There are several considerations for the deliverable logistics. This starts with the timeline, which can range from last-day-of-testing to multiple weeks. It can also be insightful to ask who is responsible for editing the report. It could be a technical writer, a peer of the consultant, management, or something else entirely. Finally, gauge the vendor’s willingness to update reports post-readout, and the timeline (and any costs) in that case as well.

5. The methodology and tools
   

   Frankly, the specific methodology is less important than a clear signal that the vendor will ensure consistency in delivery. This is particularly important for large vendors, who have a broad slate of potential consultants. A clear methodology can also be valuable evidence when using the report for sales or compliance, which expect mention of common frameworks. This conversation should also lean into a discussion of how the vendor calculates risk. You should ensure they will be willing to rate risk within the context of your specific environment and business.

While the methodology may not be the most important component, it certainly can be vetted. Using a web application assessment as an example: “We test the OWASP Top 10” is significantly less promising than “We follow a methodology aligned to the OWASP ASVS”

My favorite methodology question (for web assessments) is “How will you test our authorization logic?” I like this question because it can take a side-channel look at elements like manual versus automated testing, as well as level of diligence and consistency.

During these initial conversations, there are a few important themes:

1. You should you hear potential vendors speak to your specific industry, business logic, threat model and needs

2. You should favor vendors who are able to ask informed questions on your stack, implementation, and prioritization

3. You should take these opportunities to judge vendors on communication, responsiveness, and collaboration skills

What to expect from vendors?

Scoping

In How you should scope your assessment, we discussed how you should enter procurement with a strong conception of the scope of the assessment. Following the initial conversation(s) with a vendor they should propose a scope of work.

While scoping, it is important to explicitly communicate your expectations and requirements. This starts with your proposed scope. Consider requesting multiple scoping options from the vendor. These can be broken down by level of effort, coverage, or inclusion/exclusion of certain elements. The proposal should document clear objectives for the assessment, as well as the expected level of effort and depth of coverage. You should also ensure the scope includes concrete reporting requirements, as necessary. For example, if you’re expecting the report to use CVSS score or CHECK severity levels for risk ratings.

Good scoping lays the groundwork for a successful assessment. A suitable vendor will make sure to take into account your needs, priorities, and specific attack surface. Beware a vendor that quotes based on a naive metric like “number of IP addresses” or “number of services.” That is insufficient to accurately scope an assessment and lends itself to false precision. The process generally involves the vendor providing a questionnaire, or asking to run a perimeter scan or review code to understand complexity. This can be followed by a discussion or demonstration of the items in scope, all in service of generating a quote.

Quotes

An assessment proposal should be accompanied by detailed scope and pricing information. This will be either fixed price or time and materials ¹⁴ .

Get a quote with a detailed breakdown of costs. Beware “one size fits all pricing.” Detailed pricing is necessary to ensure you’re evaluating possible vendor proposals like-for-like. Some line items you may see broken out include:

• Pricing per-consultant, or varying rates for different levels of expertise

• Dedicated project management

• Reporting time

• Surcharges for additional deliverables, such as attestations of engagement, executive reports or retest reports

• Surcharges for travel, or expenses. Be wary as these can add up fast, especially for onsite engagements.

• Surcharges for out-of-hours testing or for retesting

The quote may also include denote sticker price, with a discount listed. This might be earmarked due to volume, a first time customer, or a recurring customer. However, don’t let a proposed discount obscure the actual value of the rate.

The quote should also spell out payment terms, such as:

• Net 30 (or 60, 180, 365, with penalties for late payment)

• Percent upfront (commonly half, either as a deposit or delivered at kickoff)

• Milestone based

Why are security assessments so expensive?

This is a common question about security services. It’s often paired with a perceived breakdown of the cost-per-consultant (e.g “Why is the company charging $X,XXX/day, that’s $XXX/day/consultant?”). While many vendors will charge as much as they can get away with, there are a variety of contributing factors to the price:

1. A significant portion of the cost is the need to compensate technical staff competitively. Hiring security engineers is difficult. A lack of experienced candidates has raised the average salary, and the market is only getting more competitive.

2. This cost is increased when you account for the fact that consultants may not be billable 100% of the time. This occurs due to scheduling constraints, inconsistent demand, ongoing professional development, and general burnout prevention.

3. Cost of staff also goes beyond salary, with the general rule of thumb for fully loaded cost of an employee (benefits, office space) as double raw salary.

4. The vendor also will incur overhead for non-technical functions, including sales and marketing. This tends to scale for larger organizations, which require more sophisticated logistics and need to drive increased demand. Sales commissions can be sizable, as well as advertising and executive costs.

rybolov broke this down in 2010 over at guerilla-ciso.com:

Signing the deal

Once you have multiple quotes for your security assessment, you should be ready to lock down a vendor. There are a few steps in this process, beyond blindly signing on a dotted line.

Negotiation

Security assessment services are variably priced, which means there can be room for negotiation on:

1. Rate: Price is an obvious axis for negotiation. A strong client may be able to directly negotiate a lower rate. However you can also consider cost savings tied to the consultant profile. As discussed in Quotes, some vendors charge per-consultant, with cost tied to expertise. In these cases, it may be possible to suggest a less experienced consultant if that meets your needs, providing cost savings. For smaller vendors, flexibility around payment terms may be met with more flexibility on pricing.

2. Scope: When looking to cut scope to save on costs, it is important to understand the tradeoffs. Cutting scope involves sacrificing breadth for depth. This may involve a sampling approach, or removing some assets or asset classes entirely.

3. Level of effort: Lowering the level of effort is the corollary to cutting scope. You will not receive the same coverage in fewer days/hours, so balancing cuts to level of effort with cuts to scope can be a multi-dimensional strategy. Cutting the level of effort means you sacrifice depth for breadth.

4. Reporting: In proposals that include retesting, multiple revisions, or multiple forms of report, it may be possible to drop those line items to reduce overall cost. Before taking that step, make sure you’ll have the right artifacts to drive your business outcomes.

5. Relationship: Some vendors may provide a discount on volume. Negotiating multi-assessment contracts (whether multi-year or different services) can decrease cost-per-assessment.

Vetting

It is important to vet the proposals you receive and to not just take the lowest bid. Some considerations:

1. Prefer explicit proposals: Vendor proposals should clearly document goals and capture contracted level of effort in terms of person-hours/days/weeks. Ideally, the proposal will also clearly state the methodology to be applied, as well as timelines for delivery.

2. Compare like-for-like: One common trap new clients fall into is going for the lowest quote, without comparing what they’ll get. Underscoping proposals in order to provide the lowest quote is a common tactic for less scrupulous vendors. I’d recommend suspicion of any quotes that significantly deviate from the pack. Go back to those vendors to interrogate how they can offer cost savings over competitors.

3. Think long-term: When reviewing proposals, it can be valuable to take a step back. If your organization is large or scaling, you should look for a vendor who can support that growth. You may also want your vendor to support both security and compliance. This allows you to develop a partnership, not just commission one-off testing. Working with a single vendor for more of your assessments opens up the potential for economies of scale and allows you to tap more strategic advisory work based on a history of collaboration.

4. Vet specific consultants: If you’re a large enterprise client, you may have the leverage to specifically approve the consultants who will be working on your projects. I have been required to interview to onboard to engagements for some large clients. However I’ve also seen some confusion over why it isn’t generally possible to choose which consultants will work on a project, or review bios. This is for a few reasons:

• Often, a specific consultant is unavailable due to scheduling constraints. Sometimes a vendor will have a consultant booked on long-term or recurring engagements occupying their calendar for months.

• You, the client, may not actually be a good judge of skill matching. Not all consultants deliver all types of engagements. A good vendor will pair consultants with projects where they can have the most impact.

• Consultancies are incentivized to balance utilization across their workforce. This isn’t, as clients suspect, to foist less experienced or less competent consultants on clients. But it can certainly be a means of ensuring junior consultants are applied to reasonable engagements.

• Frequently, a client may want specific consultants - whether well-known or a personal connection - written into contracts. Vendors are unlikely to lock in a consultant’s schedule, especially for a deal that could easily fall through.

How does this triad play into proposals?

If you are more flexible on timeline, you can save on cost and quality. Trying to book in Q4 can be especially tenuous or expensive to get on vendors’ crowded calendars.

Similarly, while there are certainly varying prices for good services, the speed and quality of an abnormally cheap quote should be considered.

References

One of the more effective ways to vet a consultancy is to conduct a reference check. This is similar to leveraging your network for recommendations. Reviewing example reports is a start, but ideally you will find a reference client through your network. If not, you can often request that the vendor place you in touch with someone. This may not be feasible if you’re pursuing a small deal. Vendors are hesitant to ask too much of their reference clients, but this can definitely be demanded at a certain deal size. Remember that this is the vendor trying to groom their image, but a signal remains. Case studies are another options, albeit marketing material.