After the engagement

Above anything else, how you process the results of your engagement is where most of the value can be derived.

Make sure your vendor cleans up after themselves. More than once on a security assessment I stumbled on escalated access or shells left behind by a previous assessment. Dr. McGrew has discussed extensively how vendors can reduce these risks, for example in his 2016 Black Hat talk “Secure Penetration Testing Operations: Demonstrated Weaknesses in Learning Material and Tools”

Receiving the deliverable

Almost all assessments result in a deliverable, the concrete artifact of the assessment and its outcomes. The deliverable is crucial. It is the main part of the assessment that remains beyond the time-boxed period.

Consider negotiating progressive deliverables for larger engagements. Instead of needing to process the results all at once, you can get a jump on analysis and remediation. Milestone deliverables can also drive confidence in vendor performance. In the best case, you can ask for re-prioritization or pivot targeting based on progress.

The Readout

The deliverable is often paired with a readout. This is a meeting or presentation focused on communicating the outcomes, providing any necessary context, and formally marking the close of the assessment. As a final touch point with the technical resources from your assessment, it is another opportunity to resolve any open questions and garner additional value. It is also the best opportunity to impart additional business context that may require revised ratings or front matter.

Consider asking the following questions at your next readout:

• If you had more time, where would you dig further or look next?

• What would you recommend we do differently for our next engagement?

• Were there any trends you observed? Are there any systemic mitigations you’d recommend?

• Were there any areas that were particularly well hardened?

• How does our posture compare to the (industry/benchmark/average engagement)?

Your vendor is a resource, take full advantage! One massive benefit of consultants is that they can see inside more companies in a year than most security practitioners will in a career.¹⁵

Reading an assessment report

In order to act on your assessment results, you need to understand how to navigate the report. You can refer back to (Public reports)[#public-reports] for examples.

The average report will contain contain:

• Assessment details: Scope, level of effort, tools and methodology, vendor and consultant information

• Executive summary: A (often narrative) recounting of the overall outcome, including risk posture, findings of note, and executive or meta recommendations

• Findings: Summarized, and then each with details, impact and risk, reproduction information, and remediation guidance

• Appendix: Expanding on contents of the report, such as additional information on bug classes, detailed remediation steps, custom tools or scripts developed, or raw data from testing

Generally, the idea is that the report can be decomposed. Sections present different levels of detail for different audiences. The assessment details and executive report should target leadership, providing the high level outcomes in a risk oriented and narrative fashion. The findings summary is generally suited to security leadership, to understand the types and quantities of vulnerabilities found, and their relative risk. Finding details are then ideal for line-level engineers who have to address the vulnerabilities, with all the details necessary to understand, reproduce, and fix the issues.

From the Survey: what do vendors do when there are no findings?

1. They manage client expectations, and note limitations

2. They provide extensive details on testing methodology or coverage

3. They strive to provide assurance of diligence

4. They highlight defenses encountered and “true negatives”

5. Some firms perform internal quality control investigations

6. They follow up and validate the results with client expectations (as a sanity check)

7. They try to add value via additional conversation on “other security related observations and best practices that can be deployed given the lack of findings”

Ingesting the results

Processing the outcome of a security assessment should be a team effort. In order to smooth this process.

1. Manage follow up work through your standard processes: You should already have vulnerability management procedures. This should include clear ownership and SLAs for remediation. Ensure you feed assessment results into this process. It allows you another opportunity to exercise that flow, centralizes tracking and reporting, and takes advantage of existing organizational infrastructure.

2. Triage all findings: It is important to reassess all findings, no matter the vendor. You have substantially more context on your organization and its risks. This is your opportunity to re-rate vulnerabilities and ensure they are prioritized accordingly. Beyond risk, you should also perform root cause and variant analysis. Go from an assessment that can identify an instance of a vulnerability to a posture that has killed the bug class. For each finding, as with all risks, you will have the choice to fix, mitigate, or accept it. At this point, you should also start estimating fix difficulty and level of effort. Mature organizations may find that the assessment reinforces their pending remediation efforts.

3. Consider requesting a parsable report: Assessment reports are generally delivered as either a document or presentation. Depending on your internal process for handling vulnerabilities, it can be helpful to request a different format. Enterprise clients will go so far as to have vendors file tickets within their issue tracker. At the very least, a CSV file (or similar) of the findings will let you programmatically import them. Set aside capacity to handle this process, including remediation within your established SLAs.