The Blog
Build an Efficient Security Vendor Management

Build an Efficient Security Vendor Management

By Caleb Sima

How to Build an Efficient Security Vendor Management Process

There are countless companies selling cybersecurity products and services - from large established pure-play security brands to software companies offering cybersecurity solutions as part of their enterprise software portfolio to newly funded startups pitching entirely new categories of solutions. IT analyst firm Gartner predicts security spending to total $96.3 billion in 2018, an 8% increase over 2017. While a healthy and growing cybersecurity industry is essential for innovation, it also creates a crowded, noisy market that enterprise security executives must carefully wade through to find the right mix of solutions they need to reduce risk and protect assets.


In fact, as a managing VP for a large enterprise, I would be working with 70 vendors at any given time. Managing the vendor lifecycle from initial pitch through evaluation to deployment and ongoing management was a challenge on its own. And while many organizations have a complete process for managing vendors post-deployment, the first half of the lifecycle is often handled on a reactive, ad hoc basis. It started to become apparent that our largely reactive approach to vendor evaluation was straining resources and diverting the attention of already-taxed security team members away from our primary goals.

To address this, my team developed the Cyber Test Kitchen, which was a formalized and proactive approach to managing the evaluation phases of the vendor management lifecycle that yielded significant operational benefits for our team. To help CISOs avoid some of the more commonly voiced frustrations with vendor evaluation, I’ve developed this short guidebook based on using lessons learned from my experience leading this program for a large enterprise. While many of the practices themselves are quite simple, the most significant value will result from the initial investment in creating

a consistent and proactive process across your security enterprise.


Too Much of a Good Thing

Enterprise security programs rely on a mix of products and services, and CISOs are always looking for ways to do things better and more efficiently. But identifying and effectively evaluating the right products is too often more art than science.


Anyone with CISO, VP, Director, or manager in their LinkedIn profile is constantly inundated with vendor pitches, some of which are on target for their area of responsibility and others that are far afield of what they do. Sometimes pitches look interesting and may be relevant for planned future projects, but don't align with current priorities. Often those on the receiving end are interested in a new product’s potential but are not authorized to review and recommend products. Further, when a new need does arise, looking for the right product can be like searching for a needle in that proverbial haystack.


For products that do make it to the pitch stage, the value of vendor meetings can be inconsistent. It is not unheard of for members of the enterprise security team to actively try to avoid participating in vendor pitch meetings, fearing their time will be wasted, and it can be difficult to get their feedback even when they do attend. This perception of time wasting frequently carries over into subsequent phases of evaluation, including technology demos and proof of concept testing. Inconsistent processes for moving vendors through the evaluation creates broken feedback loops, redundant reviews and unnecessary delays in getting needed technology on board quickly. Sometimes products may be evaluated by one team and deemed not sufficient and then unknowingly re-evaluated by a different part of the organization only to reach the same conclusion.


Enterprise security team members recognize that managing the lengthy sales cycle of a large corporation is a challenge for vendors. But they don’t have the time or are simply not empowered to support them through the sale process. While this holds obviously frustrations for vendors, it can also create challenges for security team members who wish to engage with a specific vendor but encounter numerous time-consuming obstacles during the evaluation process.


Cyber Test Kitchen


To address these concerns, my team created the Cyber Test Kitchen, which extended our vendor management lifecycle to address the full scope of vendor engagement beginning with initial sales contact. In doing so, we not only streamlined processes, saved time and reduced internal frustrations with the vendor evaluation process,

but also improved the quality of relationships we had with the vendors we did choose as partners.


The Cyber Test Kitchen was designed to support our security team members with managing inbound vendor requests and executing new product searches. It provided a standard of communications and set expectations for both us and the vendors for each stage of the vendor evaluation process, which we defined as discovery, introductory pitch, assessment, proof of concept, and deployment. While we found we were doing many of these things already, by centralizing the management and driving consistency throughout the entire process, we realized significant advantages for our team and found the program was well-received by participating vendors.


We designed the Cyber Test Kitchen to address the following phases of vendor evaluation: Vendor Discovery, Introductory Pitch, Further Assessment, and Proof of Concept.




Vendor Discovery


Vendor discovery often takes the form of unsolicited inbound pitches, but it also encompasses the initial stage of internally-initiated product searches. It is essentially our first contact with a vendor. To better manage this process, we created a single point of contact for evaluating all inbound vendor pitches and supporting internally-generated vendor searches through the Cyber Test Kitchen. This contact is responsible to responding to all inbound pitches with a specific set of questions for vendors and ensuring that those who respond appropriately receive timely feedback on next steps. They also tracked all vendor interactions in an easily searchable format, so that we had a record of their interactions with us at each stage of the evaluation process. This information could be quickly searched and consulted to support subsequent evaluations of the same vendor

or internally-initiated searches for new products.


To manage this phase of vendor evaluation, clearly communicate to the vendor what information your organization requires to continue the discussion. Besides the obvious need for information on the product being pitched and the problem it solves, consider if there are specific requirements vendors must meet to work with your organization, like an established track record in your vertical or a minimum number of years in business. Asking these questions up front can eliminated time wasted evaluating products that won’t qualify.


We developed a standard questionnaire that we requested response to in a simple email format. Keeping everything in a concise email enabled our team to quickly forward relevant information to interested stakeholders in our organization who would perform initial vendor review.


We developed a standard questionnaire that we requested response to in a simple email format. Keeping everything in a concise email enabled our team to quickly forward relevant information to interested stakeholders in our organization who would

perform initial vendor review. Sample Questionnaire

  1. Please summarize your product and the problem it solves in 5 sentences or less.


  1. Please provide a link to a video demonstration of the product. Strongly prefer that no registration is required to view the video.


  1. Please provide the names of current, in-production users of the product, especially others in our industry.


  1. How long has your company been in business?


  1. How long has this product been in production and what version are you currently on?


Commit to responding quickly with a “no thank you” or an invitation to the next stage of the evaluation process. For those vendors which do not move forward, save and categorize, completed questionnaires for quick reference in case they align with internally-initiated future product searches.



Introductory Pitch


The pitch is an opportunity for the vendor to meet with relevant contacts in the organization to more fully describe the offering its benefits and review technical requirements. To ensure the meeting is as productive as possible, develop a clear and detailed agenda for the meeting and share it with the vendor. In addition, ask vendors to bring an engineer to the meeting who can answer detailed technical questions. By adding consistency and ensuring technical depth of discussion for pitch meetings, invited security team members will have a clear expectation for how their time will be spent and get more value from their participation.


Sample Agenda


  • Time available: 30 minutes


  • Meeting attendees and their roles. Please note we require all vendors to have an engineer present


  • Problem Overview: Please limit discussion of the problem being solved to one slide


  • Product Overview: Please be clear on what works today vs. what is planned for future release


  • Market Overview: Please discuss your competitors and how your product and approach differs


  • Feedback: Please allow time in your presentation for feedback from our team. We plan to take an active role in the discussion


Share feedback in real time during vendor pitch meetings and commit to a timely ‘no thank you” or invitation to the next step in evaluation.




Further Assessment


For vendors that continue beyond the introductory pitch, they enter a phase of increased engagement with the enterprise as their product is assessed further by relevant stakeholders. This phase can be long and confusing to vendors as they are often unsure as to whom to contact, how often to check in, and what additional information they should provide to support the evaluation process. This leads to inconsistent communication that does not

align with internal assessment timelines or processes and often leads to frustration for both vendors and the security team working with them. The key to streamlining this stage of the evaluation is transparency.


Provide vendors that make it to this stage with an evaluation roadmap that defines points of contact, key stakeholders, potential barriers for decision, and an estimated timeline for moving to the next phase of consideration. Though this level of transparency may

be highly unusual in the industry, we found that by setting clear expectations for vendors on when and how to follow-up and being open about our internal challenges, the quality of our communication during this extended phase of assessment rose significantly.


Sample Roadmap Elements


  • At least two primary points-of-co ntact and a request to funnel all communications through these contacts


  • A recommendation on how often the vendor should follow-up, e.g., once-per-month


  • An overview of key departments and/or stakeholders who will be involved in evaluation


  • A short overview of any known obstacles, key dates, or other information pertinent to evaluation


  • An estimated timeline for the length of the evaluation process.


  • A commitment to let the vendor know quickly if at any time the product is no longer under consideration.


Proof of Concept


The Proof of Concept phase of evaluation is typically led by the enterprise enabling them to test the product and use its features in their environment. During this phase, commit to

being transparent, provide timelines and communicate product issues or testing challenges quickly and clearly. Further, consider providing the vendor with the same analysis your team receives from internal stakeholders whether moving forward with purchase or not.


For those vendors that do not move past this stage of evaluation, ask them to respect the decision and let them know if there are any opportunities for future engagement, for example, invite them to contact you once they add certain features and have them in production for six months.



Moving Forward

Though the Cyber Test Kitchen is relatively simple in execution, an initial investment in defining our requirements at each stage of vendor evaluation and creating a consistent process to meet those needs can provide significant value for an enterprise security team. We were able to streamline the entire process of vendor evaluation and improve our internal RFP practices. We also removed the burden of responding to new product pitches from security team members focused on other priorities. From a vendor relations perspective, we found participation in the Cyber Test Kitchen eased onboarding for selected vendors and started the relationship off at a point of trust.


As you review your organization's vendor relations programs, consider extending formalized policies and processes to encompass the various phases of vendor evaluation. In this way, you can work toward partnership from the very first contact with your vendors while saving time and frustration.



About the Author

Caleb Sima has been engaged in the Internet security arena since 1994 and has become widely recognized as a leading expert in web security, penetration testing, and the identification of emerging security threats.


His pioneering efforts and expertise have helped define the web application security industry.


Caleb Sima's latest role was at Capital One serving as the Managing Vice President of Cyber Security. Prior to Capital One, he was the CEO & co-founder at Bluebox Security (acquired by Lookout) and previously operated as CEO of Armorize (acquired by Proofpoint). In the past, he served as CTO of Application Security at Hewlett-Packard via the acquisition of his first startup where he was CTO & Founder of SPI Dynamics.


As a founder of Badkode Ventures, Caleb also invests in startups. Some notable investments are FOSSA, Pindrop Security, Purewire, and Rocana.








Share this:
Three Expensive Security Operations Costs CISO 03
Three Expensive Security Operations Costs

Rapid7 - Three Expensive Security Operations Costs and How to Minimize Them with SOAR

How do you define Privileged Access? CISO 27
How do you define Privileged Access?

Back to the Basics: How do you define Privileged Access?

Request more information

You have missed out some details, please try again.

Your Name:
Job Title:
Company Name:
Please answer the above question to prove that you are human.

©2020 Global Business Intelligence | All Rights Reserved

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we'll assume that you are happy to receive all cookies. You can find out how we use cookies here.