A Tale of Two Penetration Tests

Over the past few months, we spoke with two organizations looking for a penetration test.

Both were serious. Both were acting in good faith. Both wanted to “do the right thing.”

On paper, the larger organization looked like the more mature buyer. In practice, the smaller one asked better questions.

Two First Conversations

The first organization was relatively small and preparing for its first formal penetration test.

They came in curious. They asked about:

• Our testing process
• White-box versus black-box approaches
• What types of testing we do manually versus with automation
• What insurance coverage we carry
• How we differ from other vendors they were evaluating

Some of these questions were expected. Others were thoughtful and specific. All of them were welcome.

What stood out wasn’t the volume of questions — it was the intent behind them. They weren’t trying to get the lowest price or the fastest turnaround. They were trying to understand what kind of risk this test would actually reduce.

They wanted to pick the right vendor to answer the right questions.

A Very Different Second Conversation

The second organization was much larger and had done penetration testing in the past.

They were looking to test more than 25 applications and APIs. Their previous vendor had completed testing in roughly 60 days, but we heard internal feedback that this was considered “too long.”

We also heard comments like:

• “A really thorough test isn’t necessary.”
• “Our internal security team has already reviewed most of this.”

Some of the systems in scope were genuinely complex. Others might be tested quickly — assuming environments, access, and dependencies were already in place.

To focus the engagement, we asked them to prioritize their most critical targets.

They sent back a list of eight. We appreciated that.

We scoped accordingly and proposed a 45-day engagement, along with full documentation of our methodology and approach.

After some time, they came back with a decision.

They chose to stay with their existing vendor — citing familiarity with the applications, broader coverage, and lower cost.

The Quiet Math Most People Don’t Do

Here’s the part that often gets overlooked.

A 60-day engagement sounds substantial.

But once you remove weekends — and divide that time across more than 25 applications and APIs — you’re left with less than two days per target.

That time has to include:

• Environment setup
• Authentication flows
• Understanding application logic
• Manual testing
• Reviewing and validating automated findings
• Writing and reviewing reports

Even with automation, some of the most important tests can only be done manually. False positives need to be confirmed. False negatives are always possible.

When time gets compressed, depth is usually the first thing to go.

A Quick Thought Experiment

Take your last penetration test.

Divide the total testing days by the number of applications and APIs in scope. Then subtract setup, coordination, and reporting time.

How much real testing time was left per system?

Most people don’t run this math — until they see it written down.

Two Ways Organizations Buy Penetration Tests

After many conversations like these, a pattern shows up.

Most organizations approach penetration testing in one of two ways:

Coverage-driven:

• How much can we test?
• How fast can it be done?
• How does the price compare?

Assurance-driven:

• What risks does this actually reduce?
• What assumptions are we validating?
• What would we want to know before something goes wrong?

Neither mindset is inherently wrong. But they produce very different outcomes — even when the engagement is called the same thing.

This Isn’t Negligence — It’s Incentives

This isn’t about bad actors or incompetence. It’s about procurement logic being applied to a risk problem.

Price, speed, and scope are easy to measure. Confidence is not.

But when a test becomes something you need to get through, rather than something you need to learn from, the value shifts quietly — and often invisibly.

What Actually Matters

In the large organization’s case, price and familiarity carried the most weight.

In the smaller organization’s case, thoroughness and understanding did.

Both decisions were rational. Both involved tradeoffs.

The question is whether those tradeoffs are acceptable — especially when the systems being tested are critical to your business or your customers.

A Question Worth Sitting With

If a key system failed tomorrow, would you be more upset that:

• The penetration test didn’t go deep enough, or
• It cost more than expected?

When you say, “our internal team already reviewed this,” are you trusting your controls — or your assumptions?

Penetration testing isn’t about checking a box. It’s about deciding how much uncertainty you’re willing to live with.

---

If you’re evaluating penetration testing options and want an honest conversation about what depth actually looks like, we’re happy to talk.