In-depth continuous assurance over shallow periodic monitoring

This is core value #5 in the GRC Engineering Manifesto. I've been thinking about it a lot lately, especially given all of the dunking on SOC 2 that has happened over the last year, such as all the problems with "compliance commoditization" and "SOC-in-a-box" and who is to blame for it all (Compliance Automation vendors? the AICPA? audit firms? third-party risk management teams? all of the above?!)

There’s a growing chorus of folks in our industry who claim that SOC 2, and the AICPA’s stewardship of it, is thoroughly busted. On the other hand, auditors and CPAs claim that low quality or incompetent auditors are the problem and SOC 2 itself is fundamentally sound.

In my mind, there are valid points on either side of this debate.

But I also think there are deeper issues with SOC 2 (and other security compliance frameworks) that haven't been discussed as much, let alone what it looks like for those issues to be resolved.

In this post, I’m going to lay out what I believe these deeper issues are, paired with a vision for what a better approach could look like for security compliance frameworks to finally provide in-depth continuous assurance about an organization’s security controls.

The problem with SOC 2? It SOCs 2 much!

My first premise on this topic is this: there has never been a time when SOC 2 - or really any industry standard “security compliance audit” - was ever good enough, in any way, shape, or form at providing sufficient assurance about an organization’s security controls.

The fundamental issues with SOC 2’s assurance value predate “SOC-in-a-box” automation products. These issues have been present throughout SOC 2’s history: from SAS 70 being used before the cloud was The Cloud^TM (RIP ASPs); to SSAE 16 establishing SOC as the successor to SAS 70; to SSAE 18 replacing SSAE 16; and all the way up to the 2022 “Revised Points of Focus” update to the 2017 Trust Services Criteria. SOC-in-a box compliance commoditization has merely made it easier to see the deep flaws with security compliance frameworks and audits of all kinds.

The deeper issues with security compliance frameworks and audits

Every security compliance framework shares the same fundamental ingredients:

1. A set of requirements or objectives that controls must achieve

2. An audit methodology for determining how well controls meet said requirements

3. A reporting artifact to convey assurance signals to stakeholders via findings and opinions from the audit

How these ingredients are implemented vary from framework to framework. But aside from maybe one framework, they all do a poor job at putting these ingredients together in ways that provide in-depth continuous assurance.

Let's unpack the common flaws with each of these one by one.

Control requirements: vague solutions for unclear problems

The biggest issue with how control requirements are implemented is that they are defined without any explicit association or relevance to the threats they are intended to guard against (with the exception of HITRUST - but even it suffers from deeper issues with the other two ingredients I described above).

In the case of SOC 2, control requirements in the form of Trust Services Criteria (TSC) and Additional Points of Focus (PoFs) are so vague that they are virtually useless for ensuring organizations consistently design controls that are proven to be effective at protecting against relevant threats.

Let’s use SOC 2’s Common Criteria (CC) 6.6 as an example:

• CC 6.6 TSC: “The entity implements logical access security measures to protect against threats from sources outside its system boundaries”

• CC 6.6 PoF: “Identification and authentication credentials are protected during transmission outside its system boundaries”

Imagine for a moment if this same kind of requirements framework were used for, let’s say, car safety. In such a bizarro world of vague, loose, and totally optional car safety requirements (which basically existed for ~100 years), it would be like having a TSC stating “the car implements physical restraint measures to protect passengers against threats from abrupt deceleration events” and a PoF stating “passengers are protected during high speed movements on roadways.”

This clearly would be a virtually useless way to not only provide sufficient assurance about the safety of any given car, but to ensure that all cars are equipped with universally-applicable safety measures that protect against common threats that all passengers face.

This has certainly been the case for SOC 2 and other security compliance frameworks. They need to evolve their underlying controls requirements model to be focused on specific requirements that are known to be effective at protecting against common threats.

Audit methodologies: 20th century approaches applied to 21st century systems

This is the area that SOC 2 and other security compliance audits get criticized about the most: point-in-time screenshots, hours upon hours of walkthrough meetings, and quarterly lookback reviews all make the audit world go ‘round.

While these tend to be the most frustrating and inefficient aspects of audits for those who undergo them, I find that these are not the most problematic aspects of how current audit methodologies severely limit the assurance value such audits can provide. The biggest shortcoming with our security compliance audit methodologies is that we almost never assess historical evidence for technical controls’ operating effectiveness. This can result in huge misses about the reality of an organization’s’ control operating effectiveness which is best summed up with this meme:

Not only that, but auditors still primarily use evidence sampling methods (both statistical and nonstatistical) to draw conclusions about control operating effectiveness. However, there have been long-running concerns within the accounting profession with how auditors exercise too much professional judgment, rather than more strictly sticking to rigorous statistical principles, when determining what kinds of sampling methods to use and how to apply them. This can create excessive risks with drawing inaccurate conclusions about control operating effectiveness.

Now you might be thinking, “wait, my auditors always ask for samples of control evidence that include past dates during my audit period!” - and to that I would say: yes, you’re correct!

But this usually is only done for process controls that are inherently transactional in nature. For example: access requests, access (de)provisioning events, change requests, etc. which are operated via ticketing systems which necessarily means there is always historical evidence of a control’s operations.

However, technical controls that are inherently stateful in nature, such as at-rest data encryption, endpoint detection & response (EDR) tooling, or web application firewalls (WAFs), are only assessed by auditors for their current state. It doesn’t matter how long your audit period is - it could be 1 month, 6 months, or 12 months - your technical controls’ operating effectiveness is not being properly tested.

Why might this be? Well, a lot of organizations likely aren’t maintaining a historical audit trail of their technical controls’ state over time, and for whatever reason most auditors don’t think to push their clients to provide such historical evidence, even as an opportunity for improvement to implement before the next audit happens.

We need to change our audit methodologies to fully cover historical control operating effectiveness during the entirety of an audit period and to analyze the full population of a control, especially for technical controls. Otherwise, we’re only providing (very weak) assurance about controls’ operating effectiveness for the brief moment in time we gather evidence about their state.

Reporting artifacts: static documents for providing assurance about dynamic environments

Take a look at the results from this very unscientific poll I conducted on LinkedIn to gauge what our profession thinks the “use by” date for your SOC 2 Type II report should be before its assurance value expires. The results are fascinating:

It’s shocking that anyone thinks that a SOC 2 Type II report provides sufficient assurance up to 12 months after it is issued. Modern software-as-a-service organizations are making dozens, hundreds, and even thousands of changes to their systems every day. Every change poses varying risk to the operating effectiveness of the controls that exist in and around said systems: entire workloads deployed without threat detection controls in place (ouch!), new data stores deployed without at-rest encryption enabled (oops!), or a new web API released on a domain that your WAF isn’t configured to protect (oof!).

Now let’s take into account the fact that it can take weeks to go from the last piece of evidence being reviewed by your external auditor and your SOC 2 Type II report being finalized. Why do we treat this status quo of security compliance reporting artifacts as having any assurance value more than 1 week after they’ve been finalized?

We need a new kind of reporting artifact that is dynamic enough to reflect the current, and historical, operating effectiveness of an organization's controls. Static PDFs aren’t gonna cut it anymore.

A vision for security compliance audits that provide true in-depth continuous assurance

The problems with our current security compliance audit frameworks are clear: control requirements are very vague and not explicitly threat informed, audit methodologies are way too narrowly focused, and static reporting artifacts quickly become outdated views of highly dynamic control environments.

To overcome these fundamental flaws with SOC 2 and other security compliance frameworks, I’d like to propose a new framework for providing in-depth continuous assurance about an organization’s controls viability and operating effectiveness.

This framework:

1. Should have specific control requirements that are explicitly related to relevant threats

2. Should facilitate and require comprehensive control auditing methodologies that match the scale and depth of modern organizations’ control environments

3. Should provide a reporting artifact that is as dynamic as the control environments for which it is intended to describe the operating effectiveness

I call this framework ALCOVE: Assurance Levels for Control Operating Viability & Effectiveness.

Rather than totally reinventing the wheel, I believe we should draw inspiration from wheels that have already been invented from similar disciplines within security and software engineering. There already exists a scalable and robust security assurance framework that is being adopted by software providers, large and small, across various industries: Supply-chain Levels for Software Artifacts, or SLSA ("salsa") for short, which is a software supply chain security assurance framework.

Similar to SLSA, ALCOVE has various levels of assurance that an organization can strive to provide, allowing for flexibility for organizations and their stakeholders alike to provide and require establishing certain levels of assurance based on their specific needs.

Here’s what it could look like to integrate ALCOVE with SOC 2:

SOC 2 Type I and Type II reports would still exist in this world and still provide (limited) value for organizations and their stakeholders. However, we can extend SOC 2 report types to Type III and Type IV so SOC 2 can provide higher levels of assurance in line with ALCOVE Level 3 and Level 4.

In order for this to truly overcome the three fundamental flaws I outlined above, we need to also evolve our control requirements and reporting artifacts.

In the world of ALCOVE, SOC 2’s TSCs would be more rigorous, incorporating Common Threat Criteria and Common Mitigation Criteria to ensure organizations are implementing relevant controls that are well known to protect against relevant threats so that sufficient assurance can be clearly and unambiguously provided about control operating effectiveness. Here’s an example of what that could look like - the text in black below is existing text from SOC 2’s TSC language, and the text in red are the additional ALCOVE-related control requirements:

Finally, we need to revamp our security compliance reporting artifacts so they’re as dynamic as the control environments we’re trying to provide in-depth continuous assurance around. In addition to providing an audit report, with rich context about an organization’s system, architecture, controls, and an auditor’s opinion about control operating effectiveness, we also need control operating effectiveness metrics that capture current and historical operating effectiveness. Much in the same way that StatusPage created a new paradigm for providing transparency around an organization’s system uptime and availability, an ALCOVE-specific reporting artifact would look something like this:

While existing Trust Center products on the market today provide Key Control Indicator (KCI) metrics about an organization’s controls, they are, quite frankly, junk. They only tell you what the “current state” of an organization’s controls are like. And in some cases, certain Trust Center products will hide/remove a control (and its corresponding green checkmark) when the control is in a failing state. This is pure unadulterated security assurance theater. In order for real-time control operating effectiveness dashboards to truly provide in-depth continuous assurance, they must provide an honest representation of an organization’s controls, both current and historical states.

As excited as I am about how a framework like ALCOVE could help provide stronger assurance about control operating effectiveness, there is a big obstacle to adopting a more rigorous approach like it: incentives.

Overcoming the obstacle of misaligned incentives

Right now, incentives around security compliance audits are skewed in the wrong direction: many organizations, especially smaller and newer companies, are incentivized to pursue cheaper and weaker audits that don’t sufficiently scrutinize their controls. This distorts market signals about organizations’ risk profiles, especially when third-party due diligence teams rely on basic compliance requirements to “allow” vendors into their environment (“Do you have a SOC 2 Type II without a qualified opinion or any exceptions? Ok, great, now answer these 500 other questions about your security practices - be honest! If your answers look good enough, you’ll win our $200k ARR contract.” Let’s not kid ourselves: these incentives are so horribly misaligned.)

Additionally, third party security risk management teams aren’t usually empowered to make risk-taking decisions for other stakeholders at their organization, who are looking to use a vendor solution due to the outsized value they stand to gain from doing so. Third party security risk assessments are typically highly inefficient and take too much time to get to a decision about whether or not to proceed with a vendor given their risk profile. Teams that take weeks to assess a vendor that they would then want to say “no” to because of weak security controls will get bulldozed over by leaders at their organization. In so doing, they will burn through political capital, undermining trust in their team and function across their organization which can create feelings of job insecurity. In other words: third party security risk management teams have neither the leverage nor incentive to say “no” to a vendor that provides weak assurance about their security controls.

One way we could course correct incentives in this dynamic? By better integrating cyber insurance into the mix!

Here’s what the current state of our “security assurance & insurance” dynamic looks like in the world:

The incentives that exist in this dynamic are as follows:

• Auditors want to grow their customer base and make more money, meaning they want to avoid upsetting their customers with lengthy cumbersome audits that could result in them getting a “failing grade.”

• Customers are of two minds: the actual customer team for a vendor’s solution wants to get their hands on said solution ASAP. Third-party risk management teams are encumbered by contradictory fears: the fear of approving an overly-risky vendor (which if said vendor experiences a security incident, third-party risk management teams fear they will be held accountable for making a bad risk decision); and the fear of saying “no” resulting in backlash from the customer team.

• Vendors want fast, easy, and cheap audits so they can win more business and keep their sales cycle running efficiently.

• Insurance providers want to grow their customer base and make more money while reducing uncertainty about the risk pool they’re managing, such that they reduce the likelihood and size of claim payouts.

At the end of the day, all players are motivated by two fundamental incentives: spend less resources to get more value.

So what would a better model look like where incentives are better aligned in a way that actually could drive improved controls across organizations through stronger assurance signals being provided by vendors to other stakeholders?

I am going to draw inspiration for this idea from two sources: the Artificial Intelligence Underwriting Company, which is pioneering a novel AI Assurance + Insurance framework called AIUC-1 (which, you may notice, is very ALCOVE-esque); and Progressive Insurance’s Snapshot offering, which allows drivers to get discounts on their car insurance while allowing Progressive to reduce uncertainty about their risk pool via automated continuous monitoring of driving behavior.

Now the incentives change, driven largely by insurance providers who have the most leverage of any other stakeholder in this picture. All organizations currently have a strong incentive to transfer cyber risk to insurance providers in order to avoid experiencing catastrophic losses (for example, through extensive outages and system disruptions caused by ransomware).

Insurance providers have a great opportunity to push vendors to automatically and continuously feed them evidence about their controls, instead of gathering context via questionnaires once per year.

Vendors stand to save money on their cyber insurance premiums with the same, or potentially better, coverage, which of course requires them to ensure their controls are continuously operating effectively!

Auditors stand to have an easier time performing faster, more efficient and more rigorous audits.

Customers stand to gain stronger assurance, at the time they perform due diligence and continuously thereafter.

Did I just solve all of the world’s cybersecurity problems???

(I kid, I kid)

In conclusion

All of this is very much wishful thinking on my end. But we as a civilization have achieved crazier things in less opportune circumstances (see: sending humans to the Moon and back in a fancy metal pressurized can using 1960s-era technology).

What do you think of this? What seems like it would work or not work about these ideas? What would make it more viable?

Last week, during WWDC 2024, they announced Apple Intelligence and Private Cloud Compute (PCC). They spent a surprising amount of time boasting about how they architected a secure-by-design (SbD) and private-by-design (PbD) cloud computing architecture for Apple Intelligence. In doing so, they’ve continued to deservingly earn their customers’ trust and the right to dunk on their competition for failing to walk the same talk.

Their security blog post about PCC provides compelling details about how Apple has incorporated SbD and PbD features into PCC’s architecture.

I have a strong predisposition toward visual thinking and learning, especially when it comes to abstract information systems and security concepts. I took a stab at visualizing the security and privacy architecture features of Apple’s Private Cloud Compute as a way to make it easier for myself to wrap my head around their written description of PCC and also to validate and demonstrate the rigor of their approach.

At the end of the day, the ingredients in Apple’s PCC architecture that matters more than all the others are:

1. The cryptographically tamper-proof append-only PCC build transparency log

2. Publicly accessible and reviewable PCC build images

3. Tools for analyzing PCC build images

The fact that Apple is going out of their way to allow public security and privacy researchers to easily analyze, and cryptographically authenticate, their PCC builds is almost unheard of in the industry.

Apple is basically saying they trust that no one will trust their word alone, and in order to properly provide assurance around the security and privacy “promises” they’re making, they have to be radically and robustly transparent about their new cloud-based AI features.

Admittedly, I might be a bit overly excited and not skeptical enough about what Apple has done here.

So I’m curious:

• What do you make of Apple’s PCC architecture?

• What do you find impressive about it?

• Where do you see it lacking in crucial SbD and PbD features?

• What other residual risks and concerns do they still need to address?

While working on part 3 of this series (title TBD), something strange happened in the world of vulnerability management: NIST’s NVD quietly posted a vague notice on their website about “delays in analysis efforts” due to time they’re spending establishing a “consortium to address challenges in the NVD program.” If you want to learn more about what is going on here, check out ’s really solid overview of the whole situation, including NVD outsiders’ research and observations that shine a light on the kinds of “challenges” the NVD is facing.

In this interim part 2.1 article, I’m going to explore an idea I started discussing in Dan Lorenc’s LinkedIn post on this subject: that the NVD’s recent service degradation might be a blessing in disguise, an opportunity for the InfoSec community to take a step back and seriously ask ourselves “is this really the best we can do with vulnerability management? What could or should  ‘better’ or ‘best’ look like for vulnerability management?”

Let’s dive in to see what we can make of this!

Current vs. future state of our vulnerability management ecosystem

The world and practice of vulnerability management is incredibly, and almost exclusively, CVE centric. While alternative vulnerability databases exist, like GitHub Security Advisories (GHSA), Snyk’s SNYK vulnerability database, and the Open Source Vulnerabilities (OSV) project, they merely complement (and synchronize CVE data from) NIST’s NVD. All of our vulnerability management tools, “best” practices, frameworks, etc. are built on the foundation the NVD provides.

I’m a very visual thinker and find it really helpful to draw out really complex ideas and concepts to better demonstrate and capture what they’re all about. To that end, I took a stab at drawing out what I see as the key or notable ingredients that constitute our CVE-centric vulnerability management ecosystem.

Geez, I may as well have drawn out the digital equivalent of a rat’s nest, yeah? What an absolutely overcomplicated mess this is! Forget risk-based vulnerability management (RBVM) - with this many moving parts, we should call it “Rube Goldberg Vulnerability Management” (RGVM). 

🥁

*insert laugh track*

We have the CVE Organization - now independent of, but still highly reliant on, MITRE - as well as FIRST and NIST’s NVD basically propping the entire thing up. We’ve got at least three, if not four, distinct types of tools that are used to drive remediation of vulnerabilities: vulnerability scanners, ticketing systems, SOAR or other security automation tools, and patch management tools.

I loathe this, if you couldn’t already tell. This is an absurdly complex and inefficient system for a practice of information security that strives to “find and fix vulnerabilities before attackers exploit them.” No wonder attackers are mopping the floor with us, week in and week out. We can’t get out of our own way!

So, what might it look like if we instead reimagined this to be a patch-centric vulnerability management ecosystem, one that strives to “keep software current with the latest security patches” as the primary means to fix vulnerabilities before attackers exploit them?

Look closely at this and think about it for a few minutes. 

Why wouldn’t this work? 

Why wouldn’t this work as well as our current CVE-centric model? 

Try to envision asset owners leveraging a predictable, structured, and rigorous approach to proactively applying patches using tools like Vicarius (or even free tools like App Auto-Patch) for Windows, macOS, and Linux systems, or Seal Security for precise security patch updates for third-party libraries, or Moderne for high-confidence code refactoring workflows to speed up MTTR for third-party library and framework updates, or Chainguard for already-fully-patched-for-you container images.

MacAdmins has already brought the idea of a patch-centric data feed to life with their SOFA project.

Try to envision how rigorous patch testing and quality assurance practices (drawing inspiration from modern software engineering CI practices especially) could allow us to patch proactively without needing to pick and choose patches based on complex risk-based triage logic and statistical prediction models (as cool as those are!). I’m talking about practices like regression testing and canaries rooted in strong observability and “failed patch” alerting logic. 

Additionally, there are newer tools like trackd that provide historical patch quality analysis that make it easier to understand which patches are going to cause problems that you should exclude from your proactive patching cycle until you’ve had time to test them more rigorously yourself. Rather than picking and choosing which patches to deploy, tools like trackd allow you to pick and choose which patches to exclude.

I recently started referring to this overarching approach to rigorous patch testing and QA practices as Fearless Patch Management^TM (Gartner, you need to make this a thing!!!) because that’s what we should be talking about as the solution to the long-running and pervasive culture of “fear” about patches breaking things, with this fear being used as an excuse to only selectively patch vulnerabilities that defenders think attackers are going to exploit.

…

If this still doesn’t make sense as a better alternative to a CVE-centric vulnerability management ecosystem, what different ideas can you imagine that would allow organizations to much more efficiently, effectively, and confidently apply security patches on a continuous basis?

What would we need to do differently, or what tools would we need to build, or what new best practices would we need to invent, to make this the ecosystem that makes the vulnerability management world go round?

I’d love to get a discussion going around what a better future state of vulnerability management looks like that leans far more heavily into proactively fixing vulnerabilities - or installing security patches - and leans far away from reactively fixing only a tiny subset of vulnerabilities based on increasingly-complicated, nuanced, and presumptuous triage logic.

A quick side note about software supply chain vulnerability management concerns

What if the NVD actually did fizzle out or become obsolete due to concerns around the quality of their data and their ability to maintain it?

What would an NVD-less world of vulnerability databases look like? And how would this impact software supply chain security practices?

Well, I’ve got a picture for that!

Makes sense, right? In this context, SBOMs (and supporting data provided by VDR and VEX files) play an integral part in supporting a decentralized vulnerability database ecosystem. 

This would be more complex for sure and would require more intentional cooperation and partnership between vulnerability database providers to ensure consistent and high-quality data. Some sort of data synchronization and federation model could make this work at scale.

It would remove our reliance on a single centralized highly-bureaucratic source for vulnerability data. But maybe it could work? 

Concluding thoughts

I don’t doubt that NVD will bounce back thanks to a consortium being formed that will allow it to operate more effectively and keep pace with an ever-growing list of newly published CVEs. However, this latest industry issue is as good an opportunity as ever for our profession to take a step back and seriously try to figure out what a better approach to vulnerability management can and should look like.

The sooner we figure that out, the sooner we can bring it to life.

Let’s go!

Update: part 2.1 is out in response to the NVD’s February 2024 service degradation announcement.

In part 1 of this article series, I opined on how vulnerability management programs tend to be heavily focused on reactive strategies with minimal focus on proactive ones. I proposed an opinionated proactive strategy that I dubbed Proactive Vulnerability Patch Management (PVPM). I even attempted to propose a risk-based prioritization framework to establish how an organization might start ramping up with PVPM, which I shamelessly dubbed Stakeholder-Specific Patching Prioritization (SSPP). 

I can only imagine how giddy with excitement Gartner must be with these new ready-to-go cybersecurity marketing acronyms. But what really matters is what InfoSec practitioners make of these ideas, and make of these ideas they did! Below is a sampling of the feedback I got from my fellow InfoSec-ers that resonated with me the most:

Is it considered reactive to fix issues before they are exploited? How do you suppose this gets more proactive? Fixing issues before they arise? Yes. This is security by design. But that's a different domain and different department. Vulnerability management is supposed to identify issues and get them fixed before they are exploited.

One thing your blog post is missing that I'm excited about is secure-by-default. It's *so* much easier when developers are empowered to get the secure configuration from minute zero easily.

It's not either/or, it's both proactive and reactive, every day!

I agree with you that there is too much unused software and we should identify and remove as much of it. But unfortunately it gets much more nuanced. Did you know, while we keep talking about how 80-90 percent of your modern app code is open source, only 12% of the functions/lines-of-code within those libraries are actually used

But what do you expect an engineer to do? They can't simply remove the package because they are using 10 lines of code within it.

In part 2 of this article series, I’m going to do my best to expand on these points, as well as a few other top-of-mind ideas I have about what PVPM can look like in practice.

Let’s dive in!

Proactive vuln management === secure-by-design/default?

Strictly speaking, secure-by-design/default (SBD/D) strategies applied to vulnerability management are necessarily proactive (i.e. entire classes of vulnerabilities are avoided with strong secure SDLC practices, configuration management, etc.). However, there’s more to proactive vulnerability management that secure-by-design/default approaches don’t necessarily account for.

SBD/D approaches applied to vulnerability management can make it so that, on day 1 of a system being provisioned, it’s free of vulnerabilities. However, over time a system’s vulnerability state will change as new vulnerabilities in existing software or configurations are uncovered by researchers and attackers. 

If one were to apply a reactive vulnerability management approach in this scenario of using secure-by-design/default systems on day 1, it might look like:

• Scan running systems for vulnerabilities

• Analyze and triage subset of vulnerabilities to remediate

• For mutable systems: schedule & trigger patch deployments for running systems

• For immutable systems: manually trigger re-deployment of vulnerable systems using latest vulnerability-free image/template

Conversely, a PVPM approach would look more like this:

• For mutable systems: enable auto-patching setting/service/tool to check for and apply latest security updates on a regular basis with scheduled process/system restarts as needed

• For immutable systems: schedule systems for automated re-deployment on a regular basis (weekly, monthly, etc.) using latest vulnerability-free image/template

Notably, PVPM doesn’t rely on those first two steps of vulnerability scanning → analysis/triage to drive remediation in already-provisioned systems. PVPM and SBD/D partially overlap with how they help with vulnerability management. But SBD/D doesn’t necessarily extend to post-day-1 of system provisioning.

You could argue that what I’ve described above is the definition of SBD/D, with PVPM merely being a feature of a secure-by-design system architecture. I’m not gonna argue with that because, well, it’s true!

I believe it’s beneficial, though, to be explicit about the distinction between the overarching umbrella category of SBD/D and the much more domain-specific and prescriptive concept of PVPM, such that we as a community are much more focused on figuring out practical ways to implement PVPM processes and tools vs. reiterating abstract InfoSec platitudes. Simply saying “well, that’s just SBD/D, which everyone should be doing anyway” doesn’t progress the conversation forward or drive the invention of more effective vulnerability management practices.

Alright, let’s take a break from acronyms for a moment and talk about proactive vs. reactive vulnerability management.

Reactive vs. proactive: why not both?

Indeed: why not?! While this may not have been clear in part 1 of this article series, I do believe that both reactive and proactive approaches are necessary ingredients for a strong vulnerability management program. Even if we do a perfect job at being proactive, it’s inevitable that we’ll need to keep being reactive, in large part due to 0-day vulnerabilities. 

But let me reiterate my core point here: we currently invest far too many resources in reactive approaches at the expense of proactive ones. We have literally hit a point of diminishing returns with reactive vulnerability remediation, as demonstrated by Cyentia’s research on the subject: 

The typical organization only fixes about 10% of its vulnerabilities in any given month. And that’s consistent regardless of how many assets are in the environment.

The imbalance between reactive vs. proactive vulnerability management

In my mind, there is a significant imbalance around how organizations apply reactive and proactive vulnerability management approaches. In the context of a “vulnerability management funnel” that would make any Gartner analyst swoon from pure envy, this imbalance looks something like this:

Even with reactive remediation capabilities maxed out, an average organization will only be able to fix ~10% of vulnerabilities in their environment per month. At best, an above average vulnerability management program will have automated patch management in place for their easy-to-patch mutable systems, like end user compute devices running Windows, Mac, and (occasionally) Linux. This might allow them to fix ~15% of vulnerabilities in their environment per month.

In these cases, though, it's likely that organizations are still engaging in reactive vulnerability remediation: patch management tools are configured to automatically deploy patches only when certain types of vulnerabilities with specific attributes are detected. Sometimes this reactive scan → triage → patch approach is fully handled by the patch management tool itself (which probably won't have the most comprehensive vulnerability database) or it’s triggered by a vulnerability scanner that integrates with said patch management tool (which introduces complexity-induced issues, like inconsistencies with normalizing and mapping data between tools, or each tools' vuln database not consisting of the same CVEs, etc).

Immutable systems are a whole different story. For all the benefits immutable systems provide, they are undoubtedly a vulnerability management footgun. Need to remediate a single vulnerability in an immutable infrastructure environment? Awesome! Just follow this super-fast-and-simple, definitely-not-cumbersome process that flows across three distinct systems (SCM → CI → CD):

1. Kick off base image creation process with vulnerability remediation technique applied

2. Update CD pipeline and runtime scaling configs to utilize new base image

3. Never forget: they built it, they own it! “They” being software developers (i.e. service owner teams). So, you’ll now make a heartfelt plea to them to re-deploy their service utilizing this new base image at their earliest convenience, ideally sooner than later so InfoSec will stop nagging you

4. Respond to a bunch of pages when stuff inevitably hits the fan

5. Rollback these changes and defer this pesky security work to Q5 20never.

Sorry, I lied: this is clearly the dictionary definition of cumbersome. The inherent difficulty of even reactively remediating vulnerabilities in these types of environments results in a lot of avoidance with having to engage in this process at all. Over time, immutable infrastructure ends up being riddled with a lot of lingering vulnerabilities that get patched occasionally or only when push comes to shove.

Don't get me wrong: it's 1000% possible to streamline this redeploy-to-patch process. Some organizations have automated this process to a large extent, even going as far as proactively redeploying services on a regular basis with the latest patched base image. But it requires a concerted and coordinated effort for organizations to get to this state of maturity and, as far as I know, few organizations have achieved this state.

Then there are self-hosted vendor software products. I’ve rarely seen or heard of any organization engaging in proactive remediation for these types of systems. Maybe that's part of the reason why there have been so many high-profile breaches caused by exploitation of these vuln-infested products. Think about how awful 2023 was for organizations that were running even just a few of the self-hosted vendor products that were ruthlessly exploited last year. I wouldn’t be surprised if a good number of organizations were using 5 or more of these products last year and had to deal with regular emergency patching exercises that burned out their teams:

It doesn't help that the update process for each vendors’ product vary wildly: some allow automated updates to be enabled, others don't; some require you to login to a support portal site and dig through 5 layers of menus to download an installer locally and push it to your devices with a proprietary GUI tool; others recognize you’re a living breathing human and would never subject you to this level of cruelty.

And it especially doesn't help that there aren't really any vendor-agnostic patch management tools that make it easy to centrally and automatically deploy patches across such a diverse set of vendor products. A typical organization might have one vendor for their file transfer system, one for their next gen firewalls, one for their VPN, one for their virtual desktop infrastructure, one for - ok, you get the idea. 

Sounds like the perfect kind of problem for a security startup to tackle?!

But I digress…

Re-balancing our focus on proactive vulnerability management

Back to my original premise: what if we pivoted our focus toward maxing out our proactive vulnerability management capabilities first and foremost? Yes, we still need to do reactive vulnerability management. Yes, active mitigations are still a necessary part of our strategy. Yes, we’ll probably still have residual vulnerabilities in our environment. 

However, if we do this right and we do it well, we should have far far fewer residual vulnerabilities lingering in our environment with a lot of time savings to boot. Behold: yet another Gartner-analyst-swoon-inducing vulnerability management funnel:

Imagine if we lived in this world instead of the one we currently occupy, a world where we only need to perform reactive vulnerability management in the context of zero day vulnerabilities and new classes of misconfigurations.

MOVEit-style mass exploitation events lasting months quickly become a thing of the past. When active exploitation of an N-day vulnerability starts to pick up, most organizations would have already proactively installed the latest security update without waiting to sift through vulnerability scan results before triggering their emergency vulnerability incident response process. 

Would this state be very hard to achieve? Heck yes. 

Would it be worth it? A thousand times yes. 

Is it possible to do? I believe so, yes. Think about this: in the 1960s, using 1960s-era technology, we strapped a few humans inside a metal box, precisely launched them thousands of miles to the Moon, had them bop around on its surface - in hard vacuum -  using “portable atmosphere” suits, and then safely brought them back to Earth by (slowly) crashing them into the ocean. In the 1960s. 60 years ago.

By comparison, this should be a walk in the park.

In fact, we’ve already started making progress in this direction as an industry. Proactive vulnerability management tools exist today. 

Let’s take a look at some of them, shall we?

Proactive vulnerability management tools: some examples

Disclaimer: I’m about to discuss some open source and commercial tools, largely from a point of excitement about what I currently see as useful examples of tools that fit the PVPM model. I was not compensated in any way, shape, or form for discussing these tools in a positive light. Likewise, these are personal opinions, not professional endorsements, and as such you should use your judgment when exploring or utilizing these tools.

Containers

Chainguard: someone call Gartner and tell them it’s time to coin the term “Proactive Vulnerability Annihilation (PVA)” because that’s exactly the capability Chainguard provides as the de facto visionary leader in this totally-made-up-just-for-this-article product category (credit goes to Jordi Mon Companys for inspiring this name). Chainguard Images are the best example of Proactive Vulnerability Patch Management (PVPM) in practice: unused software components are stripped out or not added in the first place, and all known CVEs for included software components are patched before a container image is published. 

In other words: Chainguard takes a highly disciplined and methodical approach to proactively yeeting all known CVEs into /dev/null each time they produce a new freely-available, zero CVE container image. It’s like vulnerability management magic!

(No, they didn’t pay me to write this (yes, they probably should have (lolol I’m just kidding, Dan!!(…or am I?))))

General purpose OSes

macOS

Patchomator & App Auto-Patch: both of these open source tools automatically install the latest updates for popular macOS apps without giving a hoot about what your vulnerability scanner has to say about detected CVEs. While these are easy to run locally, they are also designed to be deployed via Jamf other MDM tools. App Auto-Patch also has a nice SwiftDialog UI/UX that can be customized to be fully interactive for end users, partially interactive, or totally silent. They both use Installomator under the hood for consistent and secure automated installation of the latest version of hundreds software packages. Hooray for open source and the Mac Admins community!

Windows, macOS, Linux

Vicarius: they're a newer player in a space formerly dominated by BigFix and probably-currently being dominated by Automox. I hadn't heard of them before until someone mentioned them in response to part 1 of this article series (thanks, Michael!)

While Vicarius doesn’t necessarily focus on the “removed unused software” part of PVPM like Chainguard does (although I’m sure with some fiddling you could make their automation functionality do this), they seem to have a strong approach to proactively patching a lot of different software across these three OSes. At the time of this writing, they support patching  523 Windows apps (including common enterprise software like Splunk, presumably the Splunk agent), 646 macOS apps, and 2077 Linux apps. I suspect they're piggybacking off well-established package managers, but hey, good for them if that's the case! No need to reinvent the wheel here. 

As an added benefit, they also have some active mitigation features called “Patchless Protection'' which especially helps with 0-day scenarios when a patch isn't yet available. It seems IPS-esque, as if it’s…extending traditional IPS functionality to cover specific…endpoint software…like some sort …I dunno…eXtended Endpoint IPS…an…X…E…IPS? 

An XEIPS?!

We could pronounce it as “ZEE-ps”!!!

(God I hope someone from Gartner is reading this…)

trackd: even newer than Vicarius is trackd. They are the closest thing to a PVPM-centric tool I’ve ever come across: they recognize that the root cause of poor patching practices is fear of patches breaking things. Thus, their key selling point is crowdsourcing telemetry about patch stability to allow you to engage in what I like to call “Fearless Patch Management.” For that reason, a basic version of their product is free and will always be free - give it a shot!

I’m really excited about how trackd can help drive organizations toward fearless, proactive vulnerability patching. Definitely keep an eye on them!

Source code

Ok, here’s the thing about first-party and third-party code remediation, which every software developer who’s ever had to deal with it knows: it’s an absolute PITA. And while aspects of PVPM like systematically removing unused software makes sense in the context of third-party libraries, it doesn’t really apply to first-party code (unless you’re writing a bunch of vulnerable code that end users would never interact with but attackers could have a field day with, in which case you should probably be banned from using the internet until the end of time).

Likewise, trying to proactively auto-remediate vulnerabilities in third-party source code is a fool’s errand like none other: unless a fix path for a third-party library includes a security patch version, minor and major version updates tend to contain breaking changes that require time intensive analysis, code refactoring, and test suite updates to avoid.

So, it’s pretty safe to say that PVPM just isn’t applicable to the domain of source code vulnerability management.

…

……

………………

………………………………

………………

……

…

PSYCH!

PVPM is totally relevant to source code vulnerability management. There are at least two new products on the market that can help implement PVPM (insofar as it’s possible to do so) for code vulnerability management without it being a total PITA. 

From what I can tell, these two products differentiate themselves from existing proactive code vulnerability management patterns and tools, such as:

• Secure code libraries that make it easy to avoid entire classes of vulnerabilities (SQLi, XSS, XXE, etc,) by providing securely-defined classes, methods, functions, etc. to developers

• SCM and CI integrations that block commits and fail build jobs when certain vulnerability conditions are met

Let’s take a look at how these tools uniquely contribute to a PVPM model for code!

Moderne (and OpenRewrite): Yes, existing software composition analysis (SCA) tools like Snyk and Dependabot can automatically open PRs to update your third-party libraries, and even static application security testing (SAST) tools like Semgrep have “auto-fix” recommendation engines that will use regex matching to make it easy to find-and-replace vulnerable first-party code with remediated code. However, these tools are by default reactive: first, you scan your code for known vulnerabilities - then you remediate them. 

Moderne takes a different approach. First, it uses a novel code data model called Lossless Semantic Trees (LST) to perform highly accurate, format preserving, type aware, error tolerant code refactoring when remediating vulnerabilities. This is especially valuable when doing version migrations of runtime environments and frameworks (which is necessary sometimes to remediate vulnerabilities or add new security features), across multiple teams’ code (which is a huge pain-in-the-backside change to try to drive). This fits into the PVPM framework in my mind when you start to recognize that these major version migrations can now be done faster and with more confidence, making them easier to do proactively well before a runtime environment or framework end-of-life date comes about.

Because Moderne uses LSTs, it’s in a much better position to thoroughly and accurately understand how code is utilized throughout an application across multiple codebases. This presumably is what makes it possible for them to have a pretty accurate “remove unused imports” recipe, which fits in nicely with “remove unused software” stage of our PVPM framework.

There is at least one big drawback about Moderne currently: their vulnerability detections and insecure code remediation “recipes” aren’t nearly at parity with your typical SCA and SAST tool. Granted, Moderne is very new, so given enough time and intentional focus, they could certainly build out a rich library of recipes that make it easy to proactively auto-remediate vulnerabilities at scale in ways that existing SAST and SCA tools might struggle with doing.

But don’t take my word for it, either way! Test out Moderne yourself using their free community edition.

Seal Security: this company is basically a software security superhero. They are diving headfirst into other people’s open source codebases and very precisely patching (“sealing”) vulnerabilities in them, even across multiple older versions of a given codebase. Given how new of a company they are, their security patch version repository is pretty well built out, supporting multiple popular languages with a good number of patches for Node libraries especially.

Then they have their seal CLI tool that makes it easy to patch a vulnerable library on demand or on the fly. This makes it much more feasible to proactively patch third-party libraries, knowing that you’re only updating to a patch version where the only code change is a bespoke vulnerability fix, not a breaking change.

Seal Security is making it so that we can have our “third-party code vulnerability auto-remediation” cake and eat it too.

Thank you, Seal Security, for saving the world from insecure software, one patch at a time 🫡

Concluding thoughts

I hope this part 2 article of “Are we doing vulnerability management all wrong?” provided something of value for you. I’d love to get your feedback or perspective on what I’ve laid out here, especially if I misrepresented or misunderstood anything I’ve said!

Coming up next: part 3 where I cook up a hot take about risk-based vulnerability management and where I see it missing the mark. I may even offer up a universal vulnerability risk rating model that more precisely accounts for realistic risk attributes that more directly correlate with attacker behavior and motivations. Likewise, I’ll sprinkle in some thoughts on risk-based remediation SLAs.

Stay tuned!

Share

Update: part 2 is out!

Is it just me or is vulnerability management having a renaissance moment? CVSS 4.0 is hot off the presses, EPSS 3.0 was released earlier this year, and SSVC is finally getting the attention it deserves as a meaningful alternative to CVSS.

And yet, even with all of this innovation happening around vulnerability prioritization methods, truly efficient and effective vulnerability management still seems out of reach for many.

Why is that? 

Is vulnerability management a fundamentally intractable problem with an upper limit on the proportion of vulnerabilities any organization can fix? 

Is it destined to be a game of cat and mouse that attackers tend to win in the long run?  

Or, perhaps, are we approaching vulnerability management all wrong?

Background

I’ve spent the past 11+ years focusing on vulnerability management from various perspectives: security operations, application security, cloud security, network security, and GRC. In that time, through direct experience, learning from others, reviewing research, and experimenting with industry best practices, I’ve come to the conclusion that we’re approaching vulnerability management all wrong.

Ok, I’m being slightly hyperbolic. But in all honesty, we as an industry seem excessively focused on reactive vulnerability management strategies that amount to peering into a high-tech crystal ball to predict what vulnerabilities on which of our assets are most likely to be exploited and result in the worst impact, and then focusing our remediation efforts solely on those vulnerabilities. 

This bias toward reactive vulnerability management is as much understandable as it is untenable: remediating vulnerabilities is a painstaking process, and no organization’s mission is “keep our sh!t patched and vulnerability free” - indeed, far from it. However, I contend that we have overcorrected our focus on reactive strategies at the expense of proactive ones, resulting in remarkably bad vulnerability management outcomes over the years.

So, let’s talk about what proactive vulnerability management is and what else we can do to innovate it. In this article, I’m going to describe:

1. The key differences between reactive and proactive vulnerability management

2. An example framework and strategy for proactive vulnerability management 

I plan to release future articles that cover additional topics, such as how to better balance and integrate reactive and proactive vulnerability management strategies, examples of specific techniques and tools to use to put these ideas into practice, etc.

Until then, let’s dive in!

Reactive vs. Proactive Vulnerability Management

Reactive vulnerability management is the approach that the growing array of four letter acronyms and proprietary vulnerability risk scoring methods are propping up. For many security practitioners, it is the all-too-familiar Sisyphean hamster wheel of scan -> triage -> report → patch that never seems to truly move the needle over the long run. 

It posits that because we’ve tried and failed over the years to “patch all the things”, we must settle on only patching what we think attackers are most likely to exploit on systems that we as defenders believe will be impacted the worst. This reactive approach is commonly referred to as “risk-based vulnerability management.”

Proactive vulnerability management, on the other hand, is any activity that remediates vulnerabilities without waiting to scan, triage, or report on them. It posits that, over time, you can patch just about every piece of software that can be patched.

At its core, this concept isn’t all that groundbreaking: it ultimately boils down to solid patch management, configuration management, and system hardening. InfoSec practitioners have long talked about the fundamental importance of each of these, within and outside the context of vulnerability management. These aren’t new ideas by a long shot.

Unfortunately, I don’t see us as an industry investing nearly enough time and energy into innovating in areas of proactive vulnerability management. This lack of innovation is holding us back from being able to consistently fix vulnerabilities faster than attackers exploit them.

New Ideas for Proactive Vulnerability Management 

While patch management, and especially automated patching, is often mentioned in discussions about how to do vulnerability management really well, I have rarely come across any discussions about how to do patch management really well. Sure, there are well-known vendor tools in this space, such as SCCM, BigFix, and Automox. However, these are just patch automation and deployment tools that don’t have much of an opinion about how you ought to do patch management. More often than not, they simply reinforce reactive vulnerability management by focusing your patching efforts around known CVEs, CVSS scores, etc.

So, what would it look like to innovate in this area to modernize patch management with a much more proactive focus?

Proactive Vulnerability Patch Management Lifecycle (PVPM)

This wouldn’t be a thought leadership article about vulnerability management without another four letter acronym! Joking aside, what I refer to here as “PVPM” is how I envision proactive vulnerability management at a high level.

It’s a continuous lifecycle that is intended to produce high quality and highly effective auto-patching workflows that fix vulnerabilities fast without waiting to scan for them. 

It’s a framework meant to enable organizations to consistently beat attackers at the vulnerability management game by driving a threat-centric focus around what to start auto-patching, when, how, and how often.

Automation is a must for this to work and is intended to be a key aspect of each stage in every phase of this lifecycle.

Let's break down what each phase and their corresponding stages entails.

Inventory

Like any good security practitioner knows, you can’t secure what you can’t see. Whether we’re talking about desktop software, server software, application libraries, container images, or anything in between, having a complete and continuously updated software inventory is absolutely essential to understand what needs patching.

• Usage Analysis: this step of the inventory phase is all about understanding how software is used on systems. Is a particular piece of software running constantly, day in and day out? Or does it only run when a user launches it as needed? Or, perhaps, has it never run and thus is worth removing from a system entirely, permanently and proactively shrinking its vulnerability attack surface?

Prioritize

For all the software you’ve inventoried, the prioritize phase is all about figuring out which software is inherently most at risk of having vulnerabilities in it that get exploited by attackers. 

In other words: whether you already have some software being regularly patched or you're starting entirely from scratch, you should focus your efforts on incrementally building new auto-patching workflows for software based on the following vulnerability attributes. 

• Attack History: has this software ever appeared on CISA’s KEV or other “actively targeted” vulnerability intel lists? Attack history is a strong signal that a piece of software has been, and likely will continue to be, an attractive target for attackers when conducting real-world attacks.

• Exploit History: has this software ever had exploit code or methods published for vulnerabilities associated with it? The existence of publicly known exploit methods doesn't always mean attackers have bothered targeting their vulnerability exploitation efforts at a piece of software; however, it does serve as a good signal of software that is more likely to end up in attackers’ cross-hairs in the future since vulnerabilities with known exploit methods are easier to target than those without them.

• Vulnerability History: has this software ever had vulnerabilities published as CVEs, GHSAs, etc? Compared to the other two attributes listed above, this is the weakest prioritization signal to use. However, software with a much longer history of published vulnerabilities seem to have a generally higher likelihood of being targeted by attackers compared to software with less or no vulnerability history (at least, according to my interpretation of the research behind EPSS).

• Exposure Analysis: does this software process user-controllable data, and if so, from where? For example: a vulnerability that exists in a software component that processes user-controllable data that is ultimately received from the internet is most exposed (and thus most easily exploited) compared to one that processes machine-produced data from a local file system.

Patch

This is where all of the fun happens. This phase is intended all about frequently checking for new security updates and applying them as soon as they’re available. 

• Create Tests: in my experience, patch testing is one of the least rigorous and most underdeveloped aspects of patch management. Raise your hand if your organization’s approach to patch testing for things like end user devices is “roll out patches to 5% of users and wait to see if anyone files any tickets about BSODs / kernel panics.” Compared to modern approaches to software testing (unit testing, regression testing, test coverage metrics, etc.), testing in the context of patch management is generally immature - but it doesn’t have to be! We can apply these same modern software testing paradigms to patch management to ensure we are rigorously, and automatically, vetting the success of a newly installed patch before deploying it to the masses.

• Run Tests: once you’ve created your (automated) tests for your the software you’re creating an auto-patch workflow for, it’s time to…well, test your tests! Make sure they’re successfully detecting evidence of bad patches.

• Deploy Auto-Patch:  once your patch tests pass, it’s time to #shipit. If you’re patching a large fleet of similar assets (e.g. end user computers, production compute servers/containers), use deployment rings and canaries to incorporate quality control mechanisms and feedback loops for detecting bad patches that your tests may have missed. When patching-related issues are detected or suspected, halt and, if possible, rollback patches.

I suspect that the “Inventory” phase and “Patch” phase are pretty straightforward for most folks, but the “Prioritize” phase might be more obscure. To better clarify what the “Prioritize” phase entails, I’d like to introduce a patching prioritization decision making model called…

Stakeholder-Specific Patching Prioritization (SSPP)

Sorry, I had to! There’s been enough innovation in the realm of four-letter acronyms for vulnerability management concepts that I’m going to shamelessly copy SSVC for naming this next framework as SSPP. For all of my software development friends out there who seize every opportunity to optimize and compress information, you may simply refer to this as s2p (just kidding, please don’t).

The crux of this framework is to drive focus around which software to build auto-patching workflows for and in what order. The intent is that once you've created a reliable auto-patching process for a piece of software, it will ensure that the latest security updates for a piece of software are proactively installed soon after they're made available.

Unlike SSVC, the decision points in this model are intended to be as simple as possible (but no simpler) to facilitate efficient yet still effective prioritization of creating auto-patching workflows. There is still room for additional prioritization criteria to help deal with “tie breaker” scenarios (e.g. “I’ve got 100 pieces of software that fall into the P1 category - which of those should I focus on first?”). 

Here’s an example prioritization matrix based on these attributes. If you’re starting from scratch and you don’t have any auto-patching workflows created, this model is intended to guide your focus in choosing which software to create auto-patching workflows for before others.

Attack History Analysis Example

Let’s step through an example to see how finer-grained prioritization criteria at the “Attack History Analysis” stage helps with providing a risk-based approach for building auto-patching workflows. Again, the idea behind PVPM and SSPP is that all of this analysis and decision making needs to be automated for it to scale efficiently and effectively.

Let’s assume for a moment that 

• Our organization has every software product actively in use that is mentioned in CISA’s KEV list (see this snapshot of data I set aside for this example). 

• We don’t have auto-patching in place for any of them

When examining this data in the aggregate, we can quickly see which software product to focus our auto-patching efforts on first, second, third, etc.

Thankfully, software products like Windows, iOS, macOS, and Chrome have reliable built-in auto-patching mechanisms that we can enable to implement some basic auto-patching in our environment. Also, security patches for these products tend to be very stable, so we can safely skip implementing rigorous patch testing for them (for now).

But what about Cisco IOS, Oracle WebLogic, and Apache Struts? Auto-patching these components is unfamiliar, and even scary, territory for us. However, that fear shouldn’t stop us from defining and implementing a rigorous and automated patch testing and deployment process for them. Now that we’ve implemented basic built-in auto-patching for the other higher risk software in my environment, we’ve bought ourselves more time to focus on creating and rolling out rigorous auto-patching for these trickier-to-patch systems.

Proactively building out auto-patching workflows for these trickier-to-patch systems puts us in a much stronger position to engage in emergency patching in the event a new actively exploited vulnerability is discovered in the wild for them. Rather than engaging in a panicked, rushed, and error-prone reactive “emergency” vulnerability patching process, we can trigger our auto-patching workflow on demand to confidently and quickly beat attackers to the punch.

Concluding Thoughts

While I personally think these proactive vulnerability management ideas could transform the state of vulnerability management for many organizations, I also know that I’m only one person with a limited set of experience and perspectives. Before writing further on this topic, I’m hoping to get feedback from and spur a discussion within the InfoSec community about these ideas. 

What aspects of these ideas make sense? What doesn’t make sense? How could they be improved upon? 

I’d really love to hear feedback from folks about this!

Thanks for reading :)

Let’s dive a bit deeper into how best to combine and use these ingredients for hardening your online accounts in the event your password manager (or really any of your passwords in general) gets breached.

Harden your password manager and online accounts 

Here’s a summarized checklist of steps you can start taking to harden your password manager and online accounts. I’ve also ported this over to GitHub in an actual checklist format and also to encourage folks to submit pull requests for additional ideas or edits. 

NOTE: As mentioned in part 1, for the sake of simplicity and security, this guide assumes you’re only using Apple devices. I'm also assuming you’re using 1Password, BitWarden, Authy, and 2 YubiKeys.

Securely setup 1Password

Setup 1Password for creating and storing passwords and password equivalents, such as security question answers. Especially in light of LastPass’ latest incident update, 1Password’s Secret Key feature, which is built-in at the moment of account creation, guarantees universal protection against offline brute force attacks.

• Use a unique passphrase for logging into 1Password

• Setup MFA with YubiKeys only

• Store your 1Password passphrase in a secure place (fireproof safe = better security, Apple Keychain = better usability)

  • Your 1Password Secret Key can be stored in the same place since it's designed to be an entropy-boosting addition to your password

Harden foundational accounts

These are accounts that, if compromised, could be used to reset account passwords or access Passkeys.

• Ensure email account has a strong password and MFA setup with YubiKeys

  • Use Gmail (Google’s security is top notch). If you're extra paranoid, use Protonmail

• Ensure Apple ID has strong password and MFA setup with YubiKeys

Protect against SMS MFA code theft

Some of your accounts may still only support SMS MFA codes (and any form of MFA is better than no MFA). Similarly, some sites and apps, such as Authy, use mobile phone numbers for account creation. Hardening your mobile phone number and account is essential in these contexts.

• Create a strong password for your mobile carrier online account

• Setup MFA on your mobile carrier online account

• Setup a mobile carrier PIN to protect against SIM hijacking (Verizon, T-Mobile, AT&T, Sprint, Cricket)

• Setup Google Voice number to be used for SMS MFA (you can disassociate it from your mobile phone number later on for added security)

  • Ensure Google Account has strong password and YubiKey-only MFA enabled

Securely setup Authy

Use Authy whenever time-based one-time passcode (TOTP aka “Google Authenticator”) MFA is the most secure option available. If you’re extra paranoid and are willing to make some usability sacrifices, try storing your MFA codes in a YubiKey and use Yubico Authenticator to access them.

• Setup Authy using your Google Voice number

• Enable Authy Backups and create/store a randomly generated Backup Password with 1Password

• Disable Authy Multi-Device access and only re-enable when you're setting up Authy on another device

  • If you are able to, install Authy on at least two devices so you don't have to go through a painful account recovery process if your only Authy-installed device breaks, is lost, etc.

  • If you’re extra paranoid and are willing to make some usability sacrifices, try storing your MFA codes in a YubiKey and use Yubico Authenticator to access them

Securely setup BitWarden

Setup BitWarden for storing TOTP MFA recovery codes and MFA recovery code equivalents such as seed keys.

• Setup a BitWarden account using an email address from a different provider

  • E.g. if you used Gmail for your 1Password account and other accounts, use Protonmail for your BitWarden account

• Use a unique passphrase for logging into BitWarden

  • Store your BitWarden passphrase in a secure place (fireproof safe = better security, Apple Keychain = better usability) but not in your 1Password vault

• If you're already paying for BitWarden, setup MFA with YubiKeys only. Otherwise, setup TOTP MFA codes using Authy

Enable MFA everywhere you can

Don’t delay, enable MFA today! This is essential for protecting your accounts in the event their passwords are compromised.

• Use 1Password Watchtower to identify accounts that support MFA and ensure MFA is setup on all of  them

• Explore 2fa.directory to identify additional accounts that support MFA and ensure MFA is setup on all of them

Change compromised and vulnerable passwords

• Use 1Password Watchtower to identify passwords of yours that have been caught up in past data breaches

• Change each compromised or vulnerable password

Change weak and reused passwords

• Use 1Password Watchtower to identify weak and reused passwords of yours

• Change each weak or reused password

• Replace passwords with Passkeys (WebAuthN) where possible 

Detect account compromises as they happen

• Setup email rules for new device login, suspicious login, password reset, and MFA change notifications

• Start treating rogue SMS MFA codes and push notification MFA prompts with suspicion, changing passwords when benign cause of rogue codes/prompts can't be identified 

  • Especially if you receive a message from someone who claims to need your MFA codes or for you to approve push notifications. If this happens, change passwords and report to your account provider ASAP

That’s it for now. When all is said and done, this is what your password manager and online account security “architecture” will look like:

Stay tuned for part 3 of this series where I’ll cover how password manager developers can secure their software development lifecycle (SDLC) and cloud infrastructure to protect their customers.

I strongly believe everyone should use a password manager given how easy they make it to protect against credential theft attacks. Being able to automatically generate and fill unique unguessable passwords for every account is the primary benefit most people are familiar with, but modern password managers are chock full of fancy features: helping developers keep plaintext secrets out of code files and environment variables and notifying you when one of your accounts is detected in a data breach are two notable examples.

While the best password managers employ strong security measures, they can only make it harder, not impossible, for attackers to steal your data (granted, they make it very hard for attackers to steal your data). 

The most concerning aspects of LastPass’ recent incidents were that

• a developer’s account was compromised

• a third-party cloud storage system was accessed

In either case, this could have allowed the attacker to inject malicious code into LastPass’ client-side components, such as front end UI code, web browser extensions, and desktop/mobile apps. This is a viable attack path that can result in an attacker stealing decrypted user data for many, if not all of, LastPass’ customers. 

This isn’t some contrived hypothetical attack scenario, either: software supply chain attacks have been on the rise in recent years, especially since SolarWinds suffered one in 2020. This also isn’t the only way an attacker could breach your password manager.

Freaked out yet? Yeah, me too. But don't fret! Let’s take a deep breath and explore all the ways you can protect your accounts even if your password manager is breached.

Driving principles for a hardened password manager setup

Reducing the impact of a password manager breach relies on two core principles:

1. Multi-factor authentication (MFA) needs to be enabled for all sites and apps that support it 

2. Passwords and password equivalents, such as security questions, must be stored separately from MFA codes and MFA code equivalents, such as seed keys and recovery codes

In practice, these principles imply a few things: you'll need one app for storing passwords, a second app for actively using MFA one-time passcodes (OTPs), and a third app for storing MFA recovery codes. This ensures that if an attacker successfully compromises your primary password manager, they also need to carry out a separate attack against your secondary password manager app. The harder you make it for attackers to gain access to your online accounts, the more likely they’ll give up and focus on easier targets.

Ingredients for a hardened password manager “recipe”

Once you’ve implemented all the guidance outlined below, you’ll have a hardened and resilient password manager setup that resembles something like this:

Try not to strain your eyes too much just yet: we’ll dive into each ingredient shown above and how they work together to minimize the impact of a password manager breach.

NOTE: For the sake of simplicity and security, this guide assumes you’re only using Apple devices.

2 password manager apps

I recommend 1Password for storing passwords and password equivalents due to its rich set of features and excellent user interface. 

I recommend BitWarden for storing MFA recovery codes and OTP seed keys since it’s simple enough to use for this use case. While its UI isn’t as good as 1Password’s, you shouldn't need to access your MFA recovery codes and seed keys very often. 

Both use strong security measures and sync data across all major platforms and device types. You’ll need to pay for 1Password (it’s worth it) but can use BitWarden’s free version for this MFA use case.

NOTE: While many web browsers have built-in password managers, they are increasingly targeted by attackers and are not as secure as dedicated password managers. Avoid using them.

1 MFA code app

I recommend using Authy as your MFA code app (as long as you implement it in specific ways, which we’ll cover later on). It makes it easy to save, search for, and use MFA codes across your phone and/or tablet. Because computer OSes are at an increased risk of having malware installed on them (especially compared to iOS), be sure to only install Authy on a phone or tablet.

2 security keys

I recommend YubiKeys, as many security practitioners do, for hardware security keys. When “Security Key” or “Passkey” MFA (i.e. FIDO 2.0, WebAuthn, or U2F) is available for any of your accounts, you’ll want to use your YubiKeys as your MFA method. Security Key MFA “codes” can’t be intercepted by or spoofed by a remote attacker thanks to sophisticated encryption patterns established by FIDO 2.0. For maximum security, try YubiKey Bio (FIDO edition) - affordable three-factor authentication for the masses!

NOTE: you’ll want 2 YubiKeys in case one of them is lost or broken. Keep one on your keychain and another in a safe place, such as a fireproof safe (see below).

1 fireproof safe

If you’re extra cautious (i.e. paranoid), you’ll want to keep a physical copy of your 1Password Secret Key and BitWarden MFA recovery code stored separately from all of your internet-connected devices. That way, if an attacker somehow manages to compromise your 1Password account password and MFA for 1Password, they would have to physically break into your fireproof safe, too.

NOTE: FIDO 2.0-certified biometric authentication provided by Windows Hello, Touch ID, Face ID, etc. can provide similar authentication security as YubiKeys, but they aren’t portable and platform agnostic like YubiKeys are. Trade offs are hard!

While stepping through how to stick to this approach, I'm going to assume (and recommend) that you use:

• Apple devices, given their strong hardware platform + OS security architecture combined with ease of use when using features such as TouchID, FaceID, and Keychain for logging into and unlocking your password manager 

• 1Password for storing passwords and password equivalents, given its strong security architecture, additional security features like Watchtower and developer tool integrations, and excellent user interface

• Authy as your default MFA code app

  • If you’re extra paranoid and are willing to make some usability sacrifices, try storing your MFA codes in a YubiKey and use Yubico Authenticator to access them

• BitWarden (free edition) for storing MFA recovery codes or seed keys when recovery codes aren't available

That concludes part 1 of this series. Part 2 will cover written guidance on how to combine these ingredients to create a hardened password manager setup as well as password manager best practices to abide by.

UPDATE: Part 2 has been published!