Technology

Measuring Cybersecurity

December 2, 2020  • US Cybersecurity Group

We can’t solve problems if we don’t know what works and what doesn’t.

What is cybersecurity measurement?

Cybersecurity measurement comprises at least two distinct activities:

  • National and sector data collection: Collecting and analyzing high-level data that allows policymakers to assess where to direct limited resources and how to shape risk management practices (e.g., recording how many hospitals were hit by ransomware in the past year, and how many of those hospitals had dedicated cybersecurity staff).
  • Cybersecurity metrics: Creating a taxonomy as well as tools that allow organizations to assess their own compliance with cybersecurity standards and their return-on-investment for measures taken (e.g., determining whether a company’s cybersecurity program meaningfully reduces risk).
Why is this a priority?

Evidence-based cybersecurity policy requires a fact-based picture of the nature and scope of malicious activities across the economy and the ability to assess whether policy changes reduce their impact. Today, the federal government lacks the most basic, reliable data on (a) the frequency and severity of cyberattacks across all sectors, including government and private industry; (b) the most common security failures that lead to attacks; and (c) the technical, procedural, and administrative steps that thwart attacks most often. This means policymakers cannot make evidence-based decisions on how to either allocate limited resources or incentivize government agencies and private industry to better manage risk. Nor can they engage in systemic risk analysis—the key to proper national-level cybersecurity strategy. Without better data, policymakers might as well be grasping at straws.

Outcomes:
  • A standardized terminology, definition, and ontology framework for cybersecurity measurement.
  • Evidence-based federal strategy and policy to support state, local, tribal, and territorial (SLTT) entities and the private sector.
  • Higher quality risk modeling to support more accurate insurance pricing.
What have been the obstacles to progress?
  • No denominator: Without knowing the total volume of malicious activity, it is very difficult to assess the impact of specific policies, controls, or actions.
  • Lack of incentives: High-quality data on cybersecurity incidents and the effectiveness of defense measures is often proprietary, and companies may not share it unless doing so provides a competitive edge to either their security products and services or their organizational risk management capabilities.
  • Liability: Organizations that are victims of cybersecurity incidents are often reluctant to share security incidents with the public except where legally required, fearing that transparency might expose them to embarrassment or lawsuits.
  • Overly narrow focus: Many cybersecurity policy discussions tend to revolve around discrete software vulnerabilities and data breaches, instead of the risk management practices and resiliency measures that depend on better metrics (e.g., the average time it takes to restore functionality after a breach.
Action Steps
  1. Establish a Bureau of Cyber Statistics. The U.S. Cyberspace Solarium Commission has recommended “Congress should establish a Bureau of Cyber Statistics charged with collecting and providing statistical data on cybersecurity and the cyber ecosystem to inform policymaking and government programs.” Indeed, a dedicated data collection office is necessary to build a more accurate, ground-truth picture of cybersecurity. While codifying a Bureau of Cyber Statistics could take years, the White House can kickstart the effort and demonstrate its potential value by merging established data collection efforts, such as the Internet Crime Complaint Center, National Incident Based Reporting System (NIBRS), and the FTC Consumer Sentinel Network. In addition, any centralized federal office set up to gather cybersecurity statistics should adhere to at least three principles in its early days:
    • Data first, metrics later. Developing and deriving value from cybersecurity metrics has vexed the cybersecurity community for years, and private sector entities have clear incentives to identify and test metrics on their own, without any Bureau of Cyber Statistics. Addressing the gap in nation- and sector-level data presents a less technical challenge, and will yield real and more rapid benefits for the policymaking process. Such data will help policymakers address even the most basic but essential questions such as, “How many companies act on threat data provided by federal agencies?” It will also help to answer the more technical ones, such as “How many route hijacks affecting U.S.-based organizations occur every year?”
    • Government first, industry later. A common sticking point in many cybersecurity discussions arises when the federal government requests that private companies implement steps that should apply equally to federal agencies but that federal agencies have yet to adopt. The area of data collection and metrics presents an excellent opportunity for the federal government to take a leadership role and demonstrate tangible progress. Unlike efforts to gather information on the private sector or state and local government, federal leaders are already empowered and well-equipped to gather relevant data on cybersecurity incidents and defensive measures across the federal government.
    • Voluntary, not mandatory. Once the Bureau of Cyber Statistics has collected useful data from all relevant federal offices, it can begin the vital (but often overlooked) phase of demonstrating that data collection generates useful insights. Armed with a convincing case that data collection is not a pointless exercise, the Bureau should persuade (a) state, local, territorial, and tribal entities and (b) critical infrastructure owners and operators to begin providing narrow sets of data voluntarily. Mandatory data collection would be practically unenforceable and would likely poison the Bureau’s relationships with its most important partners. Dedication to voluntary partnerships with non-federal entities is how the Bureau will stay nimble, rapidly iterate its data requests based on honest feedback, and maximize its value-add.
  2. Assess the cost-effectiveness of cybersecurity frameworks. A common critique of government cybersecurity regulation is that it incentivizes “box-checking” that oversimplifies the fast-moving dynamics of real-world cyber defense. The federal government uses a wide range of frameworks to guide implementation of cybersecurity controls for agencies and private companies, but it remains unclear how cost-effective these frameworks are in reducing risk. Have agencies or companies that use the NIST Cybersecurity Framework to implement cybersecurity programs met their risk reduction goals at lower cost than those who rely on other methodologies? Do organizations that supplement frameworks with risk analysis tools like Factor of Analysis Information Risk experience better outcomes? No overarching study has answered these questions, and the federal government is well-positioned to lead one. While assessing the comparative cost-effectiveness of frameworks and risk analysis tools would fit the mission of the Bureau of Cyber Statistics, the National Institute for Standards of Technology should prioritize engaging with willing industry stakeholders to answer these questions even before the Bureau is established.
  3. Improve state and local law enforcement’s ability to report cybercrime incidents. A successful Bureau of Cyber Statistics will need capable partners in SLTT governments who have the capacity to record and communicate relevant data. In A Roadmap to Strengthen US Cyber Enforcement, Third Way found that only half of the 18,000 state and local law enforcement agencies in the United States report data to the NIBRS, the primary federal program that collects crime data. This database also faces problems with how agencies log incidents of cybercrime, leading to vast undercounts. The next administration should closely examine how to improve cybercrime reporting processes and incentives nationwide.
  4. Establish a cross-sector partnership on modeling cyber risk. Cybersecurity insurance is a nascent tool for private and public organizations that want to spread the risk and costs of cyber incidents, but the industry lacks sufficient data on cybersecurity incidents to inform its actuarial models and has yet to establish a uniform process for pricing insurance for cybersecurity risks. Many of the data sets available to insurance actuaries (including those collected directly from their customers) do not allow them to either account for the full range of cybersecurity risks that exist or develop the quantitative tools common to other insurance contexts. As the U.S. Cyberspace Solarium Commission has recommended, DHS should establish a public-private working group that convenes insurance companies and cyber risk modeling firms to explore avenues for purchasing and/or combining proprietary data that can support more rigorous methodologies for cyber insurance pricing.
Dig Deeper on Measuring Cybersecurity:

Learn More

Legislation, Regulations, Executive Orders, and Guidance