top of page

X-Analytics: Model Validation

We understand that all organizations need to be attentive to the possible adverse consequences of decisions based on models that are incorrect or misused.


Opening Statement

This support page was written to assist X-Analytics customers needing to go through the Federal Reserve and Office of the Comptroller of the Currency (OCC) Supervisory Guidance on Model Risk Management. We understand the guidance should be applied as appropriate by all banking organizations supervised by the Federal Reserve.

X-Analytics is a cloud-native application that helps organizations assess, manage, design, and communicate their cyber resilience strategy. As a cloud-native application, X-Analytics includes a user interface, a processing engine, databases, and user dashboards.

1. The user interface provides a means to capture user inputs. The inputs are used to build a risk profile or to make selections within an interactive tool. The risk profile is the unique makeup of an entity, including company exposure details, threat details, asset applicability, impact details, and cyber maturity. The interactive tools include filter and toggle features to display results in a desired fashion.

2. The processing engine provides a means to combine user inputs with backend data and coded formulas to determine risk results. The backend data is informed by historical and cybersecurity intelligence data. The coded formulas come directly from the X-Analytics methodology. The determined results include threat condition, inherent risk condition, cyber maturity condition, residual risk condition, cyber exposure condition, cyber insurance condition, loss lookup tables, and prioritized guidance.

3. The database provide a means to store and archive data. The processing engine pulls user input and backend data from the database for processing purposes and the processing engine stores determined back into the database for archive purposes.

4. The user dashboards provide a means to read and interface with the determined results. The user can read the dashboards to understand their threat condition, inherent risk condition, cyber maturity condition, residual risk condition, cyber exposure condition, cyber insurance condition, loss lookup tables, and prioritized guidance. Some of the dashboards include a report download feature to help the user propagate X-Analytics results to individuals that do not have access to the application.

In summary, X-Analytics is an improved system, method, and apparatus for risk measurement, modeling, reducing, and addressing cyber risk. The X-Analytics user provides input elements within the measurement section and reacts to outputs provided within the reducing and addressing sections. The entire system, method, and apparatus, when built within a cloud-native application, provides a means of automation that closely aligns with an ever-changing threat velocity, and provides an organization with a means to allocate finite budget and other finite enterprise resources during the entire risk resilience journey.


Model Risk Management

Since banking organizations should be attentive to the possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused, this support page includes public artifacts and answers related to the X-Analytics model risk management framework.


The Term "Model".

For the purposes of this support page, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.

1. X-Analytics is a method, system, and apparatus for measuring, modeling, reducing, developing, addressing, and communicating cyber risk resilience strategy. The publicly available components of the methodology can be found here.

2. X-Analytics is based on five U.S patents, with an additional U.S. patent pending.

11,379,773 Method and system for risk measurement and modeling Vescio, Robert

11,282,018 Method and system for risk measurement and modeling Vescio, Robert

10,453,016 Method and system for risk measurement and modeling Vescio, Robert

10,395,201 Method and system for risk measurement and modeling Vescio, Robert

9,747,570 Method and system for risk measurement and modeling Vescio, Robert

3. You may use X-Analytics to analyze your business's cyber risk condition, informing your business's cyber risk mitigation and transfer decisions, identifying and measuring your business's cyber risks, estimating potential loss exposures due to one or more cyber incidents (which could be part of the business's materiality assessment process), and for reporting and communicating the business's cyber risk condition.

4. X-Analytics determines cyber risk estimates for the next twelve months. It illustrates major cyber themes and possibilities that may present themselves to your business, based on patterns formed between historical data, your unique business profile, and the ever-changing macro cyber economic condition.

5. As part of building your unique X-Analytics business profile, you may rely on subjective, qualitative, or expert judgement to populate key inputs, such as exposure profile, threat profile, and cyber maturity profile. As an example, you may use a NIST CSF report (from a consulting firm) to populate your NIST CSF profile within X-Analytics.

6. You may use X-Analytics in conjunction with existing cyber risk observations. X-Analytics is not a prediction of a pre-determined future that precludes unknown conditions and changing human motivations. X-Analytics determines cyber exposure estimates (based on possible financial impacts and probabilities) with estimated guidance, and you may use X-Analytics as a target at which to aim your actions. Ultimately, you are the agent of your business's cyber resilience strategy. Take what you need from X-Analytics to better manage, design, and communicate your business's cyber resilience strategy.

7. However, we do not assume that you will determine X-Analytics is a model and that X-Analytics introduces model risk.


The Use of "Models" Invariably Presents Model Risk.

For the purposes of this support page, we understand the use of "models" invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. We understand that "model risk" can lead to financial loss, poor business and strategic decision-making, or damage to a banking organization's reputation.

1. Your business can use X-Analytics as an apparatus for understanding the business's cyber risk condition. Your business should not use X-Analytics outside of the context of cyber risk because X-Analytics was not designed to model any other risks besides cyber risk.

2. Your business may have experienced the realities of cyber risk in a different way than X-Analytics, and your business may want to use X-Analytics results in conjunction with your existing observations. X-Analytics was not designed to be your business's single source for all cyber risk decisions or for a complete determination of incident materiality.

3. As a standard recommendation, your business may take what you need from X-Analytics to better assess, manage, design, and communicate your business's cyber resilience strategy.

4. X-Analytics has been on the market since 2017. X-Analytics has been regularly updated and improved to account for errors, emerging conditions, and fundamental calibrations.

5. Like with all models, X-Analytics does have limitations and assumptions. Limitations and assumptions are illustrated here.

6. The design of X-Analytics is modular to reduce complexities and to allow for individual module updates without having to reconfigure the entire X-Analytics apparatus. The modular aspects of X-Analytics can be found here.

7. We do not assume the degree of "effective challenge" that your business may apply to X-Analytics. We understand that effective challenge depends on a combination of incentives, competence, and influence.

8. We do not assume to know if your business's use of X-Analytics is less pervasive that other models and if your business's use of X-Analytics has less impact on your business's financial condition, which therefore may indicate that you business adopts a less complex approach to model risk management for X-Analytics.

9. We cannot account for the potential aggregated "model risk" of your business using X-Analytics in conjunction with other "models" or processes.


Model Development, Implementation, and Use

For the purposes of this support page, we understand that "model development" relies heavily on the experience and judgement of developers, and model risk management should include disciplined model development and implementation processes that are consistent with the situation and goals of the model user.

1. We do not assume to know how "model development" relates to your business's risk management policy.

2. A clear statement of purpose, including its intended use, sound design, theory, and logic, for X-Analytics can be found within the X-Analytics methodology here. The X-Analytics methodology is robust, and it illustrates processing components.

a. The X-Analytics enumerations and structure is detailed here.

b. The X-Analytics threat module is detailed here.

c. The X-Analytics impact module is detailed here.

d. The X-Analytics inherent risk module is detailed here.

e. The X-Analytics control effectiveness module is detailed here.

f. The X-Analytics residual risk module is detailed here.

3. We understand the purpose and relevance of data quality. X-Analytics is rooted in data science and data analysis. Throughout the methodology documents, this is listed as historical data and cybersecurity intelligence data.

a. X-Analytics backend variables are updated monthly to account for changes in the threat condition, control effectiveness condition, incident probability condition, and incident severity condition. Backend variable updates can be found here.

b. X-Analytics is running statistical algorithms to determine inherent risk, residual risk, incident probability, incident severity, and cyber exposure estimates for the next twelve months.

c. We manage a vast amounts of historical data and cybersecurity intelligence data to detect complex patterns, which further improves the predictive power of X-Analytics. Below is a high-level list of our data source categories:

i. Threat: We analyze a variety of data sources to understand the cyber threat condition by industry vertical. The data sources cover a broad range of threat categories (such as web application attacks and human error), including varieties within each category (such as SQLi, XSS, and stolen credentials).

ii. Control effectiveness: We analyze a variety of data sources to understand the missing ingredients that allow for successful cyber incidents. The data sources cover a broad range of incident patterns (such as denial of service attacks intersecting with internet-facing servers and applications), which is further divided into specific vectors, multi-step patterns, and other details. The incident details are then aligned to countermeasures (or cybersecurity controls)

iii. Loss probability: We analyze a variety of data sources to understand the probability of cyber incident per each loss category. The data sources cover a broad range of scale for each loss category. As an example, data breach probability is analyzed for data breaches between 1,000 records to 10 billion records.

iv. Loss magnitude: We analyze a variety of data sources to understand the magnitude (or cost) of cyber incident per each loss category. The data sources cover a broad range of scale and cost elements (such as direct, indirect, and opportunity costs). As an example, data breach magnitude is analyzed for data breaches between 1,000 records and 10 billion records, which is further analyzed from the 10% percentile (low magnitude) to the 97% percentile (worst-case magnitude).

v. Industry-based assumption: We analyze a variety of data sources to understand the typical exposure profile, threat profile, and cyber maturity profile for each industry vertical. The data sources cover all 21 industry verticals, which are further divided into revenue tiers per each vertical. The assumptions are used to establish benchmark values and to help businesses build an initial X-Analytics profile if they do not have the time or data or build their own profile.

d. We understand that incomplete data and data bias will lead to incomplete and biased outcomes. As a result, we purposely seek and combine disparate datasets (monthly) to smooth out incomplete and biased data. Additionally, we use the data objectively and as-is.

i. Anchoring bias: We do not give certain data an added weight based on our personal experiences.

ii. Availability bias: We do not ignore data because of familiar assumptions.

iii. Confirmation bias: We do not select data that supports preconceived beliefs.

iv. Loss-aversion bias: We not purge or remove data because of the psychological or emotional opinion regarding the data.

v. Stability bias: To overcome the realities that certain historical data patterns will cease to exist, we observe all data over a three year period to account for ebbs and flows within certain patterns, and we use mathematics (with great care) to create synthetic datapoints to bridge the gap between available evidence and a self-evident reality.

e. We understand that data anomalies and errors can cause misleading or incorrect outcomes. As a result, we review our datasets monthly to correct possible anomalies and errors within X-Analytics.

4. We understand that the relationship between malice- and error-based cyber incidents and financial outcomes is always subject to change. As a result, we use fresh data to inform updates and changes to the X-Analytics apparatus. This includes missing values and errors. All X-Analytics changes are reported here.

5. We maintain and update an offline version of X-Analytics for development purposes. This offline version is used as a blueprint to write development requirements. The development requirements inform the application developers in how to write X-Analytics application code. We follow a strict quality and assurance test program to ensure there is a direct match between the X-Analytics offline version and the X-Analytics application code.

6. Throughout the entire development process, we test various components of X-Analytics and its overall function to ensure X-Analytics is performing as intended. If we discover any errors, we dissect the errors, correct the errors, and retest to confirm the errors have been resolved.

7. We understand that your inputs into X-Analytics may be judgmental and qualitative. Therefore, we cannot account for a business profile that does not truly represent your business, which means we cannot count for X-Analytics outcomes that do not align to your business because of an incorrect profile. As a result, we strongly recommend you build and maintain an X-Analytics profile that best represents your business. This includes your endpoint volume, data record volume, asset applicability, cyber maturity (such as NIST CSF), and many other profile inputs. For a complete list of all profile inputs, please see here.

8. We understand that all "models" have some degree of uncertainty and inaccuracy because all "models" are an imperfect representation of reality. As a result, we account for uncertainty in the following ways:

a. As one of our 10 major threat categories, we use everything else as a catch all category for all incidents that could not be classified within the other nine threat categories. This ensures we do not lose track of uncertain (or limited) incident reporting.

b. We govern control effectiveness at a 98.7% effectiveness, which means we leave a 1.3% chance of uncertainty in control effectiveness for unknown unknown conditions. We do not assume that any risk can be eliminated due to uncertainty.

c. We understand that the cyber risk condition is constantly changing (emerging conditions), which is why we use monthly data analysis to inform backend variables. However, we do not assume that data informed updates will eliminate uncertainty.

9. We do not assume to know how your business reduces uncertainty with your use of X-Analytics. Though we recommend that you use X-Analytics in combination with your own cyber risk realities, other models, or other approaches.


Model Validation

For the purposes of this support page, we understand that "model validation" is the set of processes and activities intended to verify that "models" are performing as expected, in line with their design objectives and business uses.

1. We understand that effective validation helps to ensure that models are sound. We use data to regularly validate and tune X-Analytics, we collect direct feedback from customers to validate and tune X-Analytics, and we regularly predict major cyber incident outcomes using X-Analytics and compare those predictions with actual results to further validate and tune X-Analytics.

2. Throughout the entire development process, we perform quality and assurance testing to ensure all X-Analytics components are sound and working correctly. This includes X-Analytics inputs, processing, outputs, and reports.

3. We designed X-Analytics as a self-contained apparatus in order to maintain integrity principles. We encourage application users to build two profiles and enter the same profile details within each profile to see that outcomes are identical between both profiles.

4. X-Analytics does not allow application users to modify backend variable weights, modify the processing engine, or modify outcomes. Additionally, X-Analytics does not use any form of random simulations (such as value at risk modeling) that would produce different outcomes upon each use of the application.

5. Since X-Analytics is updated on a regular basis, we are performing validations exercises on a regular basis. Additionally, X-Analytics limitations and assumptions are update when necessary.

6. We attempt to overcome periods of benign economic cyber conditions by informing X-Analytics with data that expands over a three year period.

7. We do not assume our model validation process meets your business's model validation process for third party models.


Model Governance, Policies, and Controls

For the purposes of this support page, we understand that developing and maintaining strong governance over the model risk management framework is fundamentally important to its effectiveness.

1. We have strong governance over X-Analytics, which includes the X-Analytics structure, methodology, tuning and calibration, and articulations of outcomes.

2. As part of governance, we document every aspect of X-Analytics. In certain cases, we translate our documents into U.S patent applications to secure our intellectual property rights.

3. The modular design of X-Analytics allows us to update and tune individual components without having to redesign the entire X-Analytics apparatus. This ensures that we maintain structure despite emerging cyber conditions. All updates are documented.

4. X-Analytics is assigned a version number to account for all changes. This includes major changes, minor adjustments to the algorithm, inputs, and outcomes, and all changes to the backend variables.

5. We include governance in how we aggregate and analyze historical data and cybersecurity intelligence data.

6. The entire development process is governed by project management, quality and assurance testing, and executive oversight to reduce the possibility of error-based conditions being released into the X-Analytics production environment

7. We require validation testing for all minor and major updates to X-Analytics.

8. We do not assume that our "model" governance, policies, and controls meets your business requirements, including board member tolerance.


Additional Model Validation Questions

If you were not able to find answers for your model validation questions, please submit a support ticket here.

Comentarios


Los comentarios se han desactivado.
bottom of page