How the scorecard is scored
Everything below is the whole rubric. Publishing it is deliberate: an instrument nobody can inspect is exactly the thing this practice is engaged to object to in somebody else’s model.
Instrument version 1.1.1, rubric version 1.1.0. Every response records both, and is always re-scored against the versions it was taken under.
The answer scale
| Answer | Meaning | Points |
|---|---|---|
| Established | Established, written down, and checkable by someone else | 3 |
| Partial | Done, informally or incompletely | 2 |
| Planned | Planned, not yet done | 1 |
| No | Not done | 0 |
| Not sure | Not sure, I cannot tell from what exists | 0 |
A “not sure” scores zero, because a control nobody can evidence is worth zero in a review. The same answers are then scored again with every unknown at its maximum, which gives the upper bound the report prints beside the index. It never trips a gating rule: capping a band on an absence of knowledge would be the one place this instrument overstated what it was told.
The fifteen domains and their weights
| Domain | Items | Weight | Anchored to |
|---|---|---|---|
| Dataset and cohort design | 3 | 7 | TRIPOD+AI: participants, data sources, outcome · PROBAST+AI domains 1 and 3 · Datasheets for Datasets · FDA GMLP principle 5 |
| Data leakage | 3 | 9 | PROBAST+AI analysis domain · TRIPOD+AI analysis items · FDA GMLP principle 4 |
| Internal and external validation | 4 | 11 | TRIPOD+AI · PROBAST+AI · FDA GMLP principle 6 · SR 11-7 outcomes analysis and benchmarking |
| Metric selection | 2 | 6 | TRIPOD+AI performance measures · STARD |
| Calibration | 3 | 8 | TRIPOD+AI calibration items · Van Calster calibration hierarchy |
| Subgroup performance | 3 | 8 | FDA GMLP principles 2 and 3 · NIST AI RMF MEASURE 2.11 · ONC HTI-1 fairness attributes |
| Generalizability and distribution shift | 2 | 5 | FDA GMLP principle 9 · NIST AI RMF MAP and MEASURE |
| Threshold and operating point | 2 | 5 | Decision-curve and net-benefit analysis · STARD |
| Reproducibility | 2 | 5 | TRIPOD+AI code and data availability · NIST AI RMF MANAGE · FDA GMLP principle 2 |
| Monitoring and drift | 3 | 9 | FDA GMLP principle 10 · FDA predetermined change control plans · SR 11-7 ongoing monitoring |
| Documentation | 2 | 4 | Model Cards · ONC HTI-1 decision-support source attributes |
| Governance and accountability | 3 | 7 | NIST AI RMF GOVERN · SR 11-7 governance pillar · ISO/IEC 42001 |
| Regulatory obligations | 3 | 6 | NIST AI RMF GOVERN 1.1 and 1.6 · SR 11-7 model inventory · California CCPA risk assessments · OMB M-25-21 minimum practices for high-impact AI |
| Human oversight and recourse | 2 | 5 | FDA GMLP principle 7 · ECOA Regulation B adverse action · CFPB Circular 2023-03 · Colorado SB 26-189 notice, explanation and human review |
| Third-party components and security | 2 | 5 | NIST AI RMF GOVERN 6.1 · SR 11-7 vendor models · NAIC AI model bulletin third-party section · FD&C Act section 524B |
| Total | 39 | 100 |
The arithmetic
A domain score is the points scored divided by the points available, times 100. The readiness index is the weighted mean of the fifteen domain scores, with the weights above. Evidence coverage is the share of items answered with something other than “not sure”.
The bands
| Index | Band | Meaning |
|---|---|---|
| 85 to 100 | Review-ready | Evidence would likely survive external review with minor requests. |
| 70 to 84 | Ready with conditions | Deployable once the named conditions are closed. |
| 50 to 69 | Moderate risk | Material gaps that a reviewer would raise. |
| 30 to 49 | High risk | Findings here would stop a deployment decision. |
| 0 to 29 | Not ready | The evidence base has not been built yet. |
The seven gating rules
A weighted sum lets a strong total hide a fatal flaw, so six answers cap the reported band whatever the total comes to. A cap can only ever lower a band.
| Rule | Trigger | Caps at |
|---|---|---|
| G1 | Reported performance is not independent Final performance was not measured on data untouched by model and feature decisions, so the reported numbers describe the data the model was tuned on. | High risk |
| G2 | Preprocessing fitted before the split Steps that learn from data were fitted on the full dataset, so the test set influenced the model that is being tested on it. | High risk |
| G3 | Subjects appear on both sides of the split The same subjects appear in more than one split, which a reviewer would treat as a finding regardless of the reported performance. | High risk |
| G4 | No external or forward-in-time evidence The model has never been evaluated on another site, another system, another population, or a later period, so nothing shows it holds outside its development data. | Moderate risk |
| G5 | No plan for measuring performance after deployment Without a written monitoring plan there is no mechanism by which anyone would learn that the model had stopped working. | Moderate risk |
| G6 | No named accountable owner No individual is accountable for the model in production, so there is nobody with the standing to stop it. | Moderate risk |
| G7 | Nobody has established which rules apply No one has determined which laws, regulations or supervisory expectations govern this model, so there is no standard against which anyone could say it is ready. | Moderate risk |
Every item
| # | Domain | Item | What 3 points looks like |
|---|---|---|---|
| 1 | Dataset | Is there a written definition of what is in the dataset, including inclusion rules, exclusion rules, and the time window? Someone new to the project should be able to rebuild the same cohort from the document alone. SR 11-7 model risk management: Robust development, implementation and use · TRIPOD+AI and PROBAST+AI: Participants, data sources, and how the cohort was assembled · STARD, and decision-curve practice: How participants were selected, and from where | Written and version-controlled |
| 2 | Dataset | Do you know how the sample was assembled, and how it differs from the population the model will actually run on? Convenience samples are fine. Undocumented convenience samples are what a reviewer objects to. FDA Good Machine Learning Practice: Principle 3: participants and datasets representative of the intended population · TRIPOD+AI and PROBAST+AI: Participants, data sources, and how the cohort was assembled · STARD, and decision-curve practice: How participants were selected, and from where | A documented comparison of the characteristics that matter |
| 31 | Dataset | Do you know how the outcome label was established, by whom, and whether they could see the model inputs? The label is what every number you report is measured against. One read off a billing code, or assigned by somebody who could see the prediction, sets a ceiling on what any of this can mean. FDA Good Machine Learning Practice: Principle 5: reference datasets based on the best available methods · TRIPOD+AI and PROBAST+AI: How the outcome was defined and determined · STARD, and decision-curve practice: The reference standard, and how it was applied | A written definition of the reference standard, with agreement between assessors measured |
| 3 | Leakage | Were the preprocessing steps that learn from data fitted only on the training split? Imputation, scaling, feature selection, encoding, and resampling all learn from data. Fitting any of them before the split leaks the test set into the model. FDA Good Machine Learning Practice: Principle 4: training data independent of test data · TRIPOD+AI and PROBAST+AI: Predictor handling, and the analysis domain of PROBAST | Fitted inside the training fold in every case |
| 4 | Leakage | Do the same subject, patient, account, or device appear in only one split? A random row split puts repeat records for the same individual on both sides, which inflates every number you report. FDA Good Machine Learning Practice: Principle 4: training data independent of test data | Grouped splits enforced, and tested |
| 5 | Leakage | Have you checked that no feature encodes the outcome, or something knowable only after it? Timestamps, record order, codes assigned at diagnosis, and downstream identifiers are the usual culprits. TRIPOD+AI and PROBAST+AI: Predictor handling, and the analysis domain of PROBAST | A systematic review, documented |
| 6 | Validation | Was final performance measured on data untouched by any model or feature decision? A holdout stops being clean the moment you look at it and change something. FDA Good Machine Learning Practice: Principle 4: training data independent of test data · NIST AI Risk Management Framework 1.0: MEASURE 2.5: validity and reliability demonstrated · TRIPOD+AI and PROBAST+AI: Validation in data not used for development | A held-out set opened once, at the end |
| 7 | Validation | Has the model been evaluated on data from a different site, system, or population? A different unit of the same organisation is a partial answer. A genuinely different data source is the full one. FDA Good Machine Learning Practice: Principle 8: performance demonstrated under the conditions of real use · ONC HTI-1 decision support certification: External validation process described · NIST AI Risk Management Framework 1.0: MEASURE 2.5: validity and reliability demonstrated · OMB M-25-21, high-impact federal use: Pre-deployment testing · TRIPOD+AI and PROBAST+AI: Validation in data not used for development | Yes, on at least one genuinely external source |
| 8 | Validation | Has it been evaluated on a time period later than the one it was trained on? Practice, coding, and case mix all move. A random split hides that; a forward-in-time split shows it. FDA Good Machine Learning Practice: Principle 8: performance demonstrated under the conditions of real use · NIST AI Risk Management Framework 1.0: MEASURE 2.5: validity and reliability demonstrated · OMB M-25-21, high-impact federal use: Pre-deployment testing · TRIPOD+AI and PROBAST+AI: Validation in data not used for development | Yes, a forward-in-time evaluation |
| 32 | Validation | Has the model been compared against a simpler alternative, or against what happens today without it? The bar is whatever happens today without it: the rule of thumb, the existing score, or the person already doing the job. NIST AI Risk Management Framework 1.0: MEASURE 2.5: validity and reliability demonstrated · SR 11-7 model risk management: Outcomes analysis, and benchmarking against alternatives · TRIPOD+AI and PROBAST+AI: Comparison against existing models | Compared against a documented baseline on the same data |
| 9 | Metrics | Do the reported metrics match how the model will be used, and how common the outcome is? When the outcome is rare, area under the ROC curve can look excellent while almost every positive prediction is wrong. FDA Good Machine Learning Practice: Principle 6: model design tailored to the data and the intended use · ONC HTI-1 decision support certification: Quantitative performance, including ongoing maintenance · TRIPOD+AI and PROBAST+AI: Discrimination and calibration both reported | Chosen and justified in writing, with a precision-recall view where the outcome is rare |
| 10 | Metrics | Do the reported numbers carry uncertainty, such as confidence intervals or a bootstrap distribution? A point estimate from a small test set can move a long way on a different sample of the same size. ONC HTI-1 decision support certification: Quantitative performance, including ongoing maintenance · TRIPOD+AI and PROBAST+AI: Sample size, and uncertainty around reported performance · STARD, and decision-curve practice: Estimates of accuracy reported with measures of precision | For every headline metric |
| 11 | Calibration | If the model outputs a probability, or a score used as one, has calibration been assessed? Calibration asks whether a stated 20% risk actually happens about 20% of the time. TRIPOD+AI and PROBAST+AI: Discrimination and calibration both reported | A calibration curve plus a summary measure such as the integrated calibration index, or calibration slope and intercept |
| 12 | Calibration | Was calibration assessed on data independent of anything used to fit a calibration adjustment? Fitting a recalibration and then reporting calibration on the same rows reports the fit, not the model. TRIPOD+AI and PROBAST+AI: Discrimination and calibration both reported | Assessed on independent data |
| 13 | Calibration | Is calibration re-checked in each setting where the model runs? Calibration is the first thing to break when prevalence differs between sites or periods. TRIPOD+AI and PROBAST+AI: Discrimination and calibration both reported | Per site and per period |
| 14 | Subgroups | Has performance been reported separately for the groups the model will affect? Age, sex, race and ethnicity where lawful and appropriate, site, payer, device, and language are the usual axes. FDA Good Machine Learning Practice: Principle 3: participants and datasets representative of the intended population · ONC HTI-1 decision support certification: Fairness assessed and recorded · NIST AI Risk Management Framework 1.0: MEASURE 2.11: fairness and bias evaluated · NAIC model bulletin on AI systems: Testing for unfair discrimination · TRIPOD+AI and PROBAST+AI: Performance in relevant subgroups | Pre-specified subgroups, with counts and intervals |
| 15 | Subgroups | Were those subgroups specified before the results were looked at? Subgroups chosen after seeing the numbers produce findings that do not replicate. ONC HTI-1 decision support certification: Fairness assessed and recorded · NIST AI Risk Management Framework 1.0: MEASURE 2.11: fairness and bias evaluated · NAIC model bulletin on AI systems: Testing for unfair discrimination | Written in advance |
| 16 | Subgroups | Is there a written position on what difference between groups is acceptable, and what happens when it is exceeded? Measuring a gap without deciding in advance what to do about it moves the decision to whoever is in the room later. ONC HTI-1 decision support certification: Fairness assessed and recorded · NIST AI Risk Management Framework 1.0: MEASURE 2.11: fairness and bias evaluated · NAIC model bulletin on AI systems: Testing for unfair discrimination | A written threshold and a defined action |
| 17 | Shift | Do you know which inputs would change most if the model moved to a new site, vendor, or coding practice? A feature that means one thing in your source system can mean something else in the next one. FDA Good Machine Learning Practice: Principle 8: performance demonstrated under the conditions of real use | A documented sensitivity analysis |
| 18 | Shift | Has the model been tested under a deliberately shifted condition? A different device, a later period, a changed case mix, or inputs deliberately made missing. FDA Good Machine Learning Practice: Principle 8: performance demonstrated under the conditions of real use · OMB M-25-21, high-impact federal use: Pre-deployment testing | Yes, at least one stress test |
| 19 | Threshold | Is the operating threshold chosen from the consequences of the two kinds of error, rather than from a metric maximum alone? A missed case and a false alarm rarely cost the same. The threshold is where that judgement gets made. ECOA Regulation B, with CFPB Circular 2023-03: The reasons given must be the reasons actually used · STARD, and decision-curve practice: The operating point, and how it was chosen | Chosen from expected cost or net benefit, and documented |
| 20 | Threshold | Was the threshold selected on data separate from the data used to report performance at it? Choosing the cut-off on the same rows you then report is a small, common, and entirely avoidable optimism. STARD, and decision-curve practice: Performance at that point reported without the optimism of choosing it | Yes, on separate data |
| 21 | Reproducibility | Can the reported result be regenerated today from a recorded state? Code, data version, environment, and seeds. The test is whether a colleague can do it without you. FDA Good Machine Learning Practice: Principle 2: good software engineering and security practices · SR 11-7 model risk management: Robust development, implementation and use · SR 11-7 model risk management: Documentation an independent reviewer could work from · TRIPOD+AI and PROBAST+AI: Code and data availability | One command, verified by somebody other than the author |
| 22 | Reproducibility | Is the artifact that would run in production the same one that was evaluated? Retraining on all the data after evaluation produces a different model from the one you have evidence for. FDA Good Machine Learning Practice: Principle 2: good software engineering and security practices · SR 11-7 model risk management: Robust development, implementation and use | The same versioned artifact, checksummed |
| 23 | Monitoring | Is there a written plan for measuring performance after deployment? Including how outcomes will be obtained, and how long they take to arrive. Label lag is what usually kills a monitoring plan. FDA Good Machine Learning Practice: Principle 10: deployed models monitored and retraining risk managed · ONC HTI-1 decision support certification: Quantitative performance, including ongoing maintenance · NIST AI Risk Management Framework 1.0: MANAGE 4.1 and 4.3: post-deployment monitoring, incident response · SR 11-7 model risk management: Outcomes analysis, and benchmarking against alternatives · SR 11-7 model risk management: Ongoing monitoring, with a defined response · OMB M-25-21, high-impact federal use: Ongoing monitoring after deployment | Written, with owners and a cadence |
| 24 | Monitoring | Will input and prediction distributions be monitored, with thresholds that trigger a review? Watching the inputs catches a broken feed long before the outcomes arrive to tell you. FDA Good Machine Learning Practice: Principle 10: deployed models monitored and retraining risk managed · NIST AI Risk Management Framework 1.0: MANAGE 4.1 and 4.3: post-deployment monitoring, incident response · SR 11-7 model risk management: Ongoing monitoring, with a defined response · OMB M-25-21, high-impact federal use: Ongoing monitoring after deployment | Yes, with defined triggers |
| 25 | Monitoring | Is there a defined response when monitoring fires, including who can pause or roll back the model? An alert with no owner and no authority attached is a log line. FDA Good Machine Learning Practice: Principle 10: deployed models monitored and retraining risk managed · NIST AI Risk Management Framework 1.0: MANAGE 2.4: a mechanism to supersede, disengage or deactivate · SR 11-7 model risk management: Ongoing monitoring, with a defined response | A written procedure with a named owner |
| 26 | Documentation | Is there a document stating intended use, intended users, populations, inputs, limitations, known failure modes, and the uses it is cautioned against? A model card, or whatever your organisation calls the same thing. The cautioned uses are the part reviewers ask for and the part most documents leave out. FDA Good Machine Learning Practice: Principle 6: model design tailored to the data and the intended use · FDA Good Machine Learning Practice: Principle 9: users given clear, essential information · ONC HTI-1 decision support certification: Source attributes available for a predictive intervention · SR 11-7 model risk management: Documentation an independent reviewer could work from · Colorado SB 26-189, automated decisions: Developer documentation: intended uses, known harmful uses, limitations, known risks | A maintained model card or equivalent, reaching the people who use the model |
| 27 | Documentation | Is the dataset documented the same way, covering source, collection, permission basis, known gaps, and licensing? A datasheet for the dataset. The permission basis is the part that gets asked about in diligence. ONC HTI-1 decision support certification: Source attributes available for a predictive intervention · SR 11-7 model risk management: Documentation an independent reviewer could work from · California CCPA automated decision-making rules: Purposes, retention and disclosures recorded for the processing · TRIPOD+AI and PROBAST+AI: Code and data availability | A datasheet or equivalent |
| 28 | Governance | Is one named person accountable for this model in production? Accountable means able to stop it, not merely informed about it. FDA Good Machine Learning Practice: Principle 1: multi-disciplinary expertise across the product life cycle · NIST AI Risk Management Framework 1.0: GOVERN 2.1: roles, responsibilities and lines of authority · SR 11-7 model risk management: Governance, policies and controls · OMB M-25-21, high-impact federal use: A named officer with authority over how high-impact AI is handled · NAIC model bulletin on AI systems: A written programme covering governance and internal controls | Named, with the authority to stop it |
| 29 | Governance | Was the model reviewed by somebody independent of the team that built it? Independent means not reporting to the people whose work is under review, and competent enough that the challenge has teeth. FDA Good Machine Learning Practice: Principle 1: multi-disciplinary expertise across the product life cycle · SR 11-7 model risk management: Effective challenge by parties with competence and standing | An independent review, with findings on record |
| 30 | Governance | Is there a written record of which changes are allowed without re-validation, and what triggers one? Retraining on new data, a changed threshold, and a new input source are the three that catch people out. Under FDA guidance this is a predetermined change control plan. FDA Good Machine Learning Practice: Principle 10: deployed models monitored and retraining risk managed · FDA predetermined change control plans: Modifications described in advance, with the protocol for making them · FDA predetermined change control plans: Impact assessment for each planned modification · ONC HTI-1 decision support certification: Intervention risk analysis, reviewed at least annually · SR 11-7 model risk management: Governance, policies and controls · Colorado SB 26-189, automated decisions: Notice to deployers of material updates or modifications | A written change-control plan, with revalidation triggers |
| 33 | Obligations | Has somebody written down which laws, regulations and supervisory expectations apply to this model? Not whether you comply. Whether anybody has established what you would be complying with, and put their name to it. NIST AI Risk Management Framework 1.0: GOVERN 1.1: legal and regulatory requirements understood and managed · SR 11-7 model risk management: Governance, policies and controls · NAIC model bulletin on AI systems: A written programme covering governance and internal controls | A written determination, reviewed by legal or compliance |
| 34 | Obligations | Is this model recorded in an inventory somebody outside the team could find, with an owner and a risk rating? An early question in a supervisory examination is how many models you have. A model nobody can enumerate cannot be governed. NIST AI Risk Management Framework 1.0: GOVERN 1.6: an inventory of the AI systems in use · SR 11-7 model risk management: A model inventory with an owner and a risk rating · OMB M-25-21, high-impact federal use: An AI use case inventory, marking which uses are high-impact | On a maintained inventory, with an owner and a risk tier |
| 35 | Obligations | Has an assessment been done of how this model affects the people it is used on, including who is worst affected when it is wrong? An impact assessment in the sense OMB M-25-21 and the California rules both use: foreseeable harms, who carries them, and what reduces them. FDA predetermined change control plans: Impact assessment for each planned modification · ONC HTI-1 decision support certification: Intervention risk analysis, reviewed at least annually · NIST AI Risk Management Framework 1.0: MAP 5.1: likelihood and magnitude of impact on individuals and groups · California CCPA automated decision-making rules: A risk assessment before processing that presents significant risk, naming foreseeable risks and safeguards · OMB M-25-21, high-impact federal use: An AI impact assessment | A written assessment, with a date and a review cycle |
| 36 | Oversight | Is it defined what the person receiving the output should do with it, and have the person and the model been evaluated together? A model that is right nine times in ten can still make the pair worse, if the people using it stop checking the cases it gets wrong. FDA Good Machine Learning Practice: Principle 7: performance of the human and the model together · Colorado SB 26-189, automated decisions: Trained people with authority to override, and meaningful human review on request · OMB M-25-21, high-impact federal use: Human oversight of the system in use | A defined role for the output, and a study of the two together |
| 37 | Oversight | If this affects a person, can they be given the specific reasons, and can they contest the outcome and reach a human? Credit, employment, insurance, housing and benefits each carry a version of this duty, and the reasons have to be the real ones. If your model decides nothing about a person, the full answer is a written finding that says so. FDA Good Machine Learning Practice: Principle 9: users given clear, essential information · ECOA Regulation B, with CFPB Circular 2023-03: Specific and accurate principal reasons for an adverse action · ECOA Regulation B, with CFPB Circular 2023-03: The reasons given must be the reasons actually used · Colorado SB 26-189, automated decisions: Clear and conspicuous notice that the technology is used in the decision · Colorado SB 26-189, automated decisions: An understandable explanation of an adverse outcome, within thirty days · Colorado SB 26-189, automated decisions: Trained people with authority to override, and meaningful human review on request · California CCPA automated decision-making rules: A pre-use notice where the technology makes a significant decision | Specific reasons and a documented route to human review, or a written finding that no decision about a person is made |
| 38 | Supply chain | Is there a list of every third-party model, dataset and service this depends on, with versions and permitted uses? A pre-trained model, a purchased score, a vendor API and a licensed dataset are all somebody else's evidence. The version matters because it changes underneath you. NIST AI Risk Management Framework 1.0: GOVERN 6.1: policies for third-party software, data and models · SR 11-7 model risk management: Vendor and third-party models governed the same way · NAIC model bulletin on AI systems: Due diligence on third-party AI systems and data | A maintained list with versions, licences and permitted uses |
| 39 | Supply chain | Has the system been tested against deliberate misuse or attack, and is there a route for somebody to report a problem with it? Adversarial inputs, prompt injection where a language model is involved, data poisoning, and plain misuse. Plus the mundane half: who a user tells when it goes wrong. FDA Good Machine Learning Practice: Principle 2: good software engineering and security practices · NIST AI Risk Management Framework 1.0: MEASURE 2.7: security and resilience evaluated · NIST AI Risk Management Framework 1.0: MANAGE 4.1 and 4.3: post-deployment monitoring, incident response | Adversarial testing on record and a working reporting route |
What each question answers
Naming five standards and leaving it there is an appeal to authority. This is the map the other way round: every duty the instrument claims to cover, and the questions that ask about it. 68 named obligations across twelve bodies of guidance. A question that answered none of them would fail the integrity check this instrument runs on itself.
FDA Good Machine Learning Practice
| What it asks for | Questions |
|---|---|
| Principle 1: multi-disciplinary expertise across the product life cycle | 28, 29 |
| Principle 2: good software engineering and security practices | 21, 22, 39 |
| Principle 3: participants and datasets representative of the intended population | 2, 14 |
| Principle 4: training data independent of test data | 3, 4, 6 |
| Principle 5: reference datasets based on the best available methods | 31 |
| Principle 6: model design tailored to the data and the intended use | 9, 26 |
| Principle 7: performance of the human and the model together | 36 |
| Principle 8: performance demonstrated under the conditions of real use | 7, 8, 17, 18 |
| Principle 9: users given clear, essential information | 26, 37 |
| Principle 10: deployed models monitored and retraining risk managed | 23, 24, 25, 30 |
FDA predetermined change control plans
| What it asks for | Questions |
|---|---|
| Modifications described in advance, with the protocol for making them | 30 |
| Impact assessment for each planned modification | 30, 35 |
ONC HTI-1 decision support certification
| What it asks for | Questions |
|---|---|
| Source attributes available for a predictive intervention | 26, 27 |
| Fairness assessed and recorded | 14, 15, 16 |
| External validation process described | 7 |
| Quantitative performance, including ongoing maintenance | 9, 10, 23 |
| Intervention risk analysis, reviewed at least annually | 30, 35 |
NIST AI Risk Management Framework 1.0
| What it asks for | Questions |
|---|---|
| GOVERN 1.1: legal and regulatory requirements understood and managed | 33 |
| GOVERN 1.6: an inventory of the AI systems in use | 34 |
| GOVERN 2.1: roles, responsibilities and lines of authority | 28 |
| GOVERN 6.1: policies for third-party software, data and models | 38 |
| MAP 5.1: likelihood and magnitude of impact on individuals and groups | 35 |
| MEASURE 2.5: validity and reliability demonstrated | 6, 7, 8, 32 |
| MEASURE 2.7: security and resilience evaluated | 39 |
| MEASURE 2.11: fairness and bias evaluated | 14, 15, 16 |
| MANAGE 2.4: a mechanism to supersede, disengage or deactivate | 25 |
| MANAGE 4.1 and 4.3: post-deployment monitoring, incident response | 23, 24, 39 |
SR 11-7 model risk management
| What it asks for | Questions |
|---|---|
| A model inventory with an owner and a risk rating | 34 |
| Robust development, implementation and use | 1, 21, 22 |
| Effective challenge by parties with competence and standing | 29 |
| Outcomes analysis, and benchmarking against alternatives | 23, 32 |
| Ongoing monitoring, with a defined response | 23, 24, 25 |
| Documentation an independent reviewer could work from | 21, 26, 27 |
| Vendor and third-party models governed the same way | 38 |
| Governance, policies and controls | 28, 30, 33 |
ECOA Regulation B, with CFPB Circular 2023-03
| What it asks for | Questions |
|---|---|
| Specific and accurate principal reasons for an adverse action | 37 |
| The reasons given must be the reasons actually used | 19, 37 |
Colorado SB 26-189, automated decisions
| What it asks for | Questions |
|---|---|
| Clear and conspicuous notice that the technology is used in the decision | 37 |
| An understandable explanation of an adverse outcome, within thirty days | 37 |
| Trained people with authority to override, and meaningful human review on request | 36, 37 |
| Developer documentation: intended uses, known harmful uses, limitations, known risks | 26 |
| Notice to deployers of material updates or modifications | 30 |
California CCPA automated decision-making rules
| What it asks for | Questions |
|---|---|
| A risk assessment before processing that presents significant risk, naming foreseeable risks and safeguards | 35 |
| A pre-use notice where the technology makes a significant decision | 37 |
| Purposes, retention and disclosures recorded for the processing | 27 |
OMB M-25-21, high-impact federal use
| What it asks for | Questions |
|---|---|
| An AI use case inventory, marking which uses are high-impact | 34 |
| An AI impact assessment | 35 |
| Pre-deployment testing | 7, 8, 18 |
| Ongoing monitoring after deployment | 23, 24 |
| Human oversight of the system in use | 36 |
| A named officer with authority over how high-impact AI is handled | 28 |
NAIC model bulletin on AI systems
| What it asks for | Questions |
|---|---|
| A written programme covering governance and internal controls | 28, 33 |
| Due diligence on third-party AI systems and data | 38 |
| Testing for unfair discrimination | 14, 15, 16 |
TRIPOD+AI and PROBAST+AI
| What it asks for | Questions |
|---|---|
| Participants, data sources, and how the cohort was assembled | 1, 2 |
| How the outcome was defined and determined | 31 |
| Predictor handling, and the analysis domain of PROBAST | 3, 5 |
| Sample size, and uncertainty around reported performance | 10 |
| Discrimination and calibration both reported | 9, 11, 12, 13 |
| Performance in relevant subgroups | 14 |
| Validation in data not used for development | 6, 7, 8 |
| Comparison against existing models | 32 |
| Code and data availability | 21, 27 |
STARD, and decision-curve practice
| What it asks for | Questions |
|---|---|
| How participants were selected, and from where | 1, 2 |
| The reference standard, and how it was applied | 31 |
| Estimates of accuracy reported with measures of precision | 10 |
| The operating point, and how it was chosen | 19 |
| Performance at that point reported without the optimism of choosing it | 20 |
Coverage is a claim about the questions, not about you. Answering all of them well is evidence you could show a reviewer; it is not a determination by anybody that you comply with any of these.
What this instrument does not do
It records what a respondent says about their own process. It does not inspect a model, read any code, or see any data, and it makes no prediction about the outcome of a review. It is not an audit and it is not legal or regulatory advice. A thirty-nine-item self-report has a coarse resolution, so the band, the domain profile and the ranked gaps carry the meaning, and small differences in the index do not.
Self-report also has a known optimism. Anchors are written behaviourally to reduce it, and nothing removes it. Where a respondent is unsure, the instrument records that rather than pushing for a guess.
No language model scores an answer or writes a word of any report. Every sentence a respondent reads is generated from this rubric by arithmetic.
