Core Concepts

Core Concepts and Definitions

Accuracy / Precision / Recall

• Accuracy: The proportion of correct predictions (both true positives and true negatives) out of all predictions. A model that is 90% accurate gets 90 out of 100 predictions right. NOT the same as usefulness. A 90% accurate fraud detection model might still generate too many false positives to be usable.
• Precision: Of the items flagged as high-risk, how many actually are high-risk? High precision means fewer false positives (lower false alarm rate). Important when you want to investigate only things likely to be real issues.
• Recall: Of all the actual high-risk items, how many did the model find? High recall means fewer false negatives (fewer missed issues). Important when you cannot afford to miss problems.

Professional use: Precision and recall are trade-offs. A vendor might say, "Our model has 95% precision," meaning "when we flag something, we're right 95% of the time." But it might have only 60% recall, meaning "we miss 40% of actual issues." Both matter; the vendor's emphasis on one might obscure weakness in the other.

Algorithm

A step-by-step procedure for solving a problem or accomplishing a task. In AI contexts, often used to describe the mathematical procedure an ML system uses to learn patterns from data. Not usually something you need to understand in detail, but the term comes up when vendors discuss their approach.

Professional use: "Our model uses a random forest algorithm" tells you something about the technical approach but not necessarily whether it will work for your use case. Ask instead: "What does the model do? How is it trained? What is its accuracy on your data?"

Bias (in AI)

A systematic error in an AI system's outputs, usually learned from biased training data. If an AI system is trained on data that overrepresents certain groups or underrepresents others, the AI system will learn and replicate that imbalance.

Example: If a credit approval AI is trained on historical data where minority applicants were approved at lower rates, the model might learn to approve minority applicants at lower rates too, replicating historical bias.

Professional use: "We've tested for bias" is vague. Ask: "What specific groups were tested? What metrics did you use? What bias was found, and how was it corrected?" Bias testing is not a one-time box; it is ongoing monitoring.

Black Box

An AI system whose decision-making process is not transparent or understandable to humans. You can see the input and output; the internal logic is opaque.

Professional use: Black-box status is common in large neural networks. It is not necessarily a reason to reject an AI system, but it is a reason to require stronger human oversight, monitoring, and performance validation.

Classification

The task of assigning items to categories. An AI system trained to classify divides items into discrete groups.

Example: Classifying documents as "policy," "procedure," or "evidence."

Professional use: Classification is one of AI's strengths. Ask about the categories (are they clearly defined?), the training data (does it represent all categories fairly?), and the accuracy on each category (some categories might be more error-prone than others).

Confidence Score / Probability Score

A number (usually 0-1 or 0-100) representing how confident the model is in its own prediction. 0.95 means the model is very confident; 0.55 means the model is barely more confident than a coin flip.

Professional use: Confidence is not accuracy. A model can be confidently wrong. Always ask: "What is the actual accuracy of high-confidence predictions on your data?" Use confidence scores to set thresholds for human review, not as a measure of how right the model is.

Data (Training Data, Test Data, Production Data)

• Training data: The dataset used to teach the AI system. The model learns patterns from this data.
• Test data: Held-back data used to estimate how well the model will perform on new, unseen data.
• Production data: Real-world data the model encounters after deployment.

Professional use: A critical question for any AI system: "What was it trained on?" If it was trained on data that doesn't represent your organization's context, performance may differ significantly. Also ask: "How is production data monitored?" to ensure the model continues to perform well.

Drift (Model Drift / Performance Drift)

A decrease in an AI model's accuracy over time, usually because the real-world data has changed and the model was trained on older, different data.

Example: A fraud detection model trained on pre-pandemic transaction patterns may perform poorly on pandemic transaction patterns. The environment drifted; the model did not.

Professional use: AI systems require ongoing monitoring. If a vendor says, "We deployed it and haven't retrained since," that's a red flag. Performance drift is expected; the question is how it is monitored and addressed.

Explainability / Interpretability

The ability to understand and explain an AI system's decision. Some AI systems (decision trees, linear models) are inherently explainable. Others (deep neural networks) are not.

Professional use: For high-stakes decisions, explainability matters. If you cannot explain why the system decided something, you cannot defend the decision to a regulator or audit. This is a governance control, not a technical luxury.

False Positive / False Negative

• False positive: The model flags something as high-risk that is actually normal. (Type I error)
• False negative: The model misses something that is actually high-risk. (Type II error)

Professional use: These are trade-offs. Reduce false positives and you increase false negatives. Set the threshold based on the cost of each type of error. In fraud detection, false negatives (missed fraud) are often more costly than false positives (false alarms requiring investigation).

Generalization

The ability of an AI model to perform well on new data it was not trained on. Good generalization means the model learned underlying patterns, not just memorized training examples.

Professional use: Ask: "How was the model tested on data it hadn't seen before?" If the answer is "we tested on a holdout set from the same dataset," that's fine for basic confidence, but not sufficient to verify it will work on your data, which may differ significantly.

Hallucination

When an AI system (especially an LLM) generates false information with confidence, as if it were fact.

Example: An LLM generates: "Section 404(b) of SOX requires annual board certification of controls," inventing an obligation that does not exist in the actual regulation.

Professional use: Hallucination is inherent to how LLMs work (they predict statistically likely text). Never assume an LLM has verified information. Require expert human review of any LLM output before it is used in governance decisions.

Large Language Model (LLM)

An AI system trained on vast amounts of text data to predict and generate language. Examples: ChatGPT, Claude, Gemini, LLaMA.

Professional use: LLMs are powerful for summarization and draft generation but dangerous for fact-dependent tasks like policy interpretation. Use them as assistants that generate drafts; always require expert validation before use.

Machine Learning (ML)

AI where the system improves its accuracy by learning from data, rather than being programmed with explicit rules.

Professional use: When a vendor says "machine learning," they mean the system learns patterns from data. This is different from a rules engine (which is programmed) or analytics (which is human-interpreted). Understanding this distinction is critical for governance.

Model

In AI contexts, the trained system that makes predictions or generates content. It's the "learned" representation of patterns found in training data.

Professional use: "Our model flagged this transaction" means "our trained AI system, based on patterns it learned, flagged this." The model is not perfect; it is a probabilistic tool that requires validation.

Neural Network

A type of AI architecture (inspired by the brain) composed of interconnected nodes. Deep neural networks (many layers) are especially powerful for complex pattern recognition but are also difficult to explain.

Professional use: You don't need to understand neural networks technically. Just know: systems using neural networks can detect complex patterns but are often opaque (black boxes). Require strong oversight.

Overfitting

When an AI model learns the training data too well, including its quirks and noise, rather than learning generalizable patterns. The model performs well on training data but poorly on new data.

Professional use: This is a common problem in AI. It's why testing on data the model hasn't seen is important. A vendor claiming 98% accuracy might be overfitted. Ask: "How was accuracy measured? On held-out test data? On production data?"

Performance Metrics

Quantified measures of how well an AI system works. Examples: accuracy, precision, recall, F1 score, AUC-ROC.

Professional use: When a vendor says "our model is 95% accurate," insist on understanding which metric and on what data. Also ask: "What are the false positive and false negative rates? What is the cost of each?" Metrics matter; context matters more.

Regression (Predictive)

An AI task where the model predicts a continuous numerical value (e.g., predicted risk score, predicted fraud amount) rather than a category.

Professional use: Regression is useful for scoring. It's different from classification (assigning to categories). Understand which task the model is designed for.

Retraining / Fine-Tuning

Updating a model with new data to maintain or improve accuracy as the environment changes.

Professional use: An AI system that is not retrained will experience performance drift. Ask vendors: "How often is the model retrained? What process ensures retraining happens?" This is part of responsible AI governance.

Supervised Learning vs. Unsupervised Learning

• Supervised: The training data includes both inputs and correct answers. The model learns to predict the correct answer for new inputs. Example: Training data includes transaction examples labeled as "normal" or "fraud." The model learns to classify new transactions.
• Unsupervised: The training data includes only inputs; the model learns to find structure or patterns without knowing the "correct answer." Example: Training data includes all transactions (without fraud labels). The model learns to group similar transactions and flag unusual patterns as anomalies.

Professional use: Supervised learning requires curated training data with correct labels (expensive but accurate). Unsupervised learning requires less curation (cheaper but may find spurious patterns). Know which one your vendor is using.

Threshold

The confidence level at which the model takes action (flags something, makes a decision, etc.). Setting the threshold is a governance choice.

Example: A fraud model might flag any transaction with > 70% fraud probability. Raising the threshold to 85% reduces false positives but misses more fraud.

Professional use: The threshold is where governance controls the model's behavior. Ask: "Who set the threshold? Based on what criteria?" This is not a technical decision; it is a governance decision.

Training Data Quality

How representative, complete, and accurate the training data is. Poor-quality training data leads to poor model performance and bias.

Professional use: Ask: "What data was the model trained on? How large is the dataset? Does it represent all relevant groups and scenarios?" Training data quality is often the biggest predictor of model success.

Validation / Cross-Validation

Testing a model on data it was not trained on to estimate real-world performance.

Professional use: A vendor claiming accuracy must show how they validated it. "Tested on a hold-out set" is the minimum. "Tested on multiple years of different data" is better. "Tested on your specific data" is ideal.