15 selected classical machine learning interview questions grouped by seniority level. Use them to review fundamentals, practical trade-offs, and senior-level production reasoning.
1What is the difference between a model parameter and a hyperparameter in supervised learning?
In supervised machine learning, model parameters are internal variables learned directly from training data via an optimization algorithm (such as gradient descent or normal equations). Examples include the regression weights and bias in linear models or split thresholds in decision trees. In contrast, hyperparameters are external configuration settings specified prior to training that govern the learning process, model capacity, or architecture. They cannot be learned directly via standard training loss minimization because the optimizer would trivially overfit (e.g., setting tree depth to infinity). Examples include learning rate, regularization strength (lambda/C), number of trees in a forest, and maximum tree depth. Hyperparameters are tuned using validation data or cross-validation.
from sklearn.linear_model import Ridge
import numpy as np
X = np.array([[1.0], [2.0], [3.0]])
y = np.array([2.0, 4.0, 6.0])
# Hyperparameter: alpha (regularization strength set beforehand)
model = Ridge(alpha=1.0)
# Fitting optimizes internal parameters on training data
model.fit(X, y)
# Learned parameters
print(f"Weight (Parameter): {model.coef_[0]:.4f}")
print(f"Intercept (Parameter): {model.intercept_:.4f}")
2What assumptions does ordinary least squares linear regression make, and how would residual diagnostics reveal assumption violations?
Ordinary Least Squares (OLS) linear regression relies on several core assumptions: 1. Linearity: The relationship between predictors and the outcome is linear in parameters. 2. Independence of errors: Observations and residual errors are mutually independent (no autocorrelation). 3. Homoscedasticity: Error terms have constant variance across all levels of predictors. 4. Normality of residuals: Error terms are normally distributed (required for valid confidence intervals and hypothesis tests). 5. No multicollinearity: Predictors are not linearly dependent (design matrix has full column rank). Residual diagnostics reveal violations as follows: - Residuals vs. Fitted Values plot: Curvature or non-random patterns reveal non-linearity; a funnel or fan shape reveals heteroscedasticity (non-constant variance). - Normal Q-Q plot: Systematically deviating from the straight diagonal line (e.g., S-curves or heavy tails) reveals non-normality. - Residuals vs. Order/Time plot: Systematic trends or cyclical patterns reveal autocorrelated errors. - Leverage / Cook's Distance plot: Identifies high-leverage outliers or influential points that disproportionately shift the fitted model.
import numpy as np
import statsmodels.api as sm
np.random.seed(42)
X = np.linspace(1, 10, 50)
# Quadratic underlying pattern creates a linearity violation
y = 2 * X + 0.5 * (X ** 2) + np.random.normal(0, 2, 50)
X_with_const = sm.add_constant(X)
model = sm.OLS(y, X_with_const).fit()
residuals = model.resid
print(f"Mean Residual: {np.mean(residuals):.4f}")
print(f"Curvature in residuals indicates model misspecification.")
3How does logistic regression model binary classification, and what is the role of the sigmoid function?
Logistic regression models binary classification by estimating the posterior class probability $P(Y=1|X)$. To ensure predicted probabilities remain bounded within $(0, 1)$, logistic regression models the log-odds (logit) of the positive class as a linear function of the inputs: $\ln\left(\frac{p}{1-p}\right) = w^T x + b$. The sigmoid (logistic) function, $\sigma(z) = \frac{1}{1 + e^{-z}}$, serves as the link function that maps any real-valued linear score $z = w^T x + b \in (-\infty, +\infty)$ monotonically into a valid probability $p \in (0, 1)$. Discrete class decisions are made by applying a decision threshold $\tau$ (typically 0.5): $\hat{y} = 1$ if $P(Y=1|X) \ge \tau$, otherwise $0$. Because $\sigma(z) = 0.5$ occurs precisely when $z = 0$, the decision boundary in feature space is the linear hyperplane $w^T x + b = 0$, making standard logistic regression a linear classifier.
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
w = np.array([1.5, -2.0])
b = 0.5
x = np.array([2.0, 1.0])
z = np.dot(w, x) + b
prob = sigmoid(z)
threshold = 0.5
pred = int(prob >= threshold)
print(f"Log-odds (z): {z:.2f}")
print(f"Probability: {prob:.4f}")
print(f"Class Prediction: {pred}")
4What is L2 regularization, and how does Ridge regression change the objective and coefficient estimates?
L2 regularization (Ridge regression) adds a penalty proportional to the sum of squared weights to the ordinary least squares (OLS) loss function: $$\min_w \|y - Xw\|_2^2 + \lambda \|w\|_2^2$$ Analytically, Ridge modifies the normal equations by adding $\lambda I$ to the gram matrix before inversion: $$w_{\text{ridge}} = (X^T X + \lambda I)^{-1} X^T y$$ Key impacts on the objective and coefficient estimates: 1. Shrinkage: Coefficients are shrunk toward zero in proportion to feature variance and correlation, reducing model complexity without forcing them to exact zero. 2. Multicollinearity and Invertibility: When features are collinear or $p > N$, $X^T X$ is singular or ill-conditioned. Adding $\lambda I$ ensures $(X^T X + \lambda I)$ is strictly positive definite and invertible, stabilizing parameter estimates. 3. Bias-Variance Trade-off: Increasing $\lambda$ introduces intentional bias into coefficient estimates while significantly reducing variance, resulting in lower expected generalization error on unseen data. 4. Feature Scaling Requirement: Because the penalty treats all weights equally, features on larger scales would be regularized disproportionately. Features must be standardized (zero mean, unit variance) prior to fitting.
import numpy as np
def ridge_regression(X, y, alpha):
X_std = (X - np.mean(X, axis=0)) / np.std(X, axis=0)
n_features = X_std.shape[1]
I = np.eye(n_features)
beta = np.linalg.inv(X_std.T @ X_std + alpha * I) @ X_std.T @ y
return beta
X = np.array([[1.0, 2.0], [2.0, 4.1], [3.0, 5.9], [4.0, 8.2]])
y = np.array([2.1, 4.0, 6.2, 8.1])
weights = ridge_regression(X, y, alpha=1.0)
print('Ridge Weights:', weights)
5How does a decision tree recursively partition feature space, and what criteria are used to choose classification splits?
A decision tree partitions feature space through a top-down, greedy algorithm called **recursive binary partitioning**. Starting at the root node with all training data, the algorithm searches over all features and possible threshold values to find the single axis-aligned split ($X_j \le t$) that maximizes the reduction in impurity. The dataset is split into two child nodes, and this procedure is applied recursively on each child node until a stopping criterion (e.g., maximum depth, minimum samples per leaf, or pure nodes) is reached. Because splits evaluate one feature at a time against a threshold, the resulting decision boundaries are orthogonal hyperplanes (axis-aligned rectangular regions in feature space). To evaluate and select the best split in classification trees, two main impurity criteria are used: 1. **Gini Impurity (used in CART)**: Measures the probability that a randomly chosen sample would be misclassified if labeled randomly according to the node's class distribution. For $K$ classes with proportions $p_k$: $$I_G = 1 - \sum_{k=1}^K p_k^2$$ 2. **Entropy and Information Gain (used in ID3, C4.5)**: Entropy measures the uncertainty in a node: $H = -\sum_{k=1}^K p_k \log_2(p_k)$. The split is chosen to maximize **Information Gain**, which is the parent node's entropy minus the weighted average entropy of the child nodes: $$IG = H(\text{parent}) - \sum_{c \in \{\text{left, right}\}} \frac{N_c}{N} H(c)$$ Both metrics reach 0 when a node is completely pure (all samples belong to a single class) and reach their maximum when classes are equally distributed.
6What is k-nearest neighbors, and how does it make predictions for classification and regression?
k-Nearest Neighbors (kNN) is a non-parametric, instance-based (lazy) supervised learning algorithm. It does not train an explicit parametric model; instead, it stores the training dataset and performs all computations during inference. Prediction Workflow: 1. Distance Calculation: When a query instance is evaluated, the algorithm computes its distance to all stored training instances using a specified metric (such as Euclidean, Manhattan, or Minkowski distance). 2. Neighbor Selection: It selects the $k$ training instances with the smallest distances to the query instance. 3. Aggregation: - Classification: It assigns the class via majority vote (mode) among the $k$ neighbors (or distance-weighted vote). - Regression: It predicts the continuous target by taking the local average (mean or median) of the $k$ neighbors' target values (or distance-weighted average). Because distance calculations depend directly on feature scales, feature normalization or standardization is essential to prevent large-magnitude features from dominating distance calculations.
7State the naive Bayes conditional independence assumption and explain why naive Bayes can still work well when it is violated.
The naive Bayes conditional independence assumption states that, given the class label Y = y, all features X_1, X_2, ..., X_d are mutually independent: P(X_1, ..., X_d | Y = y) = \prod_{j=1}^d P(X_j | Y = y). Using Bayes' theorem, the posterior probability is: P(Y = y | X) \propto P(Y = y) \prod_{j=1}^d P(X_j | Y = y), where P(Y = y) is the class prior and P(X_j | Y = y) is the class-conditional likelihood (e.g., Gaussian for continuous features, Multinomial for counts). Naive Bayes often works well in practice despite independence violations because classification relies on the argmax decision rule (argmax_y P(Y=y | X)) rather than accurate probability calibration. Even if feature correlations cause predicted probabilities to become overconfident or distorted, the correct class often retains the highest relative ranking. As long as the correlation does not flip the ranking of class likelihoods, the 0-1 classification decision remains accurate.
from sklearn.naive_bayes import GaussianNB
import numpy as np
X = np.array([[1.0, 1.1], [1.2, 0.9], [-1.0, -1.2], [-0.8, -1.1]])
y = np.array([1, 1, 0, 0])
model = GaussianNB()
model.fit(X, y)
# Prediction uses argmax over class posterior scores
print("Predicted class:", model.predict([[1.1, 1.0]]))
8Derive or explain the closed-form OLS solution and state when it exists uniquely.
The Ordinary Least Squares (OLS) objective minimizes the residual sum of squares: $S(\beta) = \|y - X\beta\|^2 = (y - X\beta)^T (y - X\beta) = y^T y - 2\beta^T X^T y + \beta^T X^T X \beta$. Setting the gradient with respect to $\beta$ to zero: $$\nabla_\beta S(\beta) = -2 X^T y + 2 X^T X \beta = 0 \implies X^T X \beta = X^T y$$ These are the normal equations. When $X^T X$ is non-singular (invertible), the unique closed-form solution is: $$\hat{\beta} = (X^T X)^{-1} X^T y$$ Geometrically, $\hat{y} = X\hat{\beta} = X(X^T X)^{-1} X^T y = H y$ represents the orthogonal projection of the target vector $y$ onto the column space of the design matrix $X$, where $H$ is the projection (hat) matrix. The solution exists uniquely if and only if $X^T X$ is invertible, which requires the $N \times P$ design matrix $X$ to have full column rank ($Rank(X) = P$). This requires $N \ge P$ and no exact multicollinearity (no feature is a linear combination of others). If $X$ is rank-deficient, $X^T X$ is singular, leading to infinitely many solutions, often addressed via regularization or the Moore-Penrose pseudoinverse $X^+ y$.
import numpy as np
# Design matrix X (with intercept column) and target y
X = np.array([[1, 1], [1, 2], [1, 3], [1, 4]])
y = np.array([2.1, 3.9, 6.2, 8.0])
# Normal equations: (X^T X)^(-1) X^T y
beta_hat = np.linalg.inv(X.T @ X) @ X.T @ y
H = X @ np.linalg.inv(X.T @ X) @ X.T
y_hat = H @ y
print(f"Beta: {beta_hat}")
print(f"Predictions: {y_hat}")
9What is maximum likelihood estimation, and how does it lead to the logistic regression cross-entropy objective?
Maximum Likelihood Estimation (MLE) is a method for estimating model parameters $\theta$ by choosing values that maximize the likelihood $L(\theta) = P(\mathcal{D}|\theta)$ of the observed dataset. In binary logistic regression, each label $y_i \in \{0, 1\}$ is modeled as an independent Bernoulli random variable conditioned on $x_i$, with success probability $p_i = \sigma(w^T x_i + b)$. The probability mass function for observation $i$ is $P(y_i|x_i) = p_i^{y_i} (1 - p_i)^{1 - y_i}$. Assuming i.i.d. samples, the joint likelihood is: $$L(w, b) = \prod_{i=1}^N p_i^{y_i} (1 - p_i)^{1 - y_i}$$ Taking the natural logarithm converts the product into a computationally tractable sum of log-likelihoods: $$\ell(w, b) = \sum_{i=1}^N \left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right]$$ Because optimization algorithms are standardly framed as minimization problems, we negate the log-likelihood and normalize by sample size $N$, yielding the Negative Log-Likelihood (NLL), which is exactly the Binary Cross-Entropy (log loss) objective: $$J(w, b) = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right]$$ This objective is convex with respect to the linear logits/weights, so appropriate numerical solvers optimize a global objective. Strict convexity and a finite unique MLE require additional conditions such as sufficient feature rank, regularization, and no perfect class separation.
10How does gradient descent optimize a classical ML objective, and how do learning rate, convergence, and convexity affect training?
Gradient descent minimizes an empirical loss function by iteratively updating model parameters in the opposite direction of the objective function's gradient with respect to those parameters: $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$. Key factors influencing training include: 1. Learning Rate ($\eta$): Controls step size. If set too small, convergence is exceedingly slow and training may stall. If set too large, updates will overshoot the minimum, causing oscillation or numerical divergence. 2. Convergence: Determined by monitoring stopping criteria such as small gradient norm ($||\nabla L(\theta)|| \le \epsilon$), minimal parameter shift, or loss plateauing across consecutive iterations. 3. Convexity: In convex objectives (e.g., standard OLS linear regression or logistic regression), any local minimum is guaranteed to be a global minimum, allowing gradient descent to converge reliably with appropriate step sizes. In nonconvex objectives (e.g., multi-layer neural networks), the loss landscape contains multiple local minima, saddle points, and plateaus, making the final solution sensitive to initialization. 4. Optimization vs. Generalization: Convergence on the training loss reflects optimization success, whereas validation loss evaluates generalization. Reaching a low training loss with high validation error indicates overfitting rather than an optimization failure.
import numpy as np
def gradient_descent(X, y, lr=0.01, max_iters=1000, tol=1e-6):
n_samples, n_features = X.shape
theta = np.zeros(n_features)
prev_loss = float('inf')
for i in range(max_iters):
predictions = X @ theta
error = predictions - y
loss = (1 / (2 * n_samples)) * np.dot(error, error)
if abs(prev_loss - loss) < tol:
print(f'Converged at iteration {i}')
break
prev_loss = loss
grad = (1 / n_samples) * (X.T @ error)
theta -= lr * grad
return theta
11Compare L1, L2, and ElasticNet regularization in terms of sparsity, correlated features, and practical model selection.
L1 (Lasso), L2 (Ridge), and ElasticNet regularization differ in penalty formulation, constraint geometry, sparsity, and handling of correlated predictors: 1. Sparsity & Geometry: - L1 uses an absolute value penalty ($\lambda \|w\|_1$). Its constraint boundary is a diamond/polytope with sharp vertices on the coordinate axes. When the loss contours intersect these corners, weights are driven to exact zero, performing automated feature selection. - L2 uses a squared Euclidean norm penalty ($\lambda \|w\|_2^2$). Its constraint boundary is a smooth hypersphere without corners, shrinking weights toward zero asymptotically but rarely setting them to exact zero. 2. Correlated Features: - Under strong collinearity, L1 tends to arbitrarily pick one feature from a group of correlated predictors and set the remaining coefficients to zero, resulting in unstable estimates across resamples. - L2 retains all correlated features, distributing weights among them and shrinking them together. 3. ElasticNet: - Combines both penalties: $\lambda_1 \|w\|_1 + \lambda_2 \|w\|_2^2$ (often parameterized with $\alpha$ and $l_1\_\text{ratio}$). - Delivers the sparsity and feature selection of Lasso while preserving the grouping effect of Ridge, selecting clusters of correlated predictors together. It is especially useful when $p > N$ or under severe multicollinearity.
12What is the difference between Ridge regression, Principal Component Regression, and Partial Least Squares at a high level?
Ridge Regression, Principal Component Regression (PCR), and Partial Least Squares (PLS) are three linear techniques used to handle multicollinearity and high dimensionality, but they differ in how they reduce variance and whether reduction is continuous or supervised: 1. Ridge Regression: Retains all original $p$ features and applies continuous shrinkage to coefficient magnitudes via an L2 penalty. It does not construct lower-dimensional latent components or discard feature dimensions; rather, it shrinks the variance along low-eigenvalue directions of $X^T X$. 2. Principal Component Regression (PCR): A two-step, unsupervised dimensionality reduction method. It first applies Principal Component Analysis (PCA) strictly to the predictor matrix $X$ to find orthogonal directions of maximal variance, keeps the top $k$ principal components, and fits an OLS regression on those $k$ components. Because PCA ignores the target variable $y$, PCR risks discarding components that have low variance in $X$ but high predictive power for $y$. 3. Partial Least Squares (PLS): A supervised dimensionality reduction method. It constructs $k$ orthogonal latent components by finding linear combinations of $X$ that maximize the covariance between $X$ and the response $y$. By explicitly incorporating target information, PLS identifies components that explain both feature variance and response variation.
from sklearn.linear_model import Ridge, LinearRegression
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from sklearn.pipeline import make_pipeline
# 1. Ridge: Regularized full feature space
ridge = Ridge(alpha=1.0)
# 2. PCR: Unsupervised PCA followed by OLS
pcr = make_pipeline(PCA(n_components=2), LinearRegression())
# 3. PLS: Supervised latent component projection and regression
pls = PLSRegression(n_components=2)
13How do modern gradient boosting implementations such as XGBoost, LightGBM, and CatBoost optimize training or handle tabular features differently?
Modern GBDT frameworks differ substantially in their split-finding algorithms, tree growth strategies, and tabular/categorical feature handling: 1. XGBoost: Traditionally relies on exact greedy or approximate quantile sketch split-finding (and later Fast Hist), uses level-wise (depth-wise) tree growth, and handles missing values by learning an optimal default branch direction during split search. 2. LightGBM: Uses histogram-based split-finding (binning continuous features into discrete buckets, typically 256), leaf-wise (best-first) tree growth for faster loss reduction, Gradient-based One-Side Sampling (GOSS) to keep large-gradient instances while subsampling small-gradient ones, and Exclusive Feature Bundling (EFB) to merge mutually exclusive sparse features. For categorical variables, it finds optimal splits by sorting categorical histogram bins ($O(K \log K)$). 3. CatBoost: Uses oblivious (symmetric) decision trees where all nodes at a given depth share the exact same split, enabling fast vectorized CPU/GPU scoring. Its primary innovation is Ordered Target Statistics and ordered boosting, which computes target statistics over random permutations of training data to prevent target leakage and prediction shift.
14How would you decide whether a custom loss is appropriate for a gradient boosting model under asymmetric business costs?
Deciding whether to implement a custom loss function in gradient boosting under asymmetric business costs requires evaluating whether the asymmetry can be handled downstream via probability calibration and threshold tuning, or whether it fundamentally alters the optimization landscape during tree induction: 1. Threshold Tuning vs Custom Loss: For classification tasks with asymmetric error costs (e.g., false negatives costing $10\times$ false positives), standard cross-entropy is a proper scoring rule that aims to estimate posterior probabilities $P(y=1|x)$, but calibration should be checked and, if needed, corrected on validation data. Shifting the classification decision threshold based on business cost matrix $\tau = \frac{C_{FP}}{C_{FP} + C_{FN}}$ or applying sample weights is often cleaner and avoids custom derivatives. However, for asymmetric regression (e.g., asymmetric pinball loss for inventory demand) or non-linear business penalties where standard objectives cannot guide split finding, a custom loss is warranted. 2. Mathematical Requirements for GBDT: In second-order boosters (XGBoost, LightGBM), a custom loss $L(y, \hat{y})$ normally needs computable first-order gradients ($g_i = \partial L / \partial \hat{y}_i$) and valid second-order curvature/Hessian values ($h_i = \partial^2 L / \partial \hat{y}_i^2$) for split gain and leaf-weight calculations ($w^* = -\sum g_i / (\sum h_i + \lambda)$). Hessians should be non-negative or safely approximated/clipped for numerical stability; some implementations support first-order or approximate objectives, so the requirement is framework-specific. Non-differentiable or discontinuous business metrics should be replaced with smooth surrogate approximations (e.g., Huberized or log-cosh variants).
import numpy as np
import xgboost as xgb
def asymmetric_mse_objective(preds, dtrain):
labels = dtrain.get_label()
residual = preds - labels
# Penalize underestimation (residual < 0) 5x more heavily than overestimation
penalty = np.where(residual < 0, 5.0, 1.0)
grad = 2.0 * penalty * residual
hess = 2.0 * penalty
return grad, hess
# Usage:
# model = xgb.train(params, dtrain, obj=asymmetric_mse_objective)
15What is LambdaMART, and how does it adapt gradient boosting for learning-to-rank objectives?
LambdaMART is a Learning-to-Rank (LTR) algorithm that combines MART (Multiple Additive Regression Trees / Gradient Boosting) with the LambdaRank framework. In ranking, target metrics like NDCG (Normalized Discounted Cumulative Gain) and MAP depend on discrete sort order (ranks), making them flat almost everywhere and non-differentiable with respect to continuous model scores. LambdaMART bypasses this by constructing virtual gradients, called 'lambda gradients' ($\lambda_{ij}$), for pairs of items $(i, j)$ within the same query. The base pairwise gradient comes from a logistic loss on score differences ($s_i - s_j$). LambdaMART scales this gradient by the exact change in the target ranking metric ($|\Delta \text{NDCG}_{ij}|$) that would occur if the positions of document $i$ and document $j$ were swapped: $$\lambda_{ij} = \frac{-\sigma}{1 + e^{\sigma(s_i - s_j)}} |\Delta \text{NDCG}_{ij}|$$ For each individual document $i$, the net gradient is calculated by aggregating pairwise lambdas across all pairs involving document $i$: $\lambda_i = \sum_{j: j \succ i} \lambda_{ij} - \sum_{k: i \succ k} \lambda_{ki}$. Standard regression trees in the boosting ensemble then fit these composite per-document lambda gradients at each boosting iteration, directly optimizing listwise ranking metrics.
import lightgbm as lgb
import numpy as np
# Simulated query-grouped data: 2 queries with 3 docs each
X = np.random.randn(6, 10)
y = np.array([3, 1, 0, 2, 0, 1]) # Relevance grades (0-3)
group = [3, 3] # Query group sizes
train_data = lgb.Dataset(X, label=y, group=group)
params = {
'objective': 'lambdarank',
'metric': 'ndcg',
'ndcg_eval_at': [1, 3],
'learning_rate': 0.1,
'n_estimators': 50
}
ranker = lgb.train(params, train_data)