Computer Vision interview preparation

Computer Vision Engineer Interview Questions

15 selected computer vision interview questions grouped by seniority level. Use them to review fundamentals, practical trade-offs, and senior-level production reasoning.

Start a Computer Vision AI InterviewNo credit card required. 1 free session available.
Technical interview practice in EnglishA mode where non-native speakers can practice passing technical interviews.

Junior questions

1What is a discrete 2D convolution on an image, and how does it relate to kernels, filters, and cross-correlation?

Discrete 2D convolution on an image is a linear, spatial filtering operation where a matrix of weights (a kernel) slides across the input image. At each spatial location, it computes an element-wise product between the kernel weights and the overlapping local image patch (receptive field) and sums the results (often adding a bias term) to produce a single pixel value in the output feature map. In classical mathematics and signal processing, true convolution involves flipping (rotating by 180 degrees) the kernel horizontally and vertically before computing the sliding inner product: $(I * K)(i, j) = \sum_m \sum_n I(i - m, j - n) K(m, n)$. In contrast, cross-correlation computes the sliding dot product directly without flipping the kernel: $(I \star K)(i, j) = \sum_m \sum_n I(i + m, j + n) K(m, n)$. In deep learning frameworks like PyTorch and TensorFlow, the operation implemented under the name "convolution" is actually cross-correlation. Because kernel weights are learnable parameters optimized directly via backpropagation, the network simply learns the appropriately oriented weights, making the mathematical kernel flip computationally redundant during both training and inference.

import numpy as np
from scipy.signal import convolve2d, correlate2d

image = np.array([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]], dtype=float)

# Asymmetric directional kernel
kernel = np.array([[1, 0],
                   [0, -1]], dtype=float)

# Convolve flips kernel by 180 degrees: kernel[::-1, ::-1]
conv_res = convolve2d(image, kernel, mode='valid')
corr_res = correlate2d(image, kernel, mode='valid')

print("Convolve output:\n", conv_res)
print("Correlate output:\n", corr_res)
Try answering this question with an AI coach

2When would you use RGB, HSV, Lab, YCbCr, or grayscale representations in a vision pipeline?

Different color spaces decouple specific physical, perceptual, and statistical properties of visual data, making them suited for specific vision tasks: 1. **RGB / BGR**: Standard format for camera sensors and display hardware representing additive primary colors (Red, Green, Blue). Color channels are highly correlated with luminance (lighting changes affect all three channels simultaneously). It is the standard input for deep neural networks (CNNs, Vision Transformers). A major production issue is RGB vs BGR channel swapping (e.g., OpenCV loads BGR, while PIL/PyTorch expect RGB), which silently hurts model accuracy. 2. **HSV / HSL**: Explicitly decouples chromaticity (Hue = color tint, Saturation = color purity) from intensity (Value/Lightness). It is ideal for classical color thresholding, object tracking by color, and color segmentation under varying illumination, because Hue is relatively invariant to shadows and brightness changes. 3. **CIE Lab**: A perceptually uniform color space where Euclidean distance between two points ($\Delta E$) directly reflects human perceptual difference. It separates Lightness ($L^*$) from opponent color axes ($a^*$ green-red, $b^*$ blue-yellow). It is used in color-difference inspection, color correction, image quality assessment, and colorization tasks. 4. **YCbCr**: Separates luminance ($Y$) from blue-difference ($Cb$) and red-difference ($Cr$) chroma components. It is the backbone of video and image compression standards (JPEG, MPEG, H.264/H.265) because the human visual system is less sensitive to chroma detail, allowing chroma subsampling (e.g., 4:2:0) to reduce bandwidth and compute. 5. **Grayscale**: Single-channel intensity representation ($Y \approx 0.299R + 0.587G + 0.114B$). It is ideal for geometry- and texture-based tasks where color is irrelevant (e.g., optical flow, SLAM, classical feature extraction like SIFT/ORB, edge detection) to save 66% memory and compute. However, it should not be used when color is a key discriminative cue (e.g., traffic light state classification).

import cv2
import numpy as np

# Sample image: saturated red under normal and shaded lighting
img_bgr = np.zeros((100, 100, 3), dtype=np.uint8)
img_bgr[:, :50] = [0, 0, 255]    # Bright Red in BGR
img_bgr[:, 50:] = [0, 0, 120]    # Shaded/Dark Red in BGR

# Convert to HSV (OpenCV H: 0-179, S: 0-255, V: 0-255)
img_hsv = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2HSV)

# Red Hue wraps around 0/180; track red with single Hue range [0, 10] & high saturation
lower_red = np.array([0, 100, 50])
upper_red = np.array([10, 255, 255])
mask_hsv = cv2.inRange(img_hsv, lower_red, upper_red)

print("HSV segmented pixels:", np.count_nonzero(mask_hsv), "out of", mask_hsv.size)
Try answering this question with an AI coach

3Why are channel-wise image normalization, per-image normalization, and consistent pixel scaling important in vision systems?

Image normalization is vital in vision pipelines to maintain numerical stability, accelerate gradient descent convergence, and prevent train-serving distribution shift: 1. **Pixel Scaling ([0, 1] or [-1, 1])**: Raw image pixels are stored as unsigned 8-bit integers (`uint8` $\in [0, 255]$). Feeding raw $[0, 255]$ values into neural networks causes extremely large early activations and gradient explosion or saturation. Scaling to $[0.0, 1.0]$ (dividing by 255.0) or $[-1.0, 1.0]$ centers inputs around zero, aligning with standard weight initialization methods (He, Xavier). 2. **Dataset-Level Channel Normalization (Mean-Std Standardization)**: Standardizes inputs via $x_{norm} = \frac{x - \mu_c}{\sigma_c}$ using precomputed per-channel global statistics (e.g., ImageNet mean `[0.485, 0.456, 0.406]` and std `[0.229, 0.224, 0.225]`). This centers each channel around zero mean with unit variance across the dataset, ensuring balanced gradient flow across channels. A critical operational failure is **train-serving skew**: if inference omits division by 255, uses mismatched mean/std constants, or applies them in the wrong channel order (RGB vs BGR), model performance severely drops. 3. **Per-Image Normalization (Instance Normalization / Min-Max / Z-score)**: Computes mean and standard deviation per individual image: $x_{norm} = \frac{x - \mu_{img}}{\sigma_{img}}$. This removes global contrast and illumination variations across different capture conditions. While effective in style transfer, MRI/CT medical imaging, or satellite imagery, it can be detrimental when absolute pixel intensity carries physical meaning (e.g., day vs night classification, material reflectance, or defect detection under calibrated lighting).

import torch
import torchvision.transforms as T
from PIL import Image
import numpy as np

# Create a mock uint8 RGB image
raw_img = Image.fromarray(np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8))

# Standard transform pipeline:
# 1. ToTensor scales uint8 [0, 255] -> float32 [0.0, 1.0] and permutes HWC -> CHW
# 2. Normalize standardizes per-channel using dataset mean and std
preprocess = T.Compose([
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

tensor_img = preprocess(raw_img)
print("Tensor shape:", tensor_img.shape)
print("Tensor dtype:", tensor_img.dtype)
print("Channel 0 min/max:", round(tensor_img[0].min().item(), 2), round(tensor_img[0].max().item(), 2))
Try answering this question with an AI coach

4How do geometric and photometric augmentations encourage invariance, and how do you decide whether an augmentation preserves labels?

Geometric augmentations (such as rotation, translation, scaling, and flipping) alter the spatial coordinate mapping of pixels, while photometric augmentations (such as brightness, contrast, hue jitter, and blur) alter pixel intensity and color channels without modifying spatial coordinates. In classification, both categories expose the model to realistic variations, encouraging it to learn invariant representations where predictions remain stable despite changes in pose, viewpoint, or lighting conditions. In spatial prediction tasks (detection and segmentation), geometric transforms require matching transformations to ground-truth boxes or masks to preserve equivariance. Deciding whether an augmentation preserves labels requires evaluating domain semantics and physical priors. A transform is label-preserving if the resulting image remains a plausible instance of the original target class without altering its semantic identity. For example, horizontal flipping preserves identity for general objects (like cars or animals), but alters identity or invalidates characters in optical character recognition (OCR) and digit classification. Similarly, full 360-degree rotation is valid in histology or satellite imagery where no canonical orientation exists, but unnatural in autonomous driving where upside-down objects violate real-world physical constraints.

import albumentations as A

# Natural scene classification / object detection: horizontal flip is valid
natural_scene_transform = A.Compose([
    A.HorizontalFlip(p=0.5),
    A.RandomBrightnessContrast(p=0.2),
], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['category_ids']))

# OCR / Digit classification: horizontal flip corrupts semantics
ocr_transform = A.Compose([
    A.Affine(scale=(0.9, 1.1), rotate=(-10, 10), p=0.5),
    A.ColorJitter(brightness=0.2, contrast=0.2, p=0.5),
])
Try answering this question with an AI coach

5What are nearest-neighbor, bilinear, bicubic, and area interpolation, and how can resizing introduce aliasing or blur?

Interpolation methods estimate pixel intensities at continuous coordinates when resampling onto a new discrete grid: - **Nearest-Neighbor:** Assigns the intensity of the closest discrete pixel. It is computationally $O(1)$ and preserves exact original values, but creates blocky artifacts when upsampling and severe jagged edges when downsampling. - **Bilinear Interpolation:** Computes a distance-weighted average of the $2 \times 2$ (4 nearest) neighbor pixels using linear interpolation along both axes, yielding smooth transitions with moderate blurring. - **Bicubic Interpolation:** Samples a $4 \times 4$ (16 nearest) neighborhood using cubic polynomial kernels. It captures local intensity gradients, producing sharper edges and smoother curves than bilinear, but can produce overshoot/ringing artifacts around sharp transitions. - **Area (Box) Interpolation:** Computes the fractional area overlap of target pixels projected onto source pixels and averages the underlying source intensities. It is especially effective for downsampling without moiré. **Aliasing vs. Blur Mechanics:** - **Aliasing** occurs during downsampling when high-frequency content exceeds the Nyquist frequency ($f_s / 2$) of the new sampling rate without sufficient low-pass filtering. High frequencies fold back into lower frequency bands, producing false patterns, moiré, and jagged edges. - **Blur** occurs when interpolation kernels act as low-pass smoothing filters that attenuate legitimate high frequencies (e.g., bilinear smoothing) or when high-frequency details are averaged out during downsampling.

import cv2

# Severe downsampling: area downsampling prevents aliasing
img = cv2.imread('high_res_pattern.jpg')
downscaled_area = cv2.resize(img, (400, 300), interpolation=cv2.INTER_AREA)
downscaled_nn = cv2.resize(img, (400, 300), interpolation=cv2.INTER_NEAREST)  # heavy aliasing/moiré

# Upscaling: bicubic preserves edge sharpness
upscaled_cubic = cv2.resize(downscaled_area, (400, 400), interpolation=cv2.INTER_CUBIC)
upscaled_linear = cv2.resize(downscaled_area, (400, 400), interpolation=cv2.INTER_LINEAR)
Try answering this question with an AI coach

6What real-world visual failure modes should a vision system be evaluated against, and how do they shift the input distribution?

Real-world visual failure modes fall into several broad categories: environmental conditions (low light, direct glare, rain, snow, fog), sensor and optical effects (motion blur, defocus blur, high ISO sensor noise, rolling shutter skew), compression and transmission degradation (JPEG blockiness, video bitrate throttling, downsampling), and semantic/spatial shifts (partial occlusion, truncation, out-of-distribution object poses). These failure modes shift the input distribution away from nominal training domains at multiple levels. At the pixel level, they alter luminance and chrominance histograms, destroy local contrast, reduce the signal-to-noise ratio (SNR), and introduce artificial high-frequency artifacts or blur kernels. At the structural and semantic level, they attenuate edge gradients, obscure key visual features, distort silhouettes, and hide critical diagnostic regions. When models trained on clean data encounter these shifted inputs, low-level convolutional or attention feature extractors fail to activate correctly, producing dropped detections, hallucinated false positives on noise patterns, and overconfident errors.

Category              | Physical Mechanism             | Input Shift & Feature Impact
-----------------------------------------------------------------------------------------
Low Light / Noise     | Low photon flux, high sensor gain | Drops SNR, suppresses gradient edges, fires spurious texture filters
Motion / Defocus Blur | Scene/camera motion, out-of-focus| Attenuates high spatial frequencies, smears object boundaries
Adverse Weather/Glare | Fog/rain scattering, lens flare  | Compresses dynamic range, lowers contrast, masks semantic regions
Occlusion / Cutoff    | Physical foreground obstructions | Masks critical keypoints/parts, breaks holistic spatial priors
Compression Artifacts | Block DCT quantization, subsampling| Introduces 8x8 grid boundaries, removes fine textural details
Try answering this question with an AI coach

7How do classical descriptors such as HOG represent image structure, and what limitations do they have compared with learned CNN features?

The Histogram of Oriented Gradients (HOG) is a classical handcrafted feature descriptor designed to capture local object shape and appearance through the distribution of intensity gradient orientations. The HOG computation pipeline involves: (1) calculating horizontal and vertical image gradients (e.g., with 1D derivative filters [-1, 0, 1]) to obtain gradient magnitude and orientation; (2) dividing the image into small spatial regions called 'cells' (such as 8x8 pixels) and accumulating 1D histograms of gradient orientations weighted by gradient magnitude; (3) grouping adjacent cells into larger overlapping 'blocks' (such as 2x2 cells) and normalizing block feature vectors (using L2-norm or L1-sqrt) to achieve robustness against local illumination, contrast, and shadowing variations; and (4) concatenating normalized block vectors into a final 1D feature representation, historically paired with linear SVMs for pedestrian and object detection. Compared to learned Convolutional Neural Network (CNN) features, HOG has major limitations: First, HOG features are fixed and handcrafted, capturing only low-level local edge orientation statistics without the ability to learn task-specific hierarchical representations (such as textures, object parts, and semantic concepts). Second, HOG provides very limited geometric invariance: while block normalization handles monotonic illumination changes and small spatial shifts within cells, HOG is fragile under 3D out-of-plane rotations, significant scale variations, non-rigid pose articulations, and heavy background clutter. In contrast, CNNs learn multi-layer non-linear abstractions optimized end-to-end for the target objective.

Input Image Patch (e.g., 64x128)
  │
  ▼
[ 1. Compute Gradients ] ──> Gx, Gy -> Mag = sqrt(Gx^2 + Gy^2), Angle = atan2(Gy, Gx)
  │
  ▼
[ 2. Cell Histograms   ] ──> 8x8 pixel cells accumulate 9-bin orientation histograms (0°-180°)
  │
  ▼
[ 3. Block Normalization] ──> Overlapping 2x2 cell blocks normalized (e.g., v / sqrt(||v||_2^2 + eps^2))
  │
  ▼
[ 4. Feature Vector    ] ──> Concatenate block vectors -> Feed to Linear SVM / Classifier
Try answering this question with an AI coach

Middle questions

8How do stride, padding, dilation, and kernel size determine convolution output size and receptive field?

The spatial output dimensions of a convolutional layer and its cumulative receptive field are determined by kernel size $k$, stride $s$, padding $p$, and dilation $d$: 1. **Output Spatial Size**: Dilation introduces $(d - 1)$ gaps between kernel elements, giving an effective kernel size of $k' = d(k - 1) + 1$. The output dimension along an axis is given by: $$O = \left\lfloor \frac{I + 2p - k'}{s} \right\rfloor + 1 = \left\lfloor \frac{I + 2p - d(k - 1) - 1}{s} \right\rfloor + 1$$ Here, stride downsamples the resolution by advancing the kernel $s$ pixels per step, padding adds virtual border pixels to preserve or adjust spatial dimensions, and dilation expands the filter's footprint without adding extra parameters. 2. **Receptive Field (RF)**: The theoretical receptive field describes the spatial extent in the input image that influences a particular activation. Across stacked layers, the receptive field $RF_l$ and cumulative stride (jump $j_l$) update recursively: - Jump: $j_l = j_{l-1} \cdot s_l$, with $j_0 = 1$ - Receptive Field: $RF_l = RF_{l-1} + (k'_l - 1) \cdot j_{l-1}$, with $RF_0 = 1$ Strides and pooling expand the receptive field multiplicatively across depth because subsequent layer kernel steps correspond to larger pixel jumps in the original input coordinate space. Dilation expands the receptive field additively within a single layer by increasing $k'_l$ without spatial downsampling.

def compute_conv_output_and_rf(layers, input_size=224):
    current_size = input_size
    rf = 1
    jump = 1
    
    for idx, layer in enumerate(layers, 1):
        k, s, p, d = layer['k'], layer['s'], layer['p'], layer['d']
        k_eff = d * (k - 1) + 1
        current_size = ((current_size + 2 * p - k_eff) // s) + 1
        rf = rf + (k_eff - 1) * jump
        jump = jump * s
        print(f"Layer {idx}: Out={current_size}x{current_size}, RF={rf}x{rf}, Jump={jump}")

layers = [
    {'k': 3, 's': 1, 'p': 1, 'd': 1},  # Conv1: 3x3 standard
    {'k': 3, 's': 2, 'p': 1, 'd': 1},  # Conv2: 3x3 strided
    {'k': 3, 's': 1, 'p': 2, 'd': 2},  # Conv3: 3x3 dilated (d=2)
]
compute_conv_output_and_rf(layers, 224)
Try answering this question with an AI coach

9How does a convolution kernel correspond to a classical linear image filter, and when can a 2D filter be made separable?

In classical linear signal and image processing, a convolution kernel represents the spatial impulse response (point spread function) of a Linear Shift-Invariant (LSI) system. Applying the kernel is a spatial-domain linear filtering operation that shapes the image's frequency response (its 2D Fourier transform). Low-pass kernels (e.g., Gaussian, box blur) attenuate high spatial frequencies to suppress noise and smooth textures, while high-pass or band-pass kernels (e.g., Sobel, Laplacian, Prewitt) amplify high frequencies to detect edges and gradients. A 2D filter kernel $K \in \mathbb{R}^{M \times N}$ is spatially separable if it can be decomposed into the outer product of two 1D filters: $K = u \cdot v^T$, where $u \in \mathbb{R}^{M \times 1}$ and $v \in \mathbb{R}^{N \times 1}$. In linear algebra terms, a 2D matrix is separable if and only if its matrix rank is 1. This can be verified via Singular Value Decomposition (SVD), where exactly one singular value is non-zero ($ sigma_1 > 0, \sigma_2 = \dots = 0$). Separability significantly reduces computational complexity. Applying a non-separable $K \times K$ kernel to an $H \times W$ image requires $O(H \cdot W \cdot K^2)$ multiplications and additions. When decomposed into two consecutive 1D passes (horizontal then vertical), the complexity drops to $O(H \cdot W \cdot 2K)$. For a $15 \times 15$ kernel, this provides roughly a $7.5\times$ speedup.

import numpy as np

# Gaussian kernel is rank-1 (separable)
gaussian_1d = np.array([1, 2, 1], dtype=float)[:, None]
gaussian_2d = gaussian_1d @ gaussian_1d.T
gaussian_2d /= gaussian_2d.sum()

# Sobel-X kernel is rank-1 (separable)
sobel_x = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]], dtype=float)

# Diagonal kernel is rank-2 (non-separable)
non_sep = np.array([[1, 0],
                    [0, 1]], dtype=float)

print("Gaussian rank:", np.linalg.matrix_rank(gaussian_2d))
print("Sobel-X rank:", np.linalg.matrix_rank(sobel_x))
print("Diagonal rank:", np.linalg.matrix_rank(non_sep))

# SVD decomposition for Sobel-X
U, S, Vt = np.linalg.svd(sobel_x)
print("Sobel-X singular values:", np.round(S, 4))
Try answering this question with an AI coach

10How do resizing, center cropping, random-resized cropping, stretching, and letterboxing change the input distribution?

Different resizing and cropping strategies shift the input data distribution across geometry, scale, content coverage, and spatial boundaries: 1. **Direct Resizing (Stretching/Squishing):** Rescales the image non-isotropically to a target dimension $(H, W)$, distorting original aspect ratios. This forces the model to process deformed object shapes (e.g., circular objects become elongated ellipses). 2. **Center Cropping:** Extracts a central crop of fixed size or ratio. It preserves the native aspect ratio and local scale, but introduces a strong center bias (assuming targets are centrally framed) and discards peripheral context or cuts off edge-positioned objects. 3. **Random-Resized Cropping (RRC):** Extracts random sub-regions across varying area scales and aspect ratios, then resizes them to a fixed dimension. This broadens the training scale distribution and encourages part-based feature learning, but extreme crops may exclude the target object entirely. 4. **Letterboxing (Isotropic Resizing with Padding):** Scales the image uniformly until its longest dimension fits the target size, then pads the remaining borders with a constant value. It maintains true object proportions and aspect ratios, but introduces artificial high-contrast borders and uninformative padded pixels.

import cv2
import numpy as np

def letterbox_image(image, target_size=(640, 640), pad_color=(114, 114, 114)):
    h, w = image.shape[:2]
    target_w, target_h = target_size
    scale = min(target_w / w, target_h / h)
    new_w, new_h = int(w * scale), int(h * scale)
    
    resized = cv2.resize(image, (new_w, new_h), interpolation=cv2.INTER_LINEAR)
    canvas = np.full((target_h, target_w, 3), pad_color, dtype=np.uint8)
    
    top = (target_h - new_h) // 2
    left = (target_w - new_w) // 2
    canvas[top:top+new_h, left:left+new_w] = resized
    return canvas, scale, (left, top)
Try answering this question with an AI coach

11What annotation noise modes occur in vision datasets, and how would you detect or mitigate them?

Computer vision datasets commonly exhibit four primary annotation noise modes: 1. **Categorical Label Noise:** An image or object is assigned the wrong class label (e.g., misclassifying a cat as a dog, or confusing subtle fine-grained classes). 2. **Missing Annotations (Omission Noise):** Valid foreground objects are left unannotated. In object detection, omitted targets are treated as background negatives, directly penalizing correct model detections during training. 3. **Bounding-Box / Keypoint Jitter (Localization Noise):** Inaccurate, loose, or shifted bounding box coordinates and landmark locations caused by human annotator inconsistency. 4. **Mask Boundary Ambiguity:** Inconsistent or coarse segmentation outlines along complex or blurry boundaries (e.g., hair, semi-transparent surfaces, motion blur). **Detection and Mitigation Strategies:** - **Loss Outlier Tracking & Confident Learning:** Tracking sample loss across training epochs identifies persistent high-loss outliers, which often indicate mislabeled or omitted instances. Algorithms like Confident Learning estimate noise distributions to prune or correct errors. - **Out-of-Fold (OOF) Prediction Discrepancies:** Training cross-validation models and comparing holdout predictions against ground truth highlights mislabeled images and missing bounding boxes. - **Inter-Annotator Agreement (IAA) & Adjudication:** Measuring metrics like Cohen's Kappa (classification) or mean Intersection-over-Union (localization) across multiple annotators flags low-agreement samples for consensus review. - **Robust Loss Formulations:** Using label smoothing, noise-tolerant classification losses, or robust IoU-based regression losses reduces sensitivity to jitter and mislabeling.

import torch
import torch.nn.functional as F

def find_label_noise_candidates(model, dataloader, device, top_k=50):
    model.eval()
    sample_losses = []
    
    with torch.no_grad():
        for batch_idx, (images, targets, sample_ids) in enumerate(dataloader):
            images, targets = images.to(device), targets.to(device)
            logits = model(images)
            loss_per_sample = F.cross_entropy(logits, targets, reduction='none')
            
            for sid, l, target, pred in zip(sample_ids, loss_per_sample.cpu(), targets.cpu(), logits.argmax(dim=-1).cpu()):
                sample_losses.append({'id': sid, 'loss': l.item(), 'target': target.item(), 'pred': pred.item()})
                
    sample_losses.sort(key=lambda x: x['loss'], reverse=True)
    return sample_losses[:top_k]
Try answering this question with an AI coach

12How does class imbalance affect image classification, detection, and dense prediction training?

Class imbalance affects optimization dynamics and loss landscapes across vision tasks: 1. **Image Classification:** When majority classes outnumber minority classes, empirical risk minimization causes parameter updates to be dominated by majority class gradients. The model learns the empirical class prior $P(Y)$ and biases its decision boundary against rare classes. This yields high overall top-1 accuracy while suffering from severe minority recall collapse. 2. **Object Detection:** - *Foreground-Background Imbalance:* In dense single-stage and anchor-based detectors, hundreds of thousands of candidate locations are evaluated per image, where >99% are background. Even if individual easy background anchors yield small losses, their vast aggregate gradient swamps the informative gradient signals from sparse foreground objects. - *Foreground Class Imbalance:* Common object categories appear orders of magnitude more frequently than rare categories, reducing the effective sample size and learning signal for rare classes. 3. **Dense Prediction (Segmentation):** Pixel-level imbalance is severe because large background and 'stuff' classes (e.g., road, sky) occupy millions of pixels, whereas small 'thing' classes (e.g., traffic signs, pedestrians) may occupy fewer than 0.1% of image pixels. Standard per-pixel cross-entropy loss is dominated by large-area classes, leading to the under-segmentation or complete disappearance of small, thin structures.

# Illustrating cumulative gradient dominance by background anchors
import torch

num_bg_anchors = 100_000
num_fg_anchors = 10

bg_loss_per_sample = torch.tensor(0.001, requires_grad=True)
fg_loss_per_sample = torch.tensor(1.5, requires_grad=True)

total_loss = (num_bg_anchors * bg_loss_per_sample) + (num_fg_anchors * fg_loss_per_sample)
total_loss.backward()

print(f'Total BG gradient contribution: {num_bg_anchors * bg_loss_per_sample.grad.item():.1f}')
print(f'Total FG gradient contribution: {num_fg_anchors * fg_loss_per_sample.grad.item():.1f}')
Try answering this question with an AI coach

Senior questions

13How would you debug NaNs or unstable loss during large-scale distributed vision training?

Debugging NaNs or loss instability in large-scale Distributed Data Parallel (DDP) vision training requires isolating whether the root cause originates from bad data inputs, numerical overflow/underflow in mixed precision (AMP), or optimization instabilities across workers. First, establish determinism and safety hooks: enable anomaly detection (`torch.autograd.set_detect_anomaly(True)`), register gradient/activation hooks to catch the exact layer where NaNs appear, and add strict data validation in the DataLoader (asserting finite values, checking for 0-byte corrupt images, empty bounding boxes, or zero-variance normalization divisors). Minibatch-level logging should track per-rank loss, input URIs, gradient norm before clipping, and GradScaler scale factors. Second, inspect Automatic Mixed Precision (AMP) and dynamic loss scaling: in FP16, large gradients easily overflow (`> 65504`), causing the loss scaler to skip steps and repeatedly halve its scale factor until scale reaches zero; switching unstable operations (e.g., softmax, LayerNorm, focal loss exponents, or bounding box IoU denominators) to FP32 or adopting BF16 (which matches FP32's dynamic range) typically stabilizes training. Finally, check DDP-specific pitfalls like all-reduce operations propagating NaNs from a single worker to all ranks, learning rate warmup scaling (e.g., linear scaling rule with large global batch sizes), and gradient clipping.

import torch
from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

for step, (images, targets, uris) in enumerate(dataloader):
    # 1. Input sanitization
    if not torch.isfinite(images).all():
        print(f"Corrupt input detected from URIs: {uris}")
        continue
        
    optimizer.zero_grad(set_to_none=True)
    
    with autocast(dtype=torch.float16):
        outputs = model(images)
        loss = criterion(outputs, targets)
    
    if not torch.isfinite(loss):
        print(f"NaN/Inf loss at step {step} on rank {torch.distributed.get_rank()}; skipping step.")
        continue
        
    scaler.scale(loss).backward()
    
    # Unscale before clipping to inspect true gradient norms
    scaler.unscale_(optimizer)
    grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    
    if torch.isnan(grad_norm) or torch.isinf(grad_norm):
        print(f"Warning: Grad norm is {grad_norm}. Scaler will adjust.")
        
    scaler.step(optimizer)
    scaler.update()
Try answering this question with an AI coach

14How would you design an online digital video stabilization component for a camera with frame jitter?

Designing an online digital video stabilization pipeline for real-time camera jitter involves four primary stages: motion estimation, motion smoothing, motion compensation (warping), and border handling. 1. **Inter-Frame Motion Estimation:** Extract 2D sparse keypoints across consecutive frames using fast feature detectors (e.g., ORB, FAST, or Shi-Tomasi corners) and compute correspondences using Lucas-Kanade optical flow or feature descriptor matching. Estimate an inter-frame geometric transformation (such as an Affine or Homography model) using RANSAC to reject outlier matches caused by independently moving foreground objects. 2. **Motion Trajectory Accumulation & Online Smoothing:** Integrate frame-to-frame transforms over time to maintain the cumulative camera path $P_t = P_{t-1} \cdot H_t$. Apply an online smoothing filter—such as a 1D/2D Kalman filter or a short-window causal moving average—to separate high-frequency unintended jitter from low-frequency intentional camera panning. 3. **Compensation & Image Warping:** Compute the correction transform $C_t = S_t \cdot P_t^{-1}$ (where $S_t$ is the smoothed path) and warp the current frame using bilinear/bicubic interpolation. 4. **Border Handling & Latency Constraints:** Warping introduces missing edge pixels / black borders; resolve this by applying a fixed dynamic crop factor (e.g., 5-10% zoom-in) with border extrapolation or adaptive scale. In an online setting, maintain minimal buffering (1–3 lookahead frames) to bound processing latency.

import cv2
import numpy as np

class OnlineStabilizer:
    def __init__(self, crop_ratio=0.9):
        self.prev_gray = None
        self.smoothed_x = 0.0
        self.smoothed_y = 0.0
        self.smoothed_a = 0.0
        self.alpha = 0.85  # Low-pass filter smoothing coefficient
        self.crop_ratio = crop_ratio

    def process_frame(self, frame):
        curr_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        h, w = curr_gray.shape
        
        if self.prev_gray is None:
            self.prev_gray = curr_gray
            return frame
            
        # 1. Feature detection & tracking
        p0 = cv2.goodFeaturesToTrack(self.prev_gray, maxCorners=200, qualityLevel=0.01, minDistance=30)
        p1, status, _ = cv2.calcOpticalFlowPyrLK(self.prev_gray, curr_gray, p0, None)
        
        good_p0 = p0[status == 1]
        good_p1 = p1[status == 1]
        
        # 2. Estimate rigid motion (dx, dy, da) with RANSAC
        T, inliers = cv2.estimateAffinePartial2D(good_p0, good_p1, method=cv2.RANSAC)
        if T is None: 
            return frame
            
        dx = T[0, 2]
        dy = T[1, 2]
        da = np.arctan2(T[1, 0], T[0, 0])
        
        # 3. Online low-pass filtering of jitter
        self.smoothed_x = self.alpha * self.smoothed_x + (1 - self.alpha) * dx
        self.smoothed_y = self.alpha * self.smoothed_y + (1 - self.alpha) * dy
        self.smoothed_a = self.alpha * self.smoothed_a + (1 - self.alpha) * da
        
        diff_x = self.smoothed_x - dx
        diff_y = self.smoothed_y - dy
        diff_a = self.smoothed_a - da
        
        # 4. Warp and crop frame
        M = np.array([
            [np.cos(diff_a), -np.sin(diff_a), diff_x],
            [np.sin(diff_a),  np.cos(diff_a), diff_y]
        ])
        stabilized = cv2.warpAffine(frame, M, (w, h))
        
        # Center-crop to remove boundary artifacts
        cw, ch = int(w * self.crop_ratio), int(h * self.crop_ratio)
        x1, y1 = (w - cw) // 2, (h - ch) // 2
        stabilized = cv2.resize(stabilized[y1:y1+ch, x1:x1+cw], (w, h))
        
        self.prev_gray = curr_gray
        return stabilized
Try answering this question with an AI coach

15How would you optimize a real-time vision pipeline when the detector is slower than the target frame rate?

When an object detector cannot sustain the target frame rate (e.g., running at 15-20 FPS on a 60 FPS video feed), a robust production architecture decouples detection from the real-time presentation loop using an asynchronous multi-threaded pipeline combining heavy detection with lightweight tracking. In this hybrid architecture, an ingestion thread captures video frames continuously. The heavy detector runs asynchronously as a 'keyframe detector' on a background worker thread. Meanwhile, a lightweight tracker (such as optical flow Lucas-Kanade, ByteTrack/BoT-SORT, or a fast correlation filter/Kalman filter) runs synchronously on every frame at full 60 FPS, maintaining object state, identity, and smooth bounding box trajectories. To handle lag and backpressure without introducing unbounded latency or stale frames, bounded ring buffers and latest-frame drop policies are used. When the detector finishes a frame T_0 at time T_curr, its output is stale. The pipeline performs coordinate realignment/back-projection: it associates the delayed detection results with the historical tracklet state at T_0, updates identities and missed tracks, and propagates the corrections forward to T_curr via the tracking motion vectors or Kalman filter predict steps.

import queue
import threading

frame_queue = queue.Queue(maxsize=1)  # Drop stale frames, keep latest
det_result_queue = queue.Queue()

def detector_worker():
    while True:
        frame, frame_id, timestamp = frame_queue.get()
        boxes, scores, classes = heavy_detector.infer(frame)
        det_result_queue.put({'frame_id': frame_id, 'boxes': boxes, 'timestamp': timestamp})

def realtime_pipeline(video_stream):
    tracker = FastTracker()  # e.g. Optical flow or Kalman tracker
    
    for frame, frame_id, timestamp in video_stream:
        if frame_queue.empty():
            frame_queue.put((frame, frame_id, timestamp))
            
        if not det_result_queue.empty():
            det_result = det_result_queue.get()
            tracker.reconcile_and_correct(det_result, current_frame_id=frame_id)
            
        active_tracks = tracker.update(frame)
        display_or_downstream(frame, active_tracks)
Try answering this question with an AI coach