LLM and Generative AI Engineer Interview Questions
15 selected LLM and generative AI interview questions grouped by seniority level. Use them to review fundamentals, practical trade-offs, and senior-level production reasoning.
1Explain subword tokenization and why it is preferred over word-level or character-level tokenization in modern language models.
Subword tokenization is a hybrid text segmentation approach that breaks text into variable-length morphological chunks or frequency-based substrings (such as 'un', 'break', 'able') rather than whole words or individual characters. Algorithms like Byte-Pair Encoding (BPE), WordPiece, and Unigram LM learn a fixed-size vocabulary from a training corpus where frequent words remain intact as single tokens, while rare or unseen words are decomposed into known subword units. Subword tokenization is preferred in modern language models because it balances vocabulary size, sequence length, and out-of-vocabulary (OOV) robustness. Pure word-level tokenization requires an excessively large vocabulary (leading to huge embedding matrices) and still suffers from OOV tokens mapped to generic '[UNK]' tokens. Conversely, pure character-level tokenization eliminates OOV issues but yields very long sequences that drastically increase computational complexity in attention mechanisms (which scale quadratically with sequence length) and dilute semantic density per token. Subword tokenization strikes an optimal trade-off by keeping sequence lengths manageable, vocabulary sizes practical (typically 32k to 128k tokens), and OOV rates at zero (especially when combined with byte-level fallbacks).
2Explain the difference between static word embeddings, contextual embeddings, and transformer hidden states.
Static word embeddings, contextual embeddings, and transformer hidden states represent progressive evolutions in how text representations capture meaning and syntactic context. 1. Static word embeddings (e.g., Word2Vec, GloVe, FastText) assign a single, fixed vector to each vocabulary token regardless of its sentence context. In this paradigm, polysemous words like 'bank' (river bank vs. financial bank) or 'apple' (fruit vs. tech company) have identical vector representations in all contexts, relying on a static lookup table. 2. Contextual embeddings (e.g., early ELMo, BERT token representations, or sentence embeddings from Bi-Encoders) produce representations where the vector for a token is a dynamic function of its surrounding context. In BERT or ELMo, 'bank' in 'river bank' receives a completely different embedding vector than 'bank' in 'deposit money at the bank'. 3. Transformer hidden states refer to the intermediate vector representations produced at each individual layer of a transformer network during a forward pass. Given input token embeddings at layer 0, each successive transformer layer applies self-attention and feed-forward transformations, producing a sequence of hidden state vectors h_l at layer l. While the final layer's hidden states act as high-level contextual embeddings, lower and middle hidden states capture low-level syntactic, lexical, and structural features. Thus, transformer hidden states encompass the full vertical continuum of layer-by-layer representations across the network.
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
text1 = 'River bank'
text2 = 'Bank deposit'
inputs1 = tokenizer(text1, return_tensors='pt')
inputs2 = tokenizer(text2, return_tensors='pt')
with torch.no_grad():
out1 = model(**inputs1)
out2 = model(**inputs2)
# Layer 0 (Input token embeddings - static lookup before self-attention)
static_bank_1 = out1.hidden_states[0][0, 2] # 'bank'
# Layer 12 (Final contextual hidden state after all attention layers)
contextual_bank_1 = out1.hidden_states[12][0, 2]
contextual_bank_2 = out2.hidden_states[12][0, 1]
print('Cosine similarity of bank in different contexts (Layer 12):',
torch.cosine_similarity(contextual_bank_1, contextual_bank_2, dim=0).item())
3Explain what an embedding space represents and how cosine similarity is interpreted for text embeddings.
An embedding space is a continuous, high-dimensional vector space R^d where discrete textual entities (words, sentences, or documents) are mapped such that semantic, syntactic, or relational similarities correspond to geometric proximity and directional relationships. In this space, distances and angles reflect semantic relatedness. Cosine similarity measures the cosine of the angle theta between two vectors u and v, calculated as: Cosine Similarity(u, v) = (u . v) / (||u|| ||v||) Cosine similarity is interpreted in text embeddings as follows: - Range & Orientation: It produces a scalar value typically bounded in [-1, 1] (or [0, 1] for non-negative embeddings). A value close to 1.0 indicates that the two vectors point in virtually the same direction, reflecting high semantic similarity or topical alignment. A value near 0.0 implies orthogonality (semantic independence or un-relatedness), and negative values indicate opposite orientations. - Magnitude Invariance: Unlike Euclidean distance (L2 distance) or dot product, cosine similarity normalizes for vector length. In text embeddings, vector magnitude can sometimes correlate with sequence length, token frequency, or term specificity. By focusing purely on directional alignment, cosine similarity isolates semantic orientation from vector magnitude differences.
4Explain the difference between token embeddings, positional embeddings, and segment or type embeddings in transformer inputs.
In transformer inputs (notably BERT-style architectures), the input representation for each token is typically formed by element-wise summing three distinct embedding vectors: 1. Token Embeddings: Map discrete vocabulary token IDs into dense vectors representing the core semantic and lexical identity of the tokens. 2. Positional Embeddings: Inject information about token order and sequential index into the representation, compensating for the fact that self-attention is inherently permutation-invariant. 3. Segment (or Token Type) Embeddings: Distinguish between different text spans or sentences packed into a single input sequence (such as Sentence A vs. Sentence B in paired classification or question-answering tasks). Combining these embeddings provides a single dense input tensor encoding token meaning, position, and sequence grouping before passing into the first transformer layer.
import torch
import torch.nn as nn
vocab_size, max_seq_len, num_segments, d_model = 30522, 512, 2, 768
tok_embed = nn.Embedding(vocab_size, d_model)
pos_embed = nn.Embedding(max_seq_len, d_model)
seg_embed = nn.Embedding(num_segments, d_model)
input_ids = torch.tensor([[101, 7592, 102, 2023, 102]]) # Token IDs
type_ids = torch.tensor([[0, 0, 0, 1, 1 ]]) # Segment IDs (Sentence A vs B)
positions = torch.arange(input_ids.size(1)).unsqueeze(0) # Indices: [0, 1, 2, 3, 4]
# Final representation is the element-wise sum
input_rep = tok_embed(input_ids) + pos_embed(positions) + seg_embed(type_ids)
print(input_rep.shape)
5Explain attention as a mechanism for relating tokens in a sequence, including queries, keys, values, and multi-head attention.
Attention is a mechanism that allows tokens in a sequence to dynamically route information and weigh the relevance of all other tokens based on contextual matching. Linear projections convert each token's input into three vectors: - Query (Q): Represents what information the current token is seeking. - Key (K): Represents what attributes or content a token offers to match against queries. - Value (V): Contains the actual information payload to be aggregated. In scaled dot-product attention, attention scores are computed by multiplying Queries and Keys ($Q K^T$), scaled by $\frac{1}{\sqrt{d_k}}$ to prevent gradient vanishing across large dimensions, and normalized with a softmax function. The final output is the weighted sum of Values: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$ Multi-Head Attention (MHA) projects $Q$, $K$, and $V$ into multiple independent representational subspaces (heads) in parallel. This allows the model to simultaneously attend to different types of relationships (e.g., syntactic structure, coreference, long-range dependencies) across positions. The head outputs are concatenated and linearly projected back to the model dimension.
6Explain how attention masking differs for causal decoders versus bidirectional encoders and what behaviors it enables or prevents.
Attention masking controls which tokens are allowed to attend to which other tokens by setting attention logits (scores prior to softmax) to $-\infty$ for disallowed pairs, ensuring their post-softmax attention weight is strictly 0. 1. Causal Decoders (e.g., GPT, LLaMA): Use a lower-triangular causal (autoregressive) mask. A token at position $i$ can only attend to positions $j \le i$. This prevents attention to future tokens, enabling autoregressive token-by-token generation during inference and preventing future-token label leakage during parallelized training. 2. Bidirectional Encoders (e.g., BERT): Do not use a causal mask; every token can attend to all past and future tokens across the sequence. They use padding masks to prevent valid tokens from attending to empty `[PAD]` tokens in batched sequences. Bidirectional attention produces rich all-around contextual representations ideal for comprehension tasks, but prevents direct single-pass autoregressive text generation.
import torch
import torch.nn.functional as F
scores = torch.randn(3, 3)
# Create upper-triangular mask for future positions
causal_mask = torch.triu(torch.ones(3, 3, dtype=torch.bool), diagonal=1)
# Mask future positions with -inf before softmax
masked_scores = scores.masked_fill(causal_mask, float('-inf'))
atten_weights = F.softmax(masked_scores, dim=-1)
print(atten_weights)
7Explain the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures for language tasks.
The three primary transformer architectures differ fundamentally in their attention masking patterns and target operational objectives: 1. Encoder-Only (e.g., BERT, RoBERTa): Uses bidirectional self-attention where every token can attend to all other tokens in the sequence simultaneously. It produces rich contextual representations for an entire input sequence, making it ideal for classification, extractive QA, and feature representation. It cannot naturally generate autoregressive text. 2. Decoder-Only (e.g., GPT-3, Llama, Mistral): Uses causal (unidirectional) self-attention, where token $i$ can only attend to tokens at positions $j \le i$. It is trained autoregressively using next-token prediction and serves as the standard architecture for generative language models, code generation, and open-ended conversation. 3. Encoder-Decoder (e.g., T5, BART): Combines a bidirectional encoder with an autoregressive causal decoder. In addition to causal self-attention over generated tokens, the decoder uses cross-attention layers that query the encoder's output representations. This architecture is purpose-built for sequence-to-sequence transformation tasks such as translation and summarization.
8Explain how byte-level, Unicode-aware, and multilingual tokenizer choices affect model quality, cost, and fairness across languages.
Tokenizer design choices—such as byte-level vs. Unicode-aware segmentations and multilingual vocabulary allocations—directly impact downstream model quality, inference/training cost, and linguistic fairness. In terms of cost and fairness, tokenizers trained predominantly on English or Latin-script corpora allocate most vocabulary entries to English words and morphemes. Consequently, English achieves high compression (e.g., ~1.3 tokens per word), whereas non-Latin scripts (e.g., Arabic, Devanagari, Thai, Chinese) or low-resource languages are often fragmented into multiple subwords or raw UTF-8 bytes (often 3 to 6 tokens per word). This disparity is often called the 'token tax' or 'fertility rate imbalance': non-English users pay significantly more per unit of semantic content in API billing, consume context window limits much faster, and suffer higher latency. In terms of quality, byte-level tokenizers (like Byte-level BPE in GPT-2/GPT-4 or SentencePiece with byte fallback in LLaMA) avoid unseen-character crashes and out-of-vocabulary (UNK) errors entirely because any valid UTF-8 string decomposes into byte tokens. However, excessive byte fragmentation degrades representation quality because the transformer must spend layers recombining byte fragments into semantic concepts before performing high-level reasoning. Increasing multilingual vocabulary size (e.g., expanding from 32k to 128k+ tokens) balances fertility rates and improves downstream task performance across diverse languages, at the cost of a moderately larger input/output embedding layer.
9What tokenization artifacts appear in numeric reasoning, code generation, or rare Unicode text, and how can specialized tokenizers reduce them?
Tokenization artifacts occur when subword tokenizers partition structured, numeric, or rare text inconsistently, preventing the model from recognizing underlying semantic or syntactic regularity. Key artifacts include: 1. Numeric reasoning artifacts: Standard BPE tokenizers trained on general text split numbers into arbitrary chunk lengths based on frequency (e.g., '12345' might tokenize as ['12', '345'] while '12346' tokenizes as ['123', '46']). This inconsistent grouping breaks place-value alignment (digits, tens, hundreds) and hinders arithmetic reasoning. 2. Code generation artifacts: Indentation (leading spaces/tabs) and multi-character operators (e.g., '==', '!=', '->') often split irregularly across space-character boundaries, leading to indentation errors, bloated token counts in deeply indented code, and syntax corruptions. 3. Rare Unicode and Emoji artifacts: Multi-byte UTF-8 sequences (like complex emojis with zero-width joiners or rare scripts) are split into raw byte tokens that carry no individual semantic meaning, causing hallucinated characters or corrupted glyph rendering upon generation. Specialized tokenizers reduce these artifacts using tailored pre-tokenization rules and vocabulary constraints: - Digit splitting: Enforcing single-digit tokenization (e.g., regex splitting every digit `0-9` into its own token) ensures uniform place-value representation for mathematical reasoning. - Dedicated whitespace/indentation tokens: Adding explicit tokens for multi-space indentations (e.g., 2, 4, 8 spaces) and preserving programming language keywords/operators. - Regex pre-tokenization / Byte-level fallbacks: Using regex splitters (such as GPT-4/tiktoken regexes) that separate punctuation, letters, and numbers into strict categories before BPE merges are computed, preventing cross-category merges (e.g., preventing 'a=10' from merging into a single token).
import tiktoken
# tiktoken cl100k_base (GPT-4 / ChatGPT) enforces digit and whitespace handling
enc = tiktoken.get_encoding('cl100k_base')
num1 = '12345'
num2 = '12346'
code_indent = ' def foo():'
print('Tokens num1:', [enc.decode([t]) for t in enc.encode(num1)])
print('Tokens num2:', [enc.decode([t]) for t in enc.encode(num2)])
print('Tokens code:', [enc.decode([t]) for t in enc.encode(code_indent)])
10Reason about long-context attention trade-offs, including quadratic full attention, sliding-window attention, sparse or global attention, KV-cache cost, and attention dilution.
Scaling attention to long context windows presents trade-offs across computation, memory footprint, and model fidelity: 1. Quadratic Full Attention vs. Sliding-Window / Sparse Attention: Standard full attention scales quadratically ($O(N^2)$) in compute and activation memory with sequence length $N$. Sliding-window (local) attention restricts attention to a fixed neighborhood $W$, reducing complexity to $O(N \cdot W)$, but requires multiple layers to propagate information across distant tokens. Sparse or global attention patterns combine local windows with select global anchor tokens to retain $O(N)$ scaling while enabling long-range communication. 2. KV-Cache Memory Cost: During autoregressive generation, keys and values for all previous tokens are cached to avoid redundant computation. KV-cache memory scales linearly with sequence length ($O(B \cdot L \cdot H_{KV} \cdot D \cdot N)$). For very long contexts (32k–128k+ tokens), the KV-cache consumes dozens of gigabytes of GPU VRAM per batch, bottlenecking maximum batch size and memory bandwidth. 3. Attention Dilution (Lost-in-the-Middle): As context grows, the softmax denominator sums over tens of thousands of tokens, spreading probability mass thinly across irrelevant context. This entropy increase dilutes attention sharpness, degrading the model's ability to reliably recall specific information embedded in the middle of long prompts.
11Compare MHA, MQA, and GQA and explain how they affect KV-cache memory and decode throughput.
Multi-Head Attention (MHA), Multi-Query Attention (MQA), and Grouped-Query Attention (GQA) differ in how Key ($K$) and Value ($V$) heads are shared across Query ($Q$) heads: 1. Multi-Head Attention (MHA): Has an equal number of $Q$, $K$, and $V$ heads ($H_Q = H_{KV}$, 1:1 ratio). Each query head attends to its own independent key/value representations. While expressive, it requires caching distinct KV matrices for every head. 2. Multi-Query Attention (MQA): Uses multiple $Q$ heads ($H$) but only 1 shared $K$ head and 1 shared $V$ head ($H:1$ ratio). This reduces the KV cache size by a factor of $H$, but can lead to slight quality loss or training instability. 3. Grouped-Query Attention (GQA): Groups $Q$ heads into $G$ partitions, where each group shares a single $K$ and $V$ head (e.g., 8 $Q$ heads per KV head). GQA offers an optimal trade-off, recovering virtually all of MHA's modeling quality while keeping the memory benefits of MQA. Impact on KV-Cache & Decode Throughput: Autoregressive token generation (decoding) is memory-bandwidth bound because the GPU must transfer the entire KV cache from High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. By reducing the number of KV heads by $H/G$ (e.g., $4\times$ to $8\times$ in GQA, or $32\times+$ in MQA): - KV-cache memory footprint is reduced proportionally, enabling much larger serving batch sizes in GPU VRAM. - HBM memory read traffic per token drops substantially, dramatically increasing decode token throughput.
# Model with 32 Query Heads
num_q_heads = 32
mha_kv_heads = 32 # 1:1 ratio
gqa_kv_heads = 8 # 4:1 ratio (4 query heads per KV head)
mqa_kv_heads = 1 # 32:1 ratio (1 shared KV head)
print(f"KV Cache Size Relative to MHA:")
print(f"MHA: {mha_kv_heads / mha_kv_heads * 100:.1f}%")
print(f"GQA: {gqa_kv_heads / mha_kv_heads * 100:.1f}%")
print(f"MQA: {mqa_kv_heads / mha_kv_heads * 100:.1f}%")
12Explain how FlashAttention speeds up exact attention computation without changing attention outputs.
FlashAttention speeds up attention computation by making the algorithm IO-aware—minimizing the read and write memory traffic between slow GPU High Bandwidth Memory (HBM) and fast on-chip SRAM, rather than trying to reduce the total arithmetic FLOP count. Standard attention materializes intermediate N x N attention score and probability matrices in HBM, which causes a major memory bandwidth bottleneck. FlashAttention overcomes this through three key mechanisms: 1. Tiling: It splits the Query, Key, and Value matrices into blocks that fit entirely within GPU on-chip SRAM. 2. Online Softmax: It computes softmax incrementally over blocks by tracking running maximums and normalizer sums, updating partial outputs without needing the full materialized N x N matrix in memory. 3. Exact Recomputation: During the backward pass, it does not read stored intermediate attention matrices from HBM; instead, it recomputes them on the fly in SRAM from the stored running statistics. Because no approximations, low-rank factorizations, or token-dropping heuristics are used, the output is mathematically exact up to floating-point numerical precision, while reducing HBM memory footprint from O(N^2) to O(N).
13Design deterministic agentic workflows using planners, state machines, DAGs, typed intermediate state, bounded retries, and tool-result verification instead of open-ended agent loops.
Open-ended agent loops (e.g., unconstrained autonomous ReAct loops) in production often suffer from non-deterministic branching, infinite loops, runaway token spend, and state drift. A deterministic agentic workflow replaces free-form loops with structured, observable control flow: 1. State Machines and DAGs: Control flow is defined as an explicit Directed Acyclic Graph or finite state machine (e.g., LangGraph, Temporal, AWS Step Functions). Node transitions depend on explicit conditions and typed outcomes rather than open-ended model decisions. 2. Typed Intermediate State: State shared across nodes is modeled with strict schemas (e.g., Pydantic models or dataclasses). Nodes perform validated read and write operations, preventing schema drift or malformed state. 3. Planners: Structured planners emit a constrained plan upfront (e.g., an ordered list of enum-backed steps) or choose from a restricted set of valid state transitions rather than freely deciding next actions without constraints. 4. Tool-Result Verification: Outputs returned by tools are deterministically validated against schemas and business rules before updating state or passing to downstream LLM steps. 5. Bounded Retries & Fallbacks: Every step enforces explicit retry budgets, exponential backoffs, timeouts, and fallback transitions (e.g., escalating to human review or triggering safe abstention) to guarantee termination.
from pydantic import BaseModel
from typing import Optional, Literal
class WorkflowState(BaseModel):
user_query: str
extracted_id: Optional[str] = None
verification_status: Literal["PENDING", "VERIFIED", "FAILED"] = "PENDING"
retry_count: int = 0
max_retries: int = 3
def execute_validation_node(state: WorkflowState) -> WorkflowState:
if state.retry_count >= state.max_retries:
state.verification_status = "FAILED"
return state
try:
result = call_verification_service(state.extracted_id)
state.verification_status = "VERIFIED" if result.is_valid else "FAILED"
except Exception:
state.retry_count += 1
return state
14Design a confidence and abstention policy for an LLM assistant answering regulated-domain questions.
In regulated domains (such as healthcare, banking, legal, and compliance), incorrect answers carry regulatory penalties, legal liability, and safety risks. A robust confidence and abstention policy combines multi-signal calibrated confidence scoring, tiered response thresholds, and deterministic escalation workflows: 1. Multi-Signal Confidence Calibration: Raw LLM logprobs are often miscalibrated on out-of-domain queries. The confidence score should synthesize multiple independent signals: - Retrieval Grounding Score: Semantic similarity and re-ranking confidence of retrieved evidence chunks. - Claim-Level Entailment (NLI): Natural Language Inference models verifying that every extracted claim is entailed by the retrieved source context. - Semantic Entropy / Self-Consistency: Measuring semantic consistency across multiple sampled generations. - Model Token Logprobs: Minimum and average logprobs on key named entities and factual tokens. 2. Tiered Abstention Policy: - High Confidence (Score >= High Threshold): Directly serve the generated answer with inline citations. - Medium Confidence / Ambiguous (Low Threshold <= Score < High Threshold): Serve a conservative answer with explicit caveats, disclaimers, or ask the user for clarifying details. - Low Confidence / Out-of-Scope (Score < Low Threshold): Hard abstention with a standardized refusal message. 3. Escalation and Compliance Auditability: - Deterministic Escalation: Abstentions or critical discrepancies are automatically routed to human-in-the-loop (HITL) queues or agent ticketing systems with full context. - Audit Trail & Lineage: Full telemetry—including prompt hashes, retrieved document IDs, individual confidence component scores, and final routing decisions—must be logged for regulatory auditability.
15Design a model-routing strategy that chooses among small, medium, and large models based on request complexity, cost, risk, and quality requirements.
A production model-routing architecture directs incoming requests across small (e.g., 1B–8B SLMs), medium (e.g., 14B–70B models), and large (e.g., frontier or large MoE models) tiers by balancing complexity, latency, risk, and compute cost. The routing workflow generally combines static rules, predictive routing, and dynamic fallback cascades: 1. Deterministic/Static Policy Gates: Filter requests by customer tier, hard latency SLAs, regulatory/domain risk (e.g., medical diagnosis or legal drafting routed straight to top-tier models), or simple rule-matched tasks (e.g., basic regex/formatting to small models). 2. Predictive Complexity Routing: A fast, lightweight classifier (such as an embedding similarity lookup, a cross-encoder, or a small SLM router) scores request complexity, reasoning depth, and domain ambiguity to select the most cost-effective tier upfront. 3. Dynamic Execution & Escalation Cascades: Send the prompt first to a smaller model and evaluate output confidence (via token logprobs/entropy, structured schema validity, or guardrail checks). If confidence is below threshold or validation fails, the router escalates to a medium or large model. Key system trade-offs include router latency overhead versus compute savings, fallback timeout budgets under traffic spikes, and continuous evaluation (e.g., shadow evaluation to track output quality drift across tiers).
class DynamicModelRouter:
def __init__(self, small_client, medium_client, large_client, classifier, guardrail):
self.small = small_client
self.medium = medium_client
self.large = large_client
self.classifier = classifier
self.guardrail = guardrail
async def route_and_execute(self, request):
# 1. Deterministic Risk Gate
if request.risk_level == "high" or request.domain in ["legal", "medical_compliance"]:
return await self.large.generate(request.prompt)
# 2. Predictive Complexity Classifier
complexity = self.classifier.predict_complexity(request.prompt) # 0.0 to 1.0
if complexity < 0.35:
response = await self.small.generate(request.prompt)
if self.guardrail.is_acceptable(response):
return response
return await self.medium.generate(request.prompt) # Fallback
if complexity < 0.75:
response = await self.medium.generate(request.prompt)
if self.guardrail.is_acceptable(response):
return response
return await self.large.generate(request.prompt) # Fallback
# 3. High complexity frontier execution
return await self.large.generate(request.prompt)