NLP Pipelines

1. Text Preprocessing

Preprocessing converts raw text into a form suitable for feature extraction. Choices here are hyperparameters — they affect downstream performance and should be evaluated empirically.

Tokenization

• Whitespace tokenization: split on spaces/punctuation. Simple but misses sub-word structure.
• WordPiece / BPE: sub-word tokenization used by BERT/GPT; handles OOV by splitting rare words into known pieces.

Common Preprocessing Steps

• Lowercasing: reduces vocabulary size; skip for proper-noun-heavy or case-sensitive tasks.
• Punctuation removal: removes noise; be careful with negation (don't → dont changes meaning).
• Stopword removal: removes high-frequency low-signal words (the, a). Skip for sentiment/negation tasks where words like "not" matter.
• Stemming (Porter, Snowball): aggressively strips suffixes (running → run, studies → studi). Fast but may produce non-words.
• Lemmatization (spaCy): returns valid base forms using vocabulary + POS (studies → study, running → run if verb). Slower but accurate and interpretable.
• Regex cleaning: remove URLs, HTML tags, special characters before tokenization.

import spacy

nlp = spacy.load("en_core_web_sm")

def preprocess(text: str, remove_stopwords: bool = True) -> list[str]:
    doc = nlp(text.lower())
    tokens = []
    for token in doc:
        if token.is_punct or token.is_space:
            continue
        if remove_stopwords and token.is_stop:
            continue
        tokens.append(token.lemma_)
    return tokens

# Example
preprocess("The studies were running behind schedule.")
# ['study', 'run', 'behind', 'schedule']

2. Feature Extraction — TF-IDF and Embeddings

TF-IDF

$$\text{TF-IDF}(t, d) = \text{TF}(t,d) \times \log\frac{N}{df(t)}$$

• TF(t, d): frequency of term t in document d (often log-scaled).
• IDF: penalizes terms that appear in many documents — the $\log(N / df(t))$ term is small when $df(t)$ is large, down-weighting common words like "the" that appear in nearly every document.
• Result: sparse, high-dimensional, interpretable matrix. Strong baseline for classification.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = ["The cat sat on the mat", "Dogs are loyal animals", "Cats and dogs are pets"]
vec = TfidfVectorizer(max_features=5000, ngram_range=(1, 2), sublinear_tf=True)
X = vec.fit_transform(corpus)  # sparse matrix (n_docs, vocab_size)

Word2Vec / GloVe

Dense embeddings trained on large corpora. Each word maps to a 100–300 dimensional vector. Semantic similarity is measured via cosine distance (king - man + woman ≈ queen).

OOV problem: words not in training vocabulary have no vector. Mitigations: sub-word models (fastText), WordPiece tokenization.

Document embedding via averaging: average the word vectors for all tokens in a document. Fast but loses word order and can conflate antonyms and negations.

import numpy as np

def doc_embedding(tokens: list[str], model) -> np.ndarray:
    vecs = [model[t] for t in tokens if t in model]
    if not vecs:
        return np.zeros(model.vector_size)
    return np.mean(vecs, axis=0)

Sentence Transformers (SBERT)

Encode entire sentences into a single dense vector using a fine-tuned transformer. Better than averaging for multi-word meaning, negation, and sentence-level semantics. Use for: semantic search, clustering, similarity scoring.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = ["How do I reset my password?", "I forgot my login credentials"]
embeddings = model.encode(sentences)  # shape (2, 384)

sim = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
print(f"Similarity: {sim:.3f}")  # ~0.82 — semantically close

3. Text Classification Pipeline

sklearn Pipeline (Fast Baseline)

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=50000, ngram_range=(1, 2),
                               sublinear_tf=True, strip_accents="unicode")),
    ("clf", LogisticRegression(max_iter=1000, C=1.0, class_weight="balanced")),
])

# Cross-validation
scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="f1_macro")
print(f"CV macro F1: {scores.mean():.3f} ± {scores.std():.3f}")

pipeline.fit(X_train, y_train)

Handling class imbalance: use class_weight="balanced" in LogisticRegression. SMOTE is not directly applicable to sparse TF-IDF vectors — prefer class weights or oversampling raw documents before vectorization.

Fine-tuning a HuggingFace Transformer

from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                           TrainingArguments, Trainer)
from datasets import Dataset

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True,
                     padding="max_length", max_length=128)

dataset = Dataset.from_dict({"text": X_train, "label": y_train})
dataset = dataset.map(tokenize, batched=True)

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=num_classes
)

args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(model=model, args=args, train_dataset=dataset)
trainer.train()

When to use BERT vs. TF-IDF + LR: BERT wins on accuracy with sufficient data (>1K examples) and handles context and semantics. TF-IDF wins on latency, interpretability, and cost. Always benchmark both before deciding.

4. Named Entity Recognition (NER)

NER identifies and classifies named entities (persons, organizations, locations, dates) in text.

IOB / BIO Tagging Scheme

Each token is assigned a label:

• B-TYPE: Beginning of an entity of the given type
• I-TYPE: Inside (continuation) of an entity
• O: Outside any entity

Example: "Apple Inc. was founded in Cupertino"
→ Apple/B-ORG Inc./I-ORG was/O founded/O in/O Cupertino/B-LOC

IOB tagging is needed because entity spans can be multiple tokens; without the B/I distinction, adjacent same-type entities would be ambiguous.

spaCy Out-of-the-Box NER

import spacy

nlp = spacy.load("en_core_web_sm")
text = "Tim Cook announced Apple's new product in San Francisco on Monday."
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text!r:30s} {ent.label_:10s} {spacy.explain(ent.label_)}")
# 'Tim Cook'       PERSON     People, including fictional
# 'Apple'          ORG        Companies, agencies, institutions
# 'San Francisco'  GPE        Geopolitical entity
# 'Monday'         DATE       Absolute or relative dates

Fine-tuning for Custom Entities

Use spaCy's training CLI or HuggingFace token-classification task with a dataset in IOB format. Annotate ~500–1000 examples per entity type for reasonable performance.

NER Evaluation

• Exact match (strict) F1: entity span and type must both match exactly.
• Partial match F1: overlapping spans receive partial credit.
• Token-level F1: treats each token label independently; generally higher and less informative than entity-level F1.

5. Evaluation Metrics

ROUGE (Recall-Oriented Understudy for Gisting Evaluation): measures how much of the reference summary appears in the generated output. ROUGE-L uses longest common subsequence to capture fluency. Used for summarization.

BLEU (Bilingual Evaluation Understudy): precision-focused — what fraction of the generated n-grams appear in the reference. A brevity penalty prevents short outputs from gaming precision. Limitations: no fluency signal, no paraphrase credit, sensitive to tokenization.

When metrics mislead: ROUGE/BLEU reward surface overlap and miss semantically equivalent paraphrases. Supplement with BERTScore (embedding-based similarity) or human evaluation for high-stakes tasks.

Practice Questions

Action Items

• Explain TF-IDF from first principles: why $\log(N/df)$ — what does it penalize?
• Build a sklearn Pipeline with TfidfVectorizer + LogisticRegression from memory.
• Describe the IOB tagging scheme for NER and why it's needed.
• Know BLEU vs. ROUGE: which is recall-based, which is precision-based, and when each is used.
• Explain why averaging word embeddings to get document embeddings can fail (antonyms, negation).
• Know when to use a fine-tuned transformer vs. TF-IDF + logistic regression.