Agents & GenAI

1. How LLMs Work

Transformer Architecture

Input text is tokenized and embedded into dense vectors. Multi-head self-attention lets every token attend to every other token:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The $\frac{1}{\sqrt{d_k}}$ scaling prevents softmax saturation when key dimension is large. Stacked transformer blocks progressively build richer representations; the final layer projects to vocabulary logits.

Training Pipeline

Context Window

Context window = maximum tokens the model can attend to at once (4K–1M+ depending on model). Rule of thumb: ~750 words ≈ 1,000 tokens. Long-context models degrade in the middle of the context ("lost in the middle" effect) — important for document QA tasks.

2. Retrieval-Augmented Generation (RAG)

RAG is the standard pattern for giving LLMs access to proprietary or up-to-date data without fine-tuning.

Classic RAG Pipeline

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

# Index documents
docs = ["Capital One Q3 earnings beat expectations...", "Fraud detection system updated..."]
doc_embeddings = model.encode(docs)  # shape: (n_docs, 384)

# At query time
query_emb = model.encode(["What were Q3 earnings?"])
scores = np.dot(query_emb, doc_embeddings.T)  # cosine similarity
top_k = np.argsort(scores[0])[::-1][:3]
retrieved = [docs[i] for i in top_k]

RAG Evaluation Metrics

Chunking Strategy

• Fixed-size chunks (512 tokens, 50-token overlap) — simple, works well for uniform documents.
• Semantic chunking — split at sentence/paragraph boundaries; better for heterogeneous documents.
• Hierarchical indexing — store both document-level summaries and fine-grained chunks; retrieve at both levels.

3. Agents and Tool Use

The Agent Loop

while not done:
    observation = env.get_state()
    thought = llm.reason(observation, tools, history)
    action = parse_action(thought)
    result = tools.execute(action)
    history.append((action, result))

ReAct Pattern

Reasoning + Acting interleaved. The model emits a Thought: (chain-of-thought) before each Action:. CoT before action dramatically reduces hallucination in tool selection.

Thought: I need the customer's transaction history.
Action: query_database(customer_id=12345, table="transactions", limit=10)
Observation: [{"date": "2025-01-15", "amount": 250.00}, ...]
Thought: I now have the data. The last 10 transactions show...
Final Answer: ...

Multi-Agent Reliability

Each added hop multiplies failure probability. A 90%-reliable step run 8 times sequentially succeeds only $0.9^8 \approx 43\%$ of the time. Mitigations: parallelism where possible, retry logic, checkpointing long workflows, human-in-the-loop for irreversible actions.

4. Prompt Engineering

Core Techniques

Prompt Injection Defense

External content (user input, retrieved docs) can contain adversarial instructions. Defenses: separate system context from user-controlled input; sanitize input for instruction-like patterns; validate outputs against expected schema before acting on them.

5. Fine-Tuning vs. RAG vs. Prompting

LoRA: Fine-tune only low-rank decomposition of weight matrices: $\Delta W = AB$ where $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, $r \ll \min(d,k)$. Reduces trainable parameters by ~10,000× vs. full fine-tune; base weights are unchanged and the adapters can be merged at inference.

6. Evaluation of GenAI Systems

LLM-as-Judge Pitfalls

• Position bias — prefers the first option in pairwise comparisons
• Verbosity bias — prefers longer answers regardless of quality
• Self-preference bias — GPT-4 rates GPT-4 outputs higher

Mitigation: swap order across evaluation runs, use rubrics, calibrate against human judgments.

7. End-to-End RAG Example

import anthropic
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

# --- Indexing (offline) ---
embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunks = load_and_chunk_documents("reports/")
embeddings = embedder.encode(chunks).astype("float32")
index = faiss.IndexFlatIP(embeddings.shape[1])
faiss.normalize_L2(embeddings)
index.add(embeddings)

# --- Query (online) ---
client = anthropic.Anthropic()

def answer(query: str, k: int = 4) -> str:
    q_emb = embedder.encode([query]).astype("float32")
    faiss.normalize_L2(q_emb)
    _, idx = index.search(q_emb, k)
    context = "\n\n---\n\n".join(chunks[i] for i in idx[0])

    msg = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=1024,
        system=(
            "You are a financial analyst assistant. Answer only from the provided context. "
            "If the answer isn't in the context, say 'I don't have that information.'"
        ),
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {query}"
        }]
    )
    return msg.content[0].text

Capital One example: A document Q&A system over earnings reports, regulatory filings, and internal risk memos allows analysts to query thousands of pages in seconds, with citations back to source paragraphs for auditability.