LoRA fine-tuning + retrieval-augmented generation over 40 public-domain novels, built entirely on free Colab GPUs — with an evaluation section that reports the numbers that didn't work, not just the ones that did.
Most "make an LLM sound like X" projects lean entirely on prompting: a system prompt says "you are a Victorian gentleman," and a large frontier model does a passable job, because it has broad enough world knowledge to fake the register convincingly. That's a legitimate approach for a demo. It's also a thin one — it doesn't teach you anything about actually shaping a model's behavior, and it doesn't transfer to domains where the base model's prior is weak, wrong, or simply absent.
I wanted to answer a narrower, more interesting question: can a 3B-parameter open model be taught a specific register through supervised fine-tuning, cheaply enough to run on a free GPU, and can that fine-tune be measured rather than just eyeballed?
The single most common failure mode in period-style generation is faux-archaic filler — sprinkling in thou, thee, thy, dost and calling it done. That's costume, not voice. Actual Victorian prose has a distinctive sentence rhythm, vocabulary register, and level of formality that has almost nothing to do with archaic pronouns. Getting this right required an explicit ban on that vocabulary in both the synthetic data generation prompt and the inference-time system prompt.
The Victorian angle was partly for fun — period fiction has enough distinctive rhythm to make a fine-tuning effect visible and evaluable — and partly practical. Project Gutenberg gives unlimited public-domain 19th-century text, which meant I could build both the fine-tuning corpus and the retrieval corpus from the same 40 source novels, with zero licensing concerns and zero manual data collection cost.
VictorianGPT isn't one monolithic model — it's a fine-tuned generator paired with a retrieval layer, connected by a routing decision made at inference time.
Query
│
▼
Embed query (MiniLM-L6-v2)
│
▼
similarity_search_with_score against ChromaDB (40-novel corpus)
│
├── distance < threshold ──▶ HISTORIAN MODE
│ inject retrieved passage as grounding context
│
└── distance ≥ threshold ──▶ EMPATHY MODE
no passage injected, persona-only response
│
▼
LoRA fine-tuned Qwen2.5-3B-Instruct generates the reply
Early prototypes always retrieved something and injected it into every response — including for queries like "I had a rough day at work." That produced answers that awkwardly shoehorned in irrelevant plot details from whichever novel happened to score highest similarity, even when nothing in the corpus was actually relevant to what the user said.
A confidence threshold on the retrieval distance score gives the system a way to say "nothing relevant here" instead of forcing a match. Below the threshold, there's a genuinely close passage and the model should ground its answer in it — Historian Mode. Above the threshold, the model falls back to persona-only generation with period-appropriate emotional register — Empathetic Companion Mode. This is a standard RAG-with-abstention pattern, but it's the single design decision that made the chatbot feel coherent instead of randomly literary.
The whole project runs as five sequential notebooks, each depending on the artifacts of the last — data collection, dataset construction, fine-tuning, retrieval indexing, and evaluation.
40 curated 19th-century and Gothic novels scraped from Project Gutenberg — Dickens, the Brontës, Stoker, Wells, Doyle, Hardy, Austen, Eliot, Gaskell, Wilkie Collins, and more. Gutenberg boilerplate stripped, punctuation normalized, NLTK sentence tokenization, chunked into 6-sentence windows with 2-sentence overlap to preserve context across chunk boundaries.
45,881 authentic pairs extracted by regex from consecutive quoted dialogue in the novels — zero manual annotation. 432 synthetic pairs generated by Qwen2.5-3B itself, translating modern first-person emotional prompts into Victorian register, to cover conversational territory the novels don't: everyday modern statements like "I feel anxious after an interview."
Qwen2.5-3B-Instruct, 4-bit via Unsloth, LoRA rank 16 (α=16) across all attention and MLP projections, updating ~1–2% of parameters. Single epoch, effective batch size 16, on a free Colab T4 — a hard constraint, not an oversight, and one that shows up honestly in the evaluation numbers below.
Every cleaned novel embedded with sentence-transformers/all-MiniLM-L6-v2 and indexed into ChromaDB, processed file-by-file to stay within Colab's RAM limits. Confidence-threshold routing between Historian and Empathy modes at inference time, described above.
Perplexity (base vs fine-tuned), retrieval hit-rate@3 and MRR on a hand-labeled query set, router accuracy on a separate labeled set, and an LLM-as-judge persona-consistency score. Built from scratch — there's no off-the-shelf "Victorian persona benchmark" to import.
This is the part I think matters most for anyone evaluating this project critically — and it's also the part most similar "fine-tune + RAG chatbot" projects don't do at all. Reporting only the metrics that flatter the system isn't evaluation, it's a highlight reel. Here's everything, including what's currently broken.
Left: perplexity. Fine-tuning nearly halves perplexity on a sampled subset of training data. Right: retrieval vs. routing. The retriever finds the right novel 92% of the time — but the fixed distance threshold that decides whether to use it is only right half the time. Two different failure surfaces, easy to conflate if you only report "RAG works."
Base Qwen2.5-3B vs. the fine-tuned model, evaluated on a 150-example sampled subset of the training data using standard cross-entropy-based perplexity: 36.48 → 19.68, a 46.0% reduction.
This is not a true held-out evaluation. The original training run didn't reserve a validation split, so this measures in-distribution fit on a sampled subset the model has already seen — not generalization to genuinely unseen Victorian text. It's a legitimate signal that LoRA meaningfully shifted the model's distribution toward the target register, but it's a narrower claim than "generalizes well." A proper held-out split is the obvious next step before trusting this number further.
39 hand-labeled (query, expected_novel) pairs — one plot-specific question per novel in the corpus, e.g. "What happens to the ship Demeter in the storm?" → dracula.txt — checked against the top-3 ChromaDB results: 92.31% hit-rate@3, MRR 0.906. This is the strongest number in the project. Dense sentence embeddings are well-suited to exactly this task — distinctive plot content maps to distinctive semantic regions, and MiniLM is a solid, well-tested retrieval embedding model at this scale.
40 hand-labeled queries — 20 that should trigger Historian Mode, 20 that should trigger Empathy Mode — checked against the fixed-threshold router's predictions: 50.0% accuracy. Coin-flip performance.
A 50% router accuracy sitting right next to a 92% retrieval accuracy tells a specific, useful story: the retriever is finding the right documents — the distance-score threshold that decides whether to actually use them is miscalibrated. These are two separate failure surfaces that a single "RAG works" claim would have hidden. The threshold was set as a placeholder during development and never tuned against the real distribution of distance scores the corpus produces. It's fixable — a threshold sweep against this same labeled set would very likely recover most of the lost accuracy — but it's currently unfixed, and reporting a correct 50% beats implying an untested 90%.
An LLM-as-judge script rates sampled generations 1–5 for Victorian tone adherence. It currently returns 1.00/5 across every sample — implausible given the perplexity improvement, and almost certainly a pipeline bug (likely token-truncation cutting off the judge's digit response, or an under-specified prompt) rather than a real finding about persona quality. It's included here anyway, because an evaluation section that only reports numbers that "worked" isn't really an evaluation.
| Metric | Result | Sample Size | Status |
|---|---|---|---|
| Dataset size | 46,313 | full dataset | reliable |
| Perplexity (base → fine-tuned) | 36.48 → 19.68 | 150 sampled | reliable, in-distribution only |
| Retrieval hit-rate@3 | 92.31% | 39 labeled | reliable |
| Retrieval MRR | 0.906 | 39 labeled | reliable |
| Router accuracy | 50.0% | 40 labeled | needs recalibration |
| Persona-consistency | 1.00 / 5 | 5 samples | judge script broken |
A few pieces of this pipeline generalize well beyond a Victorian chatbot — the confidence-routing pattern, the synthetic-data-to-cover-a-distribution-gap trick, and computing perplexity delta as a fine-tuning sanity check are all reusable on any small-model fine-tune + RAG project.
def route_query(query, vectorstore, threshold):
result = vectorstore.similarity_search_with_score(query, k=1)
if not result:
return "Empathy", None
doc, distance = result[0]
if distance < threshold:
# strong match found — ground the response in retrieved text
return "Historian", doc.page_content
else:
# nothing relevant — fall back to persona-only generation
return "Empathy", None
import torch, math
def compute_perplexity(model, tokenizer, texts, max_length=512):
model.eval()
total_loss, total_tokens = 0.0, 0
for text in texts:
enc = tokenizer(text, return_tensors="pt",
truncation=True, max_length=max_length).to(model.device)
with torch.no_grad():
out = model(**enc, labels=enc["input_ids"])
n = enc["input_ids"].shape[1]
total_loss += out.loss.item() * n
total_tokens += n
return math.exp(total_loss / total_tokens)
base_ppl = compute_perplexity(base_model, tokenizer, held_out_texts)
ft_ppl = compute_perplexity(finetuned_model, tokenizer, held_out_texts)
# 36.48 -> 19.68, a 46.0% reduction
The authentic dialogue pairs teach the model period cadence and vocabulary — but novels don't contain first-person modern statements like "I feel anxious after an interview." Without synthetic coverage, the model would have no training signal for the single most common category of real user input: everyday emotional statements. The synthetic set is small (432 examples, under 1% of the total) precisely because its job isn't to teach the voice — the 45,881 authentic pairs already do that — its job is narrowly to teach the model to apply that voice to modern input it would otherwise never have seen during training.
Two infrastructure lessons cost real time and are worth naming plainly. First, ChromaDB doesn't persist across Colab sessions unless persist_directory explicitly points into a mounted Drive path — an earlier run's vector index lived only on the runtime's local disk and was silently wiped when that session ended, meaning the "finished" RAG layer had to be rebuilt from the cleaned novel files before evaluation could even start. Second, LoRA checkpoints save under checkpoints/checkpoint-<step>/ by default, not into a tidy models/ folder — a mismatch between the README's aspirational structure and what actually landed on disk that took a directory walk to untangle.
Neither is a deep technical insight. Both are the kind of thing that doesn't show up in a tutorial and only shows up once you've actually run the pipeline end-to-end on infrastructure you don't fully control.
The parts of this project I'm most glad I did aren't the fine-tuning or the RAG pipeline themselves — both are increasingly standard, well-tooled techniques. The parts that actually taught me something were building the evaluation from scratch, since there's no off-the-shelf "Victorian persona benchmark" to import, and then sitting with a 50% router accuracy number instead of quietly tuning the threshold before writing any of this up. That process — build, measure, find the thing that doesn't work, say so precisely — is a more honest description of doing ML work than a polished demo GIF ever is.
Full code, notebooks, and evaluation scripts: github.com/Hamza-Faarooq/Victorian_GPT