Fine-Tuning GPT-2 and T5 for Product Description Generation

E-commerce platforms host millions of products, and manually writing unique, engaging descriptions for each one is resource-intensive. That’s where AI comes in. In this project, I fine-tuned two powerful Transformer models, GPT-2 and T5, to automatically generate product descriptions from structured metadata.

The Challenge

Turning raw product metadata (title, brand, category, specs) into fluent, marketing-style descriptions is tricky. The model needs to:

• Understand structured data
• Generate coherent, natural-sounding text
• Stay factual (no hallucinations!)
• Keep descriptions concise and relevant

The challenge: transforming structured metadata into natural product descriptions

Data Source

The dataset used for this project is the Amazon Product Data (2023 version) collected by the McAuley Lab at UCSD. This massive repository contains millions of product reviews and metadata. You can find the original dataset here.

For this experiment, I focused on a diverse subset of categories:

• Baby Products
• Pet Supplies
• Industrial & Scientific
• …and more!

Amazon Product Dataset with structured metadata fields

Data Preparation: Scrubbing the Data

Real-world data is messy. I started with a massive dataset of over 500k items, but after rigorous cleaning, I whittled it down to a high-quality set of 110k examples. Here’s how I did it:

1. Aggregation: Combined multiple category files (Baby, Pet, CDs, Grocery, etc.) into one unified dataset.
2. Null Handling:
   • Dropped duplicate rows
   • Removed entries with missing titles or descriptions
3. Text Cleaning (The Important Part):
   • HTML Stripping: Removed <br>, <b>, <p> and other tags
   • URL Removal: Stripped out http://... links so models don’t learn to generate fake URLs
   • Punctuation Fixes: Normalized spacing (e.g., “word.” → “word .”)
   • Unicode Normalization: Fixed special characters to standard ASCII/Unicode

The preprocessing steps from my notebooks included removing empty values, length filtering, and language detection to ensure high-quality English descriptions.

Step 1: Removing entries with empty or null values

Step 2: Filtering descriptions by length to ensure quality

Step 3: Validating data quality and consistency

Step 4: Detecting and filtering for English-language content

Full Cleaning & Data Preparation Pipeline

Methodology: Choosing the Right Models

I compared two architectures:

T5 (Text-to-Text Transfer Transformer)

• Model Used: t5-base (220 Million parameters)
• My Fine-Tuned Checkpoint: Hugging Face
• Architecture: Encoder-Decoder
• Strength: Treats every task as text-to-text (like translation)
• Why T5?: Pre-trained on diverse tasks, excellent for conditional generation

GPT-2 (Generative Pre-trained Transformer 2)

• Model Used: gpt2-medium (355 Million parameters)
• My Fine-Tuned Checkpoint: Hugging Face
• Architecture: Decoder-only (Causal LM)
• Strength: Predicts next token based on previous context
• Why GPT-2?: Excellent fluency and coherence

Input Representation: How We Talk to the Models

This was a critical design choice. The two models process data very differently.

GPT-2: The Storyteller

GPT-2 is a causal model: it just predicts the next word. We format the input as a continuous sequence with special separator tokens so it knows where each field begins and ends.

Example:

<|TITLE|> Premium Wireless Headphones <|BRAND|> Sony <|CATEGORY|> Electronics <|DETAILS|> color is black, battery life is 30 hours <|DESCRIPTION|>

The model sees this and “completes the sentence” by generating the description after <|DESCRIPTION|>.

GPT-2 requires special tokens to distinguish metadata fields from the target description

Key Insight: Without special tokens, GPT-2 gets confused; it doesn’t know where metadata ends and the description begins, leading to mixed-up information.

T5: The Translator

T5 treats this as a “translation” task: metadata → description. We just tell it what to do with a simple prefix.

Example:

generate product description: Title: Wireless Headphones | Brand: Sony | Category: Electronics | Details: color is black...

T5 uses a simple task prefix to frame the generation as a text-to-text problem

T5’s encoder-decoder architecture naturally separates the input (metadata) from the output (description), making it less prone to confusion.

Training

I used the Hugging Face Trainer API, which made life much easier.

Key Hyperparameters:

• Epochs: 3 (more would cause overfitting)
• Batch Size: 32
• Learning Rate: 3e-4 for T5, 5e-5 for GPT-2
• Optimizer: AdamW with weight decay
• Precision: Mixed Precision (FP16) to save memory and speed up training

(Note: While these models fit on a standard GPU, using FP16 was crucial for fitting the 355M parameter GPT-2 medium model.)

Training & Validation Loss

GPT-2 Training Loss: Steady decrease over epochs showing effective learning

GPT-2 Validation Loss: Monitoring for overfitting

T5 Training Loss: Consistent improvement throughout training

T5 Validation Loss: Well-generalized model performance

Both models show steady loss reduction. This is a clear sign they’re learning the task effectively!

Evaluation: How Do We Measure “Good”?

Judging AI-generated text is hard. You can’t just check if it matches word-for-word. Instead, we used multiple NLP metrics:

Summary of evaluation metrics used to compare GPT-2 and T5 performance

1. BLEU (Bilingual Evaluation Understudy)

• What it checks: Word overlap with reference text
• The Math:

  BLEU = BP × exp(Σ(wn × log(pn)))
  where pn = n-gram precision, BP = brevity penalty
  

• Why: Good for checking if key phrases are present

2. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• What it checks: Recall—did the model capture ALL the important information?
• The Math:

  ROUGE-N = (Count of matching n-grams) / (Count of n-grams in reference)
  ROUGE-L = LCS-based F-measure (Longest Common Subsequence)
  

• Why: Ensures we aren’t missing product specs

3. METEOR (Metric for Evaluation of Translation with Explicit ORdering)

• What it checks: Word matches, synonyms, and stemmed forms
• The Math:

  METEOR = (1 - Penalty) × Fmean
  where Fmean = harmonic mean of precision and recall
  Penalty accounts for word order via chunk fragmentation
  

• Why: More flexible than BLEU, as it recognizes “great” vs “excellent” as similar

4. BERTScore (The Modern Way)

• What it checks: Semantic similarity using BERT embeddings
• The Math:

  Precision = Σ max(cosine_sim(xi, yj)) / |x|
  Recall = Σ max(cosine_sim(xi, yj)) / |y|
  F1 = 2 × (P × R) / (P + R)
  

  (where x, y are token embeddings from BERT)
• Why: Closest to human judgment as it captures meaning, not just words

5. Length Ratio

• What it checks: Generated length / Reference length
• The Math:

  Length Ratio = |generated tokens| / |reference tokens|
  

• Why: Detects rambling (ratio > 1.5) or incomplete outputs (ratio < 0.5)

Results: T5 Takes the Crown 👑

T5 won decisively. It was:

• More accurate (higher BLEU, ROUGE, BERTScore)
• More concise (Length Ratio ~0.9 vs GPT-2’s ~2.35)
• Less likely to hallucinate

GPT-2 was creative but often rambled, generating descriptions more than twice as long as the reference. It would sometimes make up features the product didn’t have!

Output Examples

Here’s a side-by-side comparison of what each model generated for the same product:

Comparison of T5, GPT-2, and reference outputs for the same product

As you can see, T5 generates concise descriptions. While both models can occasionally hallucinate (invent details), T5 does this far less than GPT-2, which tends to be wordier and frequently adds features not present in the input.

Deployment: Putting It Into Production

A model is only useful if people can actually use it. I built a full-stack demo:

• Backend: FastAPI server that loads both fine-tuned models
• Frontend: React (Vite) app where users input product metadata and see outputs side-by-side
• Tunneling: ngrok to expose the local server (required because the backend API was running in a Google Colab instance, which sits behind a private firewall)

Users can compare GPT-2 vs T5 outputs in real-time!

Limitations & Future Work

Like any project, this had some limitations:

• Dataset Bias: Amazon data may not generalize to other e-commerce platforms
• Hallucination: GPT-2 occasionally invents features
• Computational Cost: Fine-tuning transformers requires GPUs
• Language: Only trained on English descriptions

Potential Improvements:

• Multi-lingual support
• Reinforcement learning from human feedback (RLHF)
• Combine both models (ensemble approach)
• Fine-tune on domain-specific data (fashion, electronics, etc.)

Wrap Up

Fine-tuning transformers isn’t just for big tech companies. With the right dataset (thanks McAuley Lab!) and smart preprocessing, T5 proved to be an incredibly reliable product copywriter.

Key Takeaways: ✅ T5’s encoder-decoder architecture is better for structured → text tasks
✅ Special tokens are critical for decoder-only models like GPT-2
✅ Multiple evaluation metrics give a fuller picture than any single metric
✅ Real-world deployment requires thoughtful UX (side-by-side comparison helps users trust the AI)

If you want to dive deeper, check out the McAuley Lab dataset and the official docs for GPT-2 and T5. You can also try out my fine-tuned models directly: GPT-2 and T5.