Synopsis
A general-purpose language model can produce fluent answers while still missing the needs of a specific application. This talk examines how to adapt a model, what data each approach needs, and how to evaluate whether the changes help users.
The material combines research and deployment experience to explain parameter-efficient fine-tuning (PEFT), LoRA, quantization and preference optimization. The focus is the intuition and practical tradeoffs behind these techniques, with an evaluation example drawn from RAG.
Practical takeaways from the talk
These notes summarize the accompanying slides. The model examples and implementation choices in the recording belong to the original talk; the explanations below focus on the methods and the questions to evaluate.
Start with a dataset and metrics that reflect the application
The evaluation section argues for a curated dataset specific to the use case. Academic benchmark scores do not necessarily reflect the value users perceive in a product. Define what a useful answer looks like for the task, then use that evaluation to compare model and training choices. The same examples remain useful as candidate models change. See the evaluation principles in the slides.
LoRA reduces the number of parameters that need training
Full fine-tuning updates the model's parameters, with substantial compute, memory and model management costs. LoRA keeps the pretrained weights frozen and learns an update represented by two smaller matrices. Their rank is a hyperparameter: it controls the size of the adaptation rather than changing every weight independently.
This is the core idea behind the talk's discussion of parameter-efficient fine-tuning. It reduces the trainable parameter count and produces adapters that can be managed separately or merged into the weights after training. The right settings still need evaluation for the task. See the frozen weights and low-rank update.
Quantization trades numerical precision for a smaller memory footprint
Quantization concerns how model parameters are represented. The talk distinguishes post-training quantization from approaches that account for quantization during training. These choices have different requirements for calibration data, access to the training pipeline and training work.
A smaller representation can make a model easier to run on constrained hardware, but the quality tradeoff must be measured. Treat a quantization format as a configuration to test on the application, rather than assuming the smallest file or a particular bit width is always best. See the comparison of quantization approaches.
Match the training method to the data you can provide
Supervised fine-tuning uses examples of instructions and the responses the model should produce. Preference optimization instead needs comparisons: a prompt, a chosen response and a rejected response. These datasets express different kinds of supervision.
The talk contrasts a reinforcement learning pipeline using a reward model with Direct Preference Optimization (DPO), which learns from preference pairs without that separate reinforcement learning stage. Choosing a method therefore also means checking whether the required examples or comparisons are available. See the preference dataset example and the DPO explanation.
For RAG, separate retrieval and generation errors
The error-analysis example separates three questions when assessing a RAG answer:
- Context relevance: did retrieval supply information relevant to the user's query?
- Groundedness: is the generated answer supported by that retrieved information?
- Answer relevance: does the answer address what the user asked?
Looking at these dimensions separately helps identify where the system failed. Record good and bad generations with their prompts and answers so that error analysis has concrete examples. When users choose between answers, retaining the chosen and rejected responses can also support preference-based training. See the RAG error-analysis diagram.
The related GenAI in production talk extends this discussion to RAG product choices, model selection and costs. For hardware dependencies and the practical constraints around model access, see the Centrale Nantes course on LLM sovereignty and supply chains.
Watch the replay
Full recording of the talk at DataCraft Paris, exploring LLM fine-tuning approaches and evaluation frameworks.
Audience
Machine learning engineers, data scientist, research engineers, applied AI scientists