What Is Transfer Learning? And Why Fine-Tuning Is Rarely the Answer

Transfer learning means reusing what a model has already learned on one task in order to do a different, related task. You start from a system that has absorbed a great deal of general structure, and you adapt it, instead of training something from zero. Every chat assistant you use is the product of that process several times over.

The term shows up in two conversations that have less in common than they look. In research it describes how large models get built at all. In an office it usually arrives as a proposal: we should fine-tune a model on our own data. This entry covers both, and it is honest about the fact that the second one is a narrower and less available option in 2026 than most guides still assume.

What it means in practice

The classic version comes from image recognition. Train a network on millions of ordinary photographs and its lower layers end up encoding things that are useful everywhere: edges, corners, textures, repeated shapes. Swap the last part of the network for a new one, show it a comparatively small set of labeled X-rays, and you get a working detector without the millions of X-rays that training from scratch would have demanded, because the general knowledge has been inherited and only the task-specific piece had to be built.

Language models complicate that picture. The transfer step is the pre-training itself, and adapting the finished model afterwards does not usually mean freezing part of it and replacing the end. Full fine-tuning updates every weight in the network. The cheaper method that made adaptation ordinary is LoRA, which freezes the pre-trained weights and trains a small set of extra parameters injected into the layers, so a customized version can be kept as a small add-on instead of a second full copy of the model.

Both cases share the same economics, in that the expensive part has already been paid for by whoever did the pre-training.

What fine-tuning can teach, and what it cannot

This is where the office conversation usually goes wrong. OpenAI’s documentation lists what supervised fine-tuning is best for: classification, nuanced translation, generating content in a specific format, and correcting instruction-following failures. Those are questions of shape and behavior. The scale is modest as well. OpenAI puts the minimum at ten examples, reports improvements in the range of fifty to a hundred, and suggests starting with fifty well-crafted demonstrations, while noting that the right number varies greatly by use case. Nobody there is describing a company archive being poured into a model.

The research points the same way. In a controlled study of closed-book question answering, Gekhman and colleagues found that language models “struggle to acquire new factual knowledge through fine-tuning”, because examples carrying new facts are learned much more slowly than examples the model already agrees with. The uncomfortable half of the result is what happens once those examples do get learned: they “linearly increase the model’s tendency to hallucinate”. The authors say their findings support the view that models mostly acquire facts during pre-training, and that fine-tuning teaches them to use those facts more efficiently. It is one study, on one model family and one dataset, and the authors flag that limitation themselves. It is still the clearest experimental result available on the question.

One qualification, because the picture is not uniform. OpenAI’s reinforcement fine-tuning aims at harder targets, and its own examples include medical diagnosis against diagnostic guidelines and finding relevant passages in case law. The same page is explicit that this method amplifies ability the model already has: your model “must have some success at the desired task”, and “if a model has a 0% success rate at a given task, you cannot bootstrap to higher performance levels through RFT”. Sharpening judgment against expert grading is a real capability. Loading documents into a model’s memory is not what is happening.

The routes are moving

Search for advice on fine-tuning and almost all of it assumes you can go and do it on the platform you already use. Two of the most-read providers have published changes to that.

It would be wrong to read that as the door closing. Some of the same OpenAI models remain fine-tunable through Microsoft’s Azure AI Foundry, where the GPT-4.1 family is listed as generally available for both supervised fine-tuning and preference optimization. Open-weight models are the second route, and the cheaper one: Llama, Qwen, Mistral and similar families can be adapted with LoRA on hardware a small team can rent by the hour. The first-party, self-service path at the two biggest consumer brands is what is disappearing. The technique is not.

OpenAI’s wording carries a consequence that outlasts the deadlines. A fine-tuned model lives and dies with the base model it was built on, so when that base model is retired the customized version goes with it, on the provider’s schedule and not yours. That is worth pricing in before a product depends on a hosted fine-tune.

If you want a model to work with your documents

In my experience, people who propose fine-tuning are usually describing a different goal: they want the system to know their material. Three options, from least to most effort.

1. Put the material in front of the model. Custom instructions, a saved project, or attaching the files to the conversation. Unglamorous, and it covers a good deal of ordinary use, because context windows have grown to the point where a policy handbook simply fits.
2. Retrieval. The documents stay in a searchable store and the relevant passages get pulled in with each question. Corrections mean editing a document instead of retraining anything, and answers can carry a reference back to the passage used. Do check that reference. When Liu, Zhang and Liang audited four generative search engines by hand, they found that only 74.5 percent of citations supported the statement they were attached to.
3. Training. Worth the effort for a consistent output format or a narrow classification job at volume, where you can supply examples of what a good answer looks like.

What a fine-tuned model brings with it

A fine-tuned model is the original, nudged. It carries the base model’s phrasing habits, its gaps, and whatever skew was present in the data it first learned from. Training can also cost you abilities that were there before, which is why anyone doing this seriously tests the model on tasks outside the fine-tuning set, both before and after.

That has a practical use when you are buying rather than building. If a supplier describes their product as a model trained on your industry, ask which model it started from. Most of what you will experience comes from there.