LinkedIn has detailed the infrastructure behind its AI-powered job search system, revealing how multi-teacher knowledge distillation and a series of training optimizations helped the company achieve an approximately eightfold improvement in training speed. The approach transfers knowledge from larger artificial intelligence models into a compact 0.6-billion-parameter ranking model designed for high-volume search workloads.
LinkedIn Uses Multi-Teacher Distillation for AI Job Search
Training a small language model to improve search relevance and engagement metrics, including clicks and job applications, can require querying one or more larger teacher models for every training example.
That process can become a significant bottleneck at LinkedIn’s scale, where ranking infrastructure must accommodate hundreds of thousands of queries per second. The challenge is particularly relevant as search and recommendation platforms move beyond traditional keyword-based systems towards unified ranking models supervised by large language models.
LinkedIn developed a multi-teacher distillation framework using SGLang, an open-source serving engine for large language models. The framework can load and serve teacher models of different sizes while managing tensor-parallel and data-parallel configurations.
Online and Offline Distillation Reduce Training Bottlenecks
LinkedIn calls its first approach Online Multi-teacher Distillation. During training, an asynchronous client queries teacher models, processes their outputs and incorporates the results into the student’s distillation losses.
Deploying local teacher replicas across multiple computing nodes accelerated the distillation process by three times while keeping latency low enough for rapid experimentation.
LinkedIn also introduced Offline Multi-teacher Distillation to reduce serving overhead and avoid repeatedly calculating the same teacher outputs. Under this approach, teacher results are generated in advance and stored on HDFS or NFS before being used directly during training.
The distinction gives engineering teams flexibility. Online distillation can be useful during early experimentation, when teacher models are still changing, while offline caching becomes more efficient once the teachers stabilize and training volumes increase.
Multiple Optimizations Deliver Eightfold Training Speed Increase
The online and offline distillation architecture forms only part of LinkedIn’s broader training strategy.
LinkedIn adopted LiGer to reduce memory consumption and support batch sizes up to twice as large. Multi-node training provided an additional speed improvement of as much as 3.5 times, while FSDP2 delivered a further 20 per cent gain.
The company also deployed multi-node clusters using NVIDIA H200 GPUs, providing up to another 30 per cent improvement.
LinkedIn evaluated FP8 mixed-precision training but found that it offered no advantage for models smaller than eight billion parameters. For those models, the overhead associated with data-type casting outweighed potential computational savings.
Combined, these improvements account for the roughly eightfold acceleration in training reported by LinkedIn.
0.6B Student Model Improves Job Search Ranking
LinkedIn’s research indicates that its 0.6B-parameter student model produced substantial improvements in job search quality.
The student model was distilled from two larger systems: an 8B-parameter relevance oracle and a 1.7B-parameter engagement teacher.
The resulting model increased NDCG@10, a widely used measure of ranking quality, by 24.48 per cent. The score rose from 0.7583 to 0.9432.
LinkedIn also optimized inference through structured pruning and context compression. Together, those techniques increased ranking throughput from approximately 290 items per second per GPU to more than 2,000.
System Already Powers Natural-Language Job Search
The technology is operating in production and powers LinkedIn’s natural-language job search experience for users in the United States.
Rather than relying on a proprietary serving platform, LinkedIn built the system around SGLang, which it had previously used for ranking workloads.
The architecture is intended to provide ranking quality approaching that of larger cross-encoder models while maintaining the latency required for real-time search. It also reduces the cost that would result from running frontier-scale LLM inference for every user request.
Pinterest and Other AI Developers Explore Similar Techniques
LinkedIn is not alone in applying distillation to large-scale recommendation systems.
Pinterest described a related strategy in June 2026, explaining how multi-node training supported larger teacher models whose knowledge could then be distilled into smaller student models for Homefeed and Related Pins ranking.
Pinterest reported that the approach shortened experimentation cycles that previously took weeks. Its work focuses heavily on scaling the training framework and migrating to Distributed Checkpoint to support multi-node teacher training.
LinkedIn’s approach differs by introducing a custom SGLang-based framework that allows teacher models to be queried directly while the student model is being trained.
Multi-teacher distillation is also being explored in larger frontier AI systems. Recent examples include NVIDIA’s Nemotron 3 Ultra, MiMo-V2-Flash and DeepSeek-V4, which use larger pools of specialized teacher models for dense, token-level supervision.
Those configurations can involve 10 or more teachers, making them considerably more complex than industrial ranking systems such as LinkedIn’s and Pinterest’s, where a smaller number of task-specific teachers may be sufficient.
LinkedIn Shows How AI Ranking Can Scale Efficiently
LinkedIn’s results demonstrate that faster AI ranking training does not depend on a single optimization. Instead, gains from teacher caching, LiGer, multi-node training, FSDP2 and newer GPU hardware can accumulate to produce a substantial overall improvement.
For large search and recommendation platforms, the approach provides a practical route to using knowledge from powerful language models without requiring costly frontier-model inference for every search request. LinkedIn’s production deployment also shows how compact AI models can deliver improved ranking quality while meeting the demanding throughput and latency requirements of large-scale online services.
Francesco Petrarca is a writer for ThePacket.ca, covering news, politics, business, technology, sports, entertainment, and lifestyle. He is committed to clear and reliable reporting, providing readers with useful information, timely updates, and stories that highlight important developments and current affairs.