From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix
A production-derived post-training recipe consolidated more than 200 internal apps onto one self-hosted model by training separate experts for distinct quality gaps, then merging them.
The team analyzed traffic from **200+ internal applications**, trained separate GRPO experts for instruction following, function calling, and internal task distribution, then merged them with **two-stage SLERP** to avoid cross-domain reward interference.
For a shared internal agent model, derive eval slices from production errors and isolate objectives whose rewards conflict. Their non-reasoning recipe beat a roughly **7× larger baseline** on the in-house Arena, instruction following, and function calling.
The team analyzed traffic from **200+ internal applications**, trained separate GRPO experts for instruction following, function calling, and internal task distribution, then merged them with **two-stage SLERP** to avoid cross-domain reward interference. For a shared internal agent model, derive eval slices from production errors and isolate objectives whose rewards conflict. Their non-reasoning recipe beat a roughly **7× larger baseline** on the in-house Arena, instruction following, and function calling. These are internal benchmarks using deterministic verifiers or calibrated model judges, so external comparability is limited. The resulting model handled **116M requests per month**, but the material gives neither model identity nor an exact serving-cost figure.
This shifts model selection from choosing among general models to post-training one self-hosted model around the measured corporate request mix. It confirms production traffic and error-derived evaluations as useful design inputs, while identifying conflicting rewards as a reason to train specialists before merging them. The result remains an internal validation: absent model identity, external benchmarks, and serving costs, it does not establish broader efficiency.