· · ·
DeepSeek's mixture-of-experts large-language model with 671B total and 37B active parameters per token. The report details training efficiency innovations including FP8 mixed-precision training and a Multi-head Latent Attention architecture.
Alibaba's Qwen team release covering open-weights models from 0.5B to 72B parameters, with extensions for coding (Qwen2.5-Coder) and mathematics (Qwen2.5-Math). The report describes pre-training data curation and post-training techniques.
Meta's open-weights large-language-model release covering models from 8B to 405B parameters. The technical report runs to many hundreds of pages and credits hundreds of contributors across pre-training, post-training, safety, evaluation, and infrastructure work.
Apple's foundation-model report introducing AFM-on-device and AFM-server, the models powering Apple Intelligence. The report covers pre-training, post-training, and on-device deployment, with attention to privacy-preserving techniques.
Google DeepMind's open-weights model family at 2B, 9B, and 27B parameters, distilled from the Gemini line. The report covers architectural choices including local–global attention interleaving and logit soft-capping.
Google DeepMind's multimodal long-context model family, with a context window of up to one million tokens. The technical report describes the architecture, training, and evaluation across text, code, audio, image, and video.
Mistral's sparse mixture-of-experts model with eight expert blocks per layer and two active per token. The report introduced an open-weights MoE architecture with strong performance at low active-parameter cost.