MachineLearning
RWKV: Recurrent Architecture with Constant State Size for Parallel Inference
{{RWKV}} is a recurrent language-model architecture whose internal state has a fixed size independent of context length, making large-batch inference and parallel-perturbation training dramatically cheaper than for {{transformer}}-based models with their growing {{KV-cache}}.
Evolution Strategies for LLM Fine-Tuning: A Revival of a Pre-Deep-Learning Optimizer
Two 2025 papers revive {{evolution strategies}} (ES) as a credible alternative to {{reinforcement learning}} for fine-tuning large language models, exploiting the fact that RL fine-tuning rewards are already scalar at the sequence level — the regime where ES is naturally competitive.
EGGROLL: Low-Rank Perturbations Make Evolution Strategies 100x Faster at Hyperscale
{{EGGROLL}} (Evolution Guided GeneRal Optimisation via Low-rank Learning), from an Oxford/MILA/NVIDIA collaboration in November 2025, structures each {{evolution strategies}} perturbation as a low-rank matrix so that thousands of perturbations can be computed in a single batched forward pass — yielding a claimed 100-fold training-speed increase over naive ES at billion-parameter scale.
Evolution Strategies at Scale (Cognizant 2025): First Full-Parameter ES on Billion-Parameter LLMs
A September 2025 paper from {{Cognizant AI Lab}} demonstrated full-parameter {{evolution strategies}} fine-tuning of billion-parameter {{LLMs}} using a population of just 30 perturbations, breaking the prior assumption that ES could not scale past roughly a million parameters.