Research-First Coding Agents Beat Code-Only Ones

SkyPilot demonstrated a coding agent that reads academic papers and competing projects before touching code, applied to optimizing llama.cpp CPU inference. The results were concrete: 15% faster text generation on Intel Xeon and 5% on ARM Graviton3, achieved for roughly $29 in compute and API costs over three hours. Without the research phase, the agent produced negligible gains - it was optimizing for compute when the real bottleneck was memory bandwidth, something only the literature review surfaced. The five optimizations included softmax fusion, RMS norm fusion, and flash attention tile merging, drawn from studying FlashAttention papers and how CUDA and Metal backends already solved similar problems. As coding agents mature, the gap between agents that just write code and agents that read first is becoming one of the clearer performance differentiators.