AutoIndex with Sam O'Nuallain - Weaviate Podcast #143!

Sam O'Nuallain joins the Weaviate Podcast to discuss AutoIndex, research from UMass Amherst and Databricks on learning representation programs for retrieval. Instead of tuning the retriever or re-ranker, AutoIndex asks how the data itself should be represented: a two-agent system, an analysis agent and a code agent, writes and refines Python programs that chunk, enrich, and reorganize a corpus to optimize downstream metrics like Recall and nDCG. The conversation opens with why indexing is such a natural target for code optimization. Frontier LLMs are exceptional at writing code, a representat