GraphRAG Benchmark Results Shift Dramatically Depending on Evaluation Method Used

A technical analysis highlights how GraphRAG performance figures can appear to win or lose depending solely on the evaluation method applied. When judged by a large language model without gold-standard answers, GraphRAG community summaries showed comprehensiveness win rates of 72–83 percent; however, when scored against ground-truth answers using ROUGE-2, plain RAG outperformed GraphRAG across multiple datasets. Research also found that position bias in LLM judges can independently reverse their preference, raising further questions about model-based evaluations. Cost comparisons across graph-based retrieval methods vary by over two orders of magnitude, meaning the assumption that graph approaches are uniformly expensive is too simplistic. The core takeaway is that a benchmark result reflects the specific question being measured, and conflating model-preference scores with ground-truth accuracy scores can mislead practical decision-making.