How Robust Is LLM Economic Decision-Making? An Auction Benchmark
Abstract — How Robust Is LLM Economic Decision-Making? An Auction Benchmark
Economic robustness requires both appropriate adjustment to changing fundamentals and resistance to information that theory predicts should not affect decisions. Using a benchmark of 78 auction environments, we evaluate six large language models (LLMs) in nine configurations against equilibrium predictions for risk-neutral bidders. We find that strong performance is often specific to particular conditions and decisions. In first-price auctions, high bidding accuracy under uniformly distributed private values does not consistently carry over to normally distributed private values. These limits also emerge across connected decisions: entry or information-purchase frequencies close to theoretical predictions can coexist with inaccurate subsequent bids. Conversely, accurate bids among entrants can coexist with entry frequencies far from those predictions. In first-price auctions with costly entry, historical-price information is associated with bidding changes and generally higher entry-error rates than without this information, despite leaving theoretical predictions unchanged. Without an entry decision, most configurations’ bids change little across historical-price conditions. These findings support evaluating LLMs under the economic conditions and across the connected decisions relevant to their intended use, since isolated successes and aggregate scores can conceal substantial gaps in performance.