Developing a Framework to Evaluate Credibility Tracking in Large Language Models

Author(s):

Avvai Chandrasekaran, Erin Zaroukian, Justine Rawal, Mark Mittrick, and Adrienne Raglin

Abstract:

Large language models (LLMs) are increasingly being used in human problem-solving tasks such as standardized tests, debugging code, and providing customer support. However, little is known about the performance of these models when reasoning over longitudinal data and in detecting changes concerning the credibility of an information source over time. Recent work has explored this gap by asking LLMs to make predictions based on different patterns of historical data, but it was limited in the prompt variations used. Because LLMs are known to be very sensitive to seemingly minor changes in their prompts, the work presented in this report addresses these shortcomings by employing a variety of modifications to how the model is asked to make a prediction based on the historical data and by changing the historical data itself. A method was developed to evaluate the results of these modifications, finding that the LLM’s ability to reason over longitudinal data and assess changes in credibility is improved by including the linguistic hedges “likely” or “probably” in the template for the model’s prediction, likely because this helps avoid hyperconservatism. While other manipulations inhibited performance, adding “most likely” was the strongest inhibitor, and we hypothesize that adding the superlative “most” exacerbated hyperconservatism. Future work will explore further manipulations and improvements in methodology.