AUTHORS: Julia Tokarska, Richard Sloggett, Graham Foxon
OBJECTIVES
To evaluate if ChatGPT is a reliable tool for insight extraction in healthcare pricing and reimbursement (P&R) research by replicating the development process of an example 2023 ISPOR research poster using targeted ChatGPT prompts. The study aims to compare the effectiveness and accuracy of ChatGPT against human researchers (HR) in poster research and development.
METHODS
A previous 2023 ISPOR poster on Evaluation and Reimbursement of Digital Therapeutics was selected as a pilot to determine if ChatGPT can replicate the research findings and provide updated insights compared to HR analysis. The process for ChatGPT replication involved identifying key aspects of poster development, creating and iterating specific prompts using ChatGPT, and comparing the results with those obtained by HR.
Analysis focused on ChatGPT’s ability to accurately synthesize information, develop structured research frameworks, derive insights and convey conclusions.
RESULTS
ChatGPT’s performance varied based on the task and nature of the source information. It showed high effectiveness when provided with a single, clear source of information to extract relevant insights. Challenges were noted in tasks requiring browsing large databases and communicating information accurately within low word limits. The task of browsing the web to identify relevant sources and provide references yielded varying results. Generally, better results were achieved when providing more context and using a chain of thought methodology with HR review to ensure accurate, complete, and contextually appropriate outputs.
CONCLUSION
ChatGPT shows significant potential for practical applications in P&R research. While its capabilities are not yet fully consistent, they are expected to improve over time. Its effectiveness relies heavily on the creation of precise prompts and thorough review by HR; fact-checking and refining prompts are essential for achieving reliable results. For optimal use, it is key to select tasks where the time invested in developing prompts is justified by the value of the results obtained.
Read our full research below (available post-conference).
Plus, read our other research abstracts for ISPOR Europe 2024 here.