Creativity and Diversity in AI-Supported Brainstorming

Abstract: This research examines how well large language models (LLMs) generate new product ideas, focusing on new products for college students priced under $50. Through a series of studies, we identify key strengths and weaknesses of using LLMs for product innovation. Our first study shows that LLM-generated product ideas have higher average quality than human ideas (based on purchase intent) and are 7 times more likely to rank in the top 10%. Our second study demonstrates that this AI-induced creativity boost is not explained by the LLM’s more persuasive pitching skills. Our third and fourth studies identify an interesting weakness of LLM-supported brainstorming, showing that AI-generated ideas are less novel (at the idea level) and less diverse (at the set level). In our fifth study, we investigate the generalizability of these findings by analyzing prior LLM-based creativity studies, finding consistently lower idea diversity across all of them. Our sixth and seventh studies investigate the efficacy of various techniques to mitigate this diversity loss. Specifically, we compare different LLMs (different vendors and different versions) and find that the more recent models are capable of generating more diverse ideas though still falling short of achieving human-level diversity. In addition, we demonstrate the effectiveness of three techniques that can be used to increase idea diversity almost to the level of human idea generation: pooling ideas across vendors, prompt engineering (specifically, Chain of Thought prompting) and creative agents that broadly explore the solution landscape with the intent of restoring diversity. We conclude our work by presenting a set of actionable recommendations targeted to managers of innovation that want to identify better new product ideas with the help of LLMs.