Can AI forecast the presidency?

ChatGPT and Claude Predict the 2024 Presidential Election

Foreword

Unless you are completely unplugged from the grid, it’s impossible to miss all the speculation regarding the US Presidential Election. Both leading candidates are facing scrutiny due to their age, particularly Joe Biden following the recent debate on June 27th. The performance was bad enough that there is significant talk of replacing the incumbent president Joe Biden at the top of the Democratic ticket. This primarily seems to be because there is a loss in confidence that he can win due to the perception (or reality) of his cognitive state. I don’t want to discuss my view on politics here, partly because I don’t think I have anything interesting to add, but also because there are enough places to find punditry. However, given the media frenzy, I became curious about how AI models would predict the 2024 Presidential Election outcome. While I don’t have a fancy statistical model like Nate Silver, I have an alternative method that may prove just as effective. This post will focus on utilizing large language models ChatGPT and Claude for election forecasting based on the ‘Keys to the White House’ framework by Allan Lichtman.

Overview of Allan Lichtman’s Model and ‘Keys to the White House’.

Allan Lichtman is a professor and historian famous for his track record in successfully predicting many US presidential elections, including 2016 and 2020. How did he predict these? Lichtman developed a checklist, 'Keys to the White House', with Russian seismologist Vladimir Keilis-Borok. This checklist is based on thirteen true/false statements reflecting political, economic, and social conditions. These statements are directly asked in reference to the incumbent candidate, which in this case would be Joe Biden. True statements favor the election of the incumbent party candidate while false statements disfavor the incumbent candidate. It has been historically found that if six or more items on the checklist are false the incumbent is predicted to lose.

What are the 13 keys?

1. Incumbent Party Mandate: The incumbent party gains more seats in the House of Representatives in the midterm elections than it did in the previous midterm elections.

2. Nomination Contest: There is no serious contest for the incumbent party nomination.

3. Incumbency: The incumbent-party candidate is the sitting president.

4. Third Party: There is no significant third-party or independent campaign.

5. Short-term Economy: The economy is not in recession during the election campaign.

6. Long-term Economy: Real per capita economic growth during the term equals or exceeds mean growth during the previous two terms.

7. Policy Change: The incumbent administration affects major changes in national policy.

8. Social Unrest: There is no sustained social unrest during the term.

9. Scandal: The incumbent administration is untainted by major scandal.

10. Foreign/Military Failure: The incumbent administration suffers no major failure in foreign or military affairs.

11. Foreign/Military Success: The incumbent administration achieves a major success in foreign or military affairs.

12. Incumbent Charisma: The incumbent-party candidate is charismatic or a national hero.

13. Challenger Charisma: The challenging-party candidate is not charismatic or a national hero.

How successful has this model been historically?

Lichtman's model was first developed prior to the 1984 election. It was retrospectively validated on elections between 1860 and 1980, of which, it accurately predicts all but 2 presidential elections. Of the elections it has been used as a forecasting tool for (elections between 1984 and 2020), it has been correct in all but 1 election, the 2000 election between Gore and Bush (Table 1).

Table 1. Predicted outcomes of United States presidential elections by Allan Lichtman with ‘Keys to the White House’ (Wiki).

It’s an interesting set of keys. While many checklist questions are straightforward and factual, e.g., ‘Incumbent Party Mandate’, others are highly subjective, e.g., ‘Incumbent Charisma’, or altogether somewhere in between ‘short-term economy’. These AI large-language models are trained and have access to enormous amounts of data from the internet, much of which comes from the American people. Will this aggregate insight into everyday Americans give LLMs like ChatGPT and Claude an advantage in answering these subjective questions?

So, let’s get to the fun part: who do AI models predict will win the 2024 presidential race based on the ‘Keys to the white house?’

ChatGPT predicts Biden will lose the Presidential Race

I began by prompting ChatGPT with “What are Allan Lichtman's 13 Keys to the White House?” I then followed this up with the request: please create a table where you assess these Keys for the Presidential Race of Joe Biden vs Donald Trump in 2024. Present this a table where the first column is a given Key, the second column is a true or false statement, and the third column is a justification:

As you can see, ChatGPT did a great job executing the task. Interestingly, it believes Joe Biden will lose the White House. Again, this is based on the rule that if six or more of these checklist items are false, the incumbent will lose. While they are interesting justifications, I was a bit surprised by a few of the answers. I decided to probe ChatGPT further to understand some of the fundamental assumptions and decision-making.

My probing:

ChatGPT revising decisions:

Based on the probing, ChatGPT changed all 3 answers that I dug deeper into. It’s unclear if this is an indicator that ChatGPT is easily persuaded and or is eager to agree and please with the prompter, or if it truly changed its mind organically with additional prompting. Either way, these updated numbers cut even worse against the incumbent, resulting in 8 False items on the checklist.

Claude3.5 also predicts Biden will lose the 2024 race

Again, I began by prompting with “What are Allan Lichtman's 13 Keys to the White House?” I then followed with the request: please create a table where you assess these Keys for the Presidential Race of Joe Biden vs Donald Trump in 2024. Present this a table where the first column is a given Key, the second column is a true or false statement, and the third column is a justification.

Claude 3.5 similarly predicts that Joe Biden will lose the presidential election with 6 False statements.

Let’s try Gemini:

Unsurprisingly, Gemini remains miles behind other LLMs. It could not even attempt to perform the task.

What are the takeaways?

This started purely out of curiosity. I wanted to know how these AI LLMs would predict the 2024 presidential race. This experiment found ChatGPT and Claude both believe the incumbent Joe Biden will lose to Donald Trump based on the ‘Keys to the White House’ checklist. There is much more work that could be done to back-validate these AI’s ability to accurately forecast with keys checklist (some of this is started in the Appendix). Moreover, some of the answers to the subjective questions could change as we get closer to November. However, I think it’s a valuable experiment to document these predictions so we can assess the AI’s ability to forecast. It will be interesting to find out what happens next.

Assuming Biden continues on as the Democrat nominee, I will revisit this article as the election comes and passes. What do you think, did ChatGPT and Claude do a good job forecasting the race, or do you disagree with their interpretations?

Appendix:

ChatGPT back-validation

Table 2. ChatGPT back-validation 1984-2020.

ChatGPT forecasted 10 of 10 elections correctly in the back-validation experiment.

Claude back-validation

Table 3. Claude3.5 back-validation 1984-2020.

Claude forecasted 9 of 10 elections correctly in the back-validation experiment.

Support:

These newsletters take a significant amount of effort to put together and are totally for the benefit of the reader. If you find these explorations valuable, there are multiple ways to show your support:

• Engage: Like or comment on posts to join the conversation.

• Subscribe: Never miss an update by subscribing to the Substack.

Subscribe

• Share: Help spread the word by sharing posts with friends directly or on social media.

References:

https://en.wikipedia.org/wiki/The_Keys_to_the_White_House

Lichtman, A., 2020. Predicting the next president: The keys to the White House. Rowman & Littlefield.