GPT-4 fails to solve coding problems it hasn’t been trained on

A guy has posted a series of tweets about his experiments with GPT-4 on Codeforces problems. He found that GPT-4 can solve 10 out of 10 problems from before 2021, but none of the recent problems. He suspects that this is due to data contamination, meaning that GPT-4 has seen some of the older problems in its training data, but not the newer ones. He also shows some examples of how he tested GPT-4 and the solutions it generated.

This is an interesting finding, as it suggests that GPT-4’s performance on coding tasks is heavily dependent on the quality and freshness of its training data. It also raises questions about how much GPT-4 actually understands the logic and syntax of programming languages, and how well it can generalize to new and unseen problems. What do you think about this? Do you think GPT-4 can ever become a competent coder, or will it always be limited by data contamination?

Here is the link to the tweet thread: https://twitter.com/cHHillee/status/1635790330854526981