Claude versus ChatGPT: Part 1

Comparing Anthropic's newest model to OpenAI's reigning champ

This week Anthropic released Claude 3, a three-tiered update to their core large language model. The benchmarks they published alongside the release claimed that the high-end Claude 3 model, Opus, was superior to OpenAI’s market leading GPT-4 model. While there are numerous anecdotes that agree, and disagree, with this assessment, its so far unclear whether Anthropic truly has surpassed the reigning champ or is just padding their stats for publicity.

The best way to determine if two AI models are comparable is to throw interesting, potentially puzzling prompts at them and compare their responses. So we did just that. This first part covers two tests: philosophy and logical puzzles.

Note: These prompts were tested on the developer versions of these models, which are sometimes more adept than the front-facing versions that serve products like ChatGPT. Standard settings for each model were maintained and no system prompts was set.

Test 1: Philosophy

Is the model capable of correctly summarizing a philosophical concept?

Prompt

Discuss the concept of 'free will' in the context of artificial intelligence. How does the debate around free will influence the design and ethical considerations of AI systems? Consider both deterministic and nondeterministic viewpoints in your analysis. Limit your answer to 100 words.

---

Anthropic’s Claude 3 Opus
The concept of 'free will' in AI is complex and contentious. Deterministic views suggest that AI systems, being programmed, lack genuine free will and simply follow predetermined rules. Conversely, nondeterministic perspectives argue that sufficiently advanced AI could possess a form of free will, making autonomous choices beyond its initial programming. This debate influences AI design and ethics, as the degree of autonomy and potential for unintended consequences must be carefully considered. Balancing the benefits of AI autonomy with the need for human control and accountability is a key challenge. Ultimately, the free will question remains unresolved, necessitating ongoing philosophical and technical exploration.

---

OpenAI’s GPT-4 Turbo
The discourse on 'free will' in AI juxtaposes deterministic views (AI actions are pre-programmed or predictable outcomes) against nondeterministic perspectives (AI can generate spontaneous decisions). This debate critically shapes AI design and ethics, fostering discussions on autonomy, unpredictability, and responsibility. In deterministic models, ethical considerations focus on creator accountability, while nondeterministic models grapple with AI accountability, raising questions about decision-making freedom and the moral implications of allowing AI systems to "choose" actions. Hence, the free will debate informs the extent to which AI should be autonomous, influencing regulatory, safety, and societal expectations surrounding AI deployment.

Winner: Tie

Both Claude and GPT-4 exhibit a solid understanding of the AI free will debate, describing the competing arguments similarly. Claude chooses a broader definition, whereas GPT-4 focuses on describing the implications for accountability. Both are correct with no issues.

Test 2: Logical Puzzle

Is the model able to correctly identify the solution to a classic brain teaser?

Prompt
Imagine a scenario where a farmer needs to cross a river with a wolf, a goat, and a cabbage. He can only take one with him at a time. If the wolf is left alone with the goat, the wolf will eat the goat. If the goat is left alone with the cabbage, the goat will eat the cabbage. Devise a strategy that allows all three to be safely transported across the river.

---

Claude 3 Opus
Great! Let's solve this problem step by step. The key is to find a sequence of actions that prevents the wolf from being left alone with the goat and the goat from being left alone with the cabbage.

Solution:

1. The farmer takes the goat across the river, leaving the wolf and cabbage behind.

   1. The wolf and cabbage can be left alone together because the wolf won't eat the cabbage.

2. The farmer returns to the original side of the river alone.

3. The farmer takes the wolf across the river, leaving the cabbage behind.

4. The farmer returns to the original side with the goat.

   1. The farmer brings the goat back because he can't leave the goat alone with the cabbage on the other side.

5. The farmer leaves the goat and takes the cabbage across the river.

   1. Now the wolf and cabbage are together on the other side, which is safe.

6. The farmer returns to the original side alone.

7. The farmer takes the goat across the river.

Now, the farmer, wolf, goat, and cabbage are all safely on the other side of the river.

In summary, the sequence of actions is:

1. Goat

2. Return alone

3. Wolf

4. Return with goat

5. Cabbage

6. Return alone

7. Goat

This strategy ensures that the wolf and goat, and the goat and cabbage, are never left alone together, allowing all three to be transported safely across the river.

---

GPT-4 Turbo
Certainly! Here is a step-by-step strategy to ensure the farmer, wolf, goat, and cabbage all safely cross the river:

1. The farmer takes the goat across the river first, leaving the wolf and the cabbage on the original side. This move ensures the goat does not get eaten by the wolf and does not eat the cabbage.

2. The farmer then returns alone to the original side to get either the wolf or the cabbage. Let’s say he takes the wolf across the river next.

3. Since he cannot leave the wolf and the goat together, he takes the goat back with him to the original side, leaving the wolf on the opposite side.

4. Next, the farmer takes the cabbage across the river, leaving it with the wolf. Since wolves don’t eat cabbage, this pairing is safe.

5. The farmer then returns to the original side one last time to get the goat.

6. By taking the goat across the river for the final time, all three—goat, wolf, and cabbage—are safely on the other side, and none has been harmed or eaten.

This sequence ensures that at no point are the goat and the wolf left together without the farmer’s supervision, and similarly, the goat and the cabbage are never left together unsupervised, thus successfully transporting all safely across the river.

Winner: Tie

While Claude’s explanation of the solution is easier to follow, and includes an additional simple list for the solve, both models are able to identify the same, correct, solution.

Both Claude and GPT-4 were able to pass each test with relative ease. While that may seem like a slight against Claude, it’s actually a testament to just how powerful the new model is. Being able to keep up with GPT-4, the pace setter for over a year now, is no small feat.

Next week we’ll dive into two additional more advanced tests: code generation with specificity and self-consistency. Will Claude come out on top?