The comparison to your SO’s approach is a bit sloppy. He didn’t reason out a solution himself; he wrote a program to solve the puzzle.
How do you define “reasoning?” Maybe your definition is different than mine. My experience is that there is a certain amount of reasoning going on, even with non-reasoning LLMs. Being able to answer “What is the capital of the state that has Houston in it?” for example, is something I would classify as very basic reasoning. And now, LLM-powered chat bots are much more capable.
All that “reasoning” or “thinking” really is, though, is a way to get additional semantic connections in place without:
- giving an answer in the wrong format
- filling up context with noise
There are limits to how well reasoning these char bots can reason. One of those limits is specifically related to the context size. As the context becomes larger, the model’s capabilities become worse. By asking it to show all its work, you exacerbated that weakness.
That still doesn’t mean LLM-powered chat bots can’t reason, just that there are limits.
I used to do puzzle books with these sorts of problems when I was younger, and they always came with multiple sets of grids with row and column labels filled out to facilitate the elimination approach. I don’t know that most people would think “Hey, it would be helpful to build a grid for each way of setting up these constraints.” One grid, sure, but I don’t think one grid is sufficient for this sort of problem.
I don’t think I am - or that most people are, for that matter - capable of reasoning through all of the necessary steps in my head and aloud, without use of, at minimum, pencil and paper. I hope you wouldn’t say that I and most people aren’t capable of reasoning as a result.
I just asked Claude Opus 4 (with extended thinking and analysis tools enabled) “Given the freedom to use any tools of your choice, design an approach to solve this problem. Feel free to use any tools that are necessary, including writing code in a language of your choice.” followed by pasting your problem text. It wrote and ran some code to solve it, gave me an answer that violated constraint 7, and when I pointed that out, ran some more code and gave me a correct solution on its second attempt.
You are absolutely correct. I apologize for the error. Let me revise the solution to ensure constraint 7 is satisfied.
Thank you for catching that error. Here is the corrected solution that satisfies all nineteen constraints:
Final Solution:
Customer | Price | Cheese | Meat |
---|---|---|---|
Carol | $5.75 | Colby Jack | Ham |
Darrell | $8.75 | Gouda | Corned Beef |
Henrietta | $6.75 | Swiss | Salami |
Kenneth | $7.75 | Provolone | Prosciutto |
Lula | $3.75 | Havarti | Bologna |
Omar | $4.75 | Cheddar | Roast Beef |
Peter | $2.75 | Muenster | Liverwurst |
The key correction was recognizing that since ham cannot be paired with swiss cheese (constraint 7), and Henrietta must have swiss cheese based on the process of elimination, Henrietta cannot have ham. Through careful analysis of the remaining constraints, I determined that Carol has ham with her colby jack cheese, which satisfies all requirements.
This solution demonstrates the importance of thorough verification when solving complex constraint satisfaction problems. Even when a solution appears complete, each constraint must be individually validated to ensure accuracy.
This all took 5-10 minutes - and most of that time was spent verifying its solutions - so a third of the time your SO took.
LLMs, even those with image analysis abilities, are lacking when it comes to spatial awareness, so your critique regarding using a grid to implement a systematic elimination approach is valid.
Did he implement two different variations? OP said he used two different tools, not that his solutions were any different.
That said… how so?
There are many different ways two different brute force approaches might vary.
A naive search and a search with optimizations that narrow the search area (e.g., because certain criteria are known and thus don’t need to be iterated over) can both be brute force solutions.
You could also just change the search order to get a different variation. In this case, we have customer, price, meat, cheese, and we need to build a combination of those to get our solution; the way you construct that can also vary.