I agree. This step-by-step vs intuition based is an interesting observation. Yes, single-pass logic (sometimes even a bit deeper one) mostly works.

Still AI is pretty stubborn at times when demanded to do something, which is off-road. (though, this is also what people do: people think in the familiar terms and it's easier to convert terms back and force than to grasp somebody's else conceptual framework. Even very intelligent people stick to their frameworks.)

The question still remains what is the way logical reasoning appears in the LLM? The key to that are kinds of mistakes chatgpt does.

It is interesting, that sometimes mistakes feel like something could be more logical (in lesser represented languages, a word can be misspelled to be more according to the language "spirit" than letter). Probably the same goes with logic.

chatgpt4 answers like this to the above two paragraphs

<<It seems like you're interested in how logical reasoning manifests in a Large Language Model (LLM) like ChatGPT, and how this relates to the kinds of errors it makes, including those related to language and logic.

LLMs, including ChatGPT, generate text based on patterns learned from a vast dataset of human language. This learning process involves statistical associations and does not inherently understand logic or the meaning of the text in the same way humans do. The logical reasoning exhibited by ChatGPT is a byproduct of this pattern recognition and not a result of internal logical understanding.

Regarding mistakes, there are a few key points to consider:

Pattern-Based Errors: LLMs can make errors because they rely on statistical patterns in the data they were trained on. If a certain concept or word is underrepresented in the training data, the model might struggle to use it correctly.

Context and Nuance: LLMs sometimes miss the nuances of context or the specific logical flow of an argument. They might generate text that is grammatically correct and sounds plausible but is logically flawed or factually incorrect.

Language Specifics: In the case of less represented languages, the model might make errors that reflect a more general understanding of the language rather than its specific nuances. Interestingly, as you mentioned, these errors might sometimes align more with the "spirit" of the language rather than strict grammatical rules, reflecting the model's exposure to various forms of language, including colloquial or non-standard variations.

Logical Reasoning: The logical reasoning of an LLM is limited to what it has seen in its training data. It does not reason in the human sense but generates responses based on patterns. This can lead to errors where the model might generate text that seems logically consistent but is actually incorrect or based on flawed reasoning.

In summary, the logical reasoning in an LLM like ChatGPT is a complex interplay of pattern recognition, statistical associations, and the limitations inherent in a system that learns from pre-existing text without an innate understanding of logic or meaning. The kinds of mistakes it makes can provide insights into both its capabilities and limitations.>>