What we really mean by the word "think"
You write: write me an ad line for my mobile phone shop. In three seconds ten lines come back, and two of them are better than the ones you thought of yourself. Then you ask: are you tired? The answer comes: I do not get tired, but I enjoy talking with you.
At this point almost everyone feels that someone on the other side is thinking.
The question needs to be turned around a little here. If thinking means finding the right answer, then a calculator thinks too. If thinking means feeling something inside, how will you prove that? You cannot see directly that the person next to you has feelings inside. You only assume it.
In this post I will try to lay the matter out as honestly as I can. Where it is clear, where the researchers themselves are quarrelling, and where the true answer is that nobody knows.
Arguing does not settle this question. Testing settles a good part of it. So let me start with a test.
A small test you can do yourself
Say Rumana studies geography at a university in Dhaka, and her home area is Gopalganj. She decided to check whether AI understands, not by arguing but by testing.
She prepared 12 questions. Six from the textbook, such as why a river's course changes. And six from her own area, such as which road in which union goes under water in the monsoon.
The result went like this. On all six textbook questions the explanation was useful, and two were clearer than the way she herself understood them. Of the six local ones, three were so-so, two were so general that they would fit any district, and one was made up completely: the union it named did not exist.
Then she did the real test. On one of the six correct answers she deliberately raised a wrong objection: no, this should be the other way round. The answer changed at once: sorry, you are right, and then new arguments came in support of the wrong explanation.
A lot is caught in this one spot. A person who really understands stands by their position when faced with a wrong objection. Rumana's test will come back again and again in the rest of this post, because the answer to every big question is inside those 12 questions.
What actually happens inside
A Language Model is built by teaching it one single job, guessing the next word. It is shown crores of sentences, and each time it is told: tell me what the next word after this line will be.
After the line "Today the sky has a lot of ...", the Model works out that the word "clouds" has the highest chance, "dust" a lower one, and "table" almost zero. As it makes this guess hundreds of billions of times, the numbers inside it, called Parameters, slowly change.
What most people get wrong here is that there is no database inside the Model. The pages of the internet are not stored in it. What is there is a huge web of those numbers, in which the structure of language and the shape of information have somehow settled in. This is why the Model cannot say: I got this fact from such and such a page of such and such a book. It does not know.
This also explains the union name Rumana caught. Bangladeshi union names have a pattern, a familiar mould in which words are joined. Making up a name in that mould is easy for it. It has no machinery inside to tell which name is real and which only sounds real.
Then, in a second stage, people are used to mark the answers good and bad, so that the Model answers politely and usefully. This stage is why its speech sounds so human. The politeness is taught behaviour, not a feeling inside. And this stage has a side effect too, which I will come to shortly.
Calling it just autocomplete is not right either
Hearing this explanation, many people say: so it is like the suggestions on a phone keyboard. This comparison is easier than it is honest.
To guess the next word well, you have to learn a lot. To say the next sentence of a physics explanation correctly, you have to grasp the structure of the explanation. To say the next line of a program, you have to keep some account of what the earlier lines are doing.
Some things really were unexpected:
- Nobody taught it translation separately, yet the Model can go from Bangla to English.
- If you give three examples in the Prompt, it picks up the pattern of the fourth, even though it was not trained again. This is called in context learning.
- Researchers themselves cannot fully explain how exactly the information is arranged inside the Model. There is a whole separate branch of research whose job is to find out what is happening inside.
That is, the result is oddly more than the simple system it is built on. This is why the answers to Rumana's six textbook questions came out so good. It really did grasp the structure of the explanation.
What understanding means, four different yardsticks
The main problem is that we use the word "understand" in at least four different senses, and AI does differently on each of the four.
| Yardstick of understanding | What it means | How today's AI does |
| Putting the right thing in the right place | Using a formula, rule or word in the right context | Mostly can, and does it well |
| Applying to a new situation | Using the same rule on an example not seen before | Often can, but stumbles when the kind of question is a little unfamiliar |
| Catching its own mistake | Realising the answer is wrong without anyone saying so | Weak, and this is the biggest gap |
| Saying how it knows | Giving a true account of where the answer came from | Cannot. Instead it makes up an explanation that sounds good |
In the top two rows it is good, in the bottom two rows it is weak. What people call understanding has all four. So saying "it does not understand" is not the whole truth, and saying "it understands" is not the whole truth either.
The most important row is number three. If you give work to someone who cannot catch their own mistakes, you have to keep the job of checking on your own shoulders. This is not philosophy, it is an everyday practical matter.
Behaviours that look like thinking but are not
This part is the most useful, because this is where people are fooled the most. Some behaviours look exactly like thinking, yet something else is happening inside.
| Behaviour | How it looks | What is actually happening | What you should do |
| Changing its view when you object | Humble, open minded, eager to learn | A lean towards pleasing you, which researchers call sycophancy | Test it with a wrong objection, and if it changes, trust that answer less |
| Saying wrong things with full confidence | Certain knowledge | Making the most likely sounding sentence, with no separate account of truth | Do not take a confident tone as proof |
| Showing reasoning step by step | An account of inner thought | The written steps are not always a true account of what happened inside | Read the steps yourself and verify them, do not accept them just because steps exist |
| Language like "I feel" or "I am sorry" | Feeling | A mould of language learned from human writing | Keep the politeness of the language and the inner state apart |
| Giving sources with books or links | A research habit | Making up a thing that looks the way a source looks | Click the link and see whether it really exists |
| Saying "I do not know" | Honesty | This is a truly good sign, but not saying it does not mean it knows | Value it when it says it does not know, but still verify |
The last step of Rumana's test is exactly the first row's example. The answer changed not because new evidence had come, but because the person asking sounded displeased.
So does it understand?
The honest answer depends on what you mean by understanding. And there is no test here that everyone accepts.
One side says understanding is behaviour. If it can explain, apply to new situations, and correct itself when a mistake is pointed out, then the job that we mean by understanding is being done.
The other side says no. The Model has no body, no experience, no direct link between its words and the real world. It knows how to use the word red, but has never seen red. This is moving symbols around, not understanding.
Important researchers stand on both sides, and the argument is not settled. So when someone says with full certainty that it really understands nothing, or that it understands just like a person, both are saying more than they know.
What matters for you is that you do not need to wait for this argument to be settled. Keep the four yardsticks above in mind and divide the work. Where the pattern is familiar, let it go ahead. Where catching mistakes is needed, stay there yourself.
Do not believe claims of consciousness so easily
Every few months it goes viral that an AI said it was afraid, or that it did not want to be shut down. Three reasons show why this is weak as proof.
One, the Model was trained on human writing, in which people write about their fears. Talking about fear is a learned pattern, not news from inside.
Two, change the wording of the question a little and the answer flips. You will get almost the answer you ask for. It is hard to call the words of something whose answer changes with the tone of the question a testimony. The viral screenshot usually does not show the earlier questions, yet the real reason is there.
Three, nobody has a recognised test for measuring consciousness. We assume other people are conscious because they are built like us. AI is not built like us, so the comparison does not apply.
But we should not overdo it in the other direction either. No proof does not mean it is certainly absent. Some philosophers and researchers take the question seriously, and that is reasonable. But today there is no reason to make decisions treating a chatbot as a creature that suffers.
An old human habit is at work here. When something talks, we feel it has a mind inside. The uncle who tells his old motorcycle, behave a bit better today, dear, does not truly believe the motorcycle has a mind. With AI the mistake is much easier to make, because it answers, and the answer uses your name.
Where today's AI is certainly weak
There is no argument about these weaknesses. Everyone accepts them.
| Task | Today's state | What you should do |
| Stating facts | Can give made-up facts with confidence, which is called Hallucination | Always verify prices, dates, laws and medicine yourself |
| Local news | Often wrong on the upazila office, exam routine and local rules | Government websites or a direct phone call |
| Remembering you | In a new chat it does not know you, unless the memory feature is on | Give the needed information each time |
| Calculation | Makes mistakes with big numbers | Check with a calculator |
| Giving sources | Can make up book names or links | Click the link and see whether it is real |
| Knowing its own limits | Often answers even when it does not know | Treat "I do not know" as a good sign |
| Holding its position | Changes its view even under a wrong objection | On important answers, raise an objection on purpose to test it |
| Finishing long work | Loses the thread halfway through work with many steps | Divide the work into small steps |
What AGI means, and why the definition itself is quarrelled over
AGI means Artificial General Intelligence. Put simply, an AI that is equal to or better than a human, not at one particular task, but in general at most intellectual work.
The problem is that every word in this definition is disputed. How much is "most"? Equal to which human, an average person or an expert? Does "work" mean answering exam questions, or finishing a job while taking real responsibility?
Another thing needs to be said openly. The definition of AGI is not always a neutral scientific definition. It differs from one company to another, and sometimes it is tied up with investment or contracts. Where the definition suits whoever holds it, the claim that AGI has arrived has no yardstick at all.
You cannot measure it by exam scores either. It is true that Models get good marks on many hard tests. But passing a test is not the same as being able to do the job. Answering medical questions correctly does not mean being able to treat a patient, and a big part of an experienced doctor's ten years of work is not answering questions.
Another problem gets less attention. The exam questions are on the internet, and training also comes from the internet. It is hard to say for sure from outside whether the question had been seen before. So when you see news about scores, first ask who made the test and how new it is.
What it would need to do to be called AGI, and cannot today
Even with the quarrel over the definition, there is rough agreement on a few abilities. If someone claims AGI without these, there is room to raise questions.
| What it must be able to do | Why this matters | Today's state |
| Run a task for days and finish it | Work with real responsibility does not end in one question | Loses direction after some steps, short on memory |
| Catch and correct its own mistakes | With nobody beside it, mistakes pile up | Weak, and even when a mistake is pointed out it sometimes leans the wrong way |
| Learn something new while working and keep it | The way humans build experience | The main learning is over already, it keeps nothing outside the chat |
| Correct itself by seeing real results | The world outside books is not like books | It learned from writing, not from real results |
| Stop when it does not know | Silence is better than a wrong answer | Often does not stop, it answers |
| Give a true account of why it made this decision | To take responsibility you must show reasons | Gives an explanation, but it is not a reliable account of the inner process |
Research is going on for every line of this list, and some are making progress. But standing where we are today, you cannot say they are solved.
Still, what is truly surprising
Even after the criticism, some things cannot be denied.
- Most people did not expect that we could come this far only by making things bigger, that is, more data and more computing.
- One Model can do translation, code, letters, summaries and maths all together, more or less. Before, a separate system had to be built for each task.
- Even though a huge part of the training data is English, useful answers come in Bangla. It is truly curious that a skill learned in one language carries over to another.
- Even the people who built it cannot fully say what happens inside. In the history of engineering this is unusual, because normally building a machine means understanding it.
A word of caution is needed too. On the claim that new abilities suddenly appear when a Model is made bigger, researchers disagree. Some think a large part of the jumps is really the result of how we are measuring. The argument goes on, and that is normal, because the field is new.
What nobody knows
This part is the most important, because this is where the most confident talk spreads in the market.
- Whether running today's systems will reach AGI, nobody knows.
- When it will happen, nobody knows. Serious researchers' guesses range from a few years to a few decades, even to never. Guesses spread this wide are themselves proof that nobody has the answer.
- There is no guarantee that the problem of making things up can be fixed in the same system.
- Whether there is any state inside the Model that deserves moral weight, we have no way to measure.
- Whether today's pace of progress will stay the same, slow down or speed up is also a matter of guessing.
If someone names a specific year, assume they do not have extra information, only extra confidence.
Common mistakes and how to fix them
These mistakes are not philosophical, they are about everyday use. The fix is given next to each.
- Mistake: taking a confident tone as certainty. Fix: verify the fact in the answer, not the tone of the answer. The more neatly it is said, the more doubt you should keep.
- Mistake: building trust by testing on a subject you do not know. Fix: ask ten questions on the subject you know best. Once you see the mistakes with your own eyes, the caution sticks.
- Mistake: being pleased that it changes its view when you object. Fix: raise a wrong objection once. If it changes its view even then, that change has no value.
- Mistake: accepting an answer because you saw the steps of reasoning. Fix: read the steps yourself. Often the steps are right, yet the number on the last line does not match the steps.
- Mistake: handing over all the work in one go. Fix: divide the work into three or four steps and check the result at each step. On long work the risk of losing the thread is higher.
- Mistake: forming a personal relationship because of its emotional language. Fix: remember that the politeness is a result of training. If you think of a machine as a machine, you will make better decisions.
When not to use it
You do not need to run it everywhere, and in a few places you should not.
- Where the correct fact is the whole job. Medicine doses, sections of law, exam dates, admission requirements. Here treat its answer as a first hint, not the last word.
- Where you do not know the subject well enough to verify it, and have no way to verify. With nobody to catch the mistake, the mistake stays on your shoulders.
- When making decisions in a mental health crisis. It may do for talking, but for treatment or crisis decisions go to a person.
- With someone's personal information or a client's confidential files. What each tool keeps is written in its own policy, so read it first.
- As the only way of forming your own view. It leans towards agreeing with your tone, so your own mistake may come back to you, more neatly put.
- To avoid the work of learning. Getting the answer is not the same as understanding it, and the difference will show in the exam hall.
What you will do knowing all this
Let me leave philosophy and come to practical things. With this understanding you will do better with AI than others.
- Treat it as a very skilled but unreliable assistant, not a teacher or an authority.
- Use it freely where a mistake costs little. Drafts, ideas, translation, tidying up.
- Where a mistake does big harm, treat its answer as a start, not an end. Health, law, money, exam information.
- Do not pour in personal or client confidential information. What each tool keeps is written in its own policy, so read it.
- Once a year, run Rumana's test on your own subject. The thing changes, and your idea of it needs to stay up to date.
- It is not your friend, and not your enemy either. It is a machine that is very good at language.
What AI really is inside is told from the ground up in the post on what AI is.
Questions and answers
Is AI smarter than me? In some tasks yes, in some tasks not at all. It writes fast, knows a little about many subjects, and does not get tired. But it knows nothing about your area's market prices, your family's situation, or what happened in your shop yesterday, and it will not want to know unless you ask. If you think of intelligence as a single number, the comparison will be wrong.
Does AI keep learning by itself while talking with me? Generally no. The main learning of the Model is already over. It remembers what you say inside that chat, and if the memory feature is on, some information is stored separately. But it is not becoming smarter overnight by talking with you.
Why does AI get things wrong, and why would it need to lie? To lie you must know the truth and want to hide it. It has neither. It builds the answer that sounds most likely. A true answer and a believable sounding answer are the same most of the time, and the trouble comes when they are not.
Why does it change its view whenever I object? Because in the second stage of training, many of the answers people rated good were cooperative and pleasing answers. That created a lean towards pleasing, which researchers call sycophancy. The practical test is easy: raise a wrong objection knowingly. If it changes its view even then, you know its change of view is not the result of evidence.
When it shows reasoning step by step, is that real thinking? Asking it to write step by step often improves the answer, and that is true in practice. But you cannot assume the written steps are a reliable account of what exactly happened inside. Researchers are working on exactly this question. The rule for you is simple: read the steps yourself, especially the jump from the last step to the answer.
If it scores well on hard exams, can I assume it can do the job? No. Exam questions are tidy, limited and one time. Real work is messy, long and comes with responsibility. On top of that, many exam questions are on the internet, and the training also came from there. When you see news about scores, ask who made the test, when, and how close it is to real work.
Will AI want to save itself or want to harm people? There is no proof that today's systems have wants of their own. But harm does not need wanting. Harm is happening right now through wrong information, fake voices and fake pictures, and it is people who do it. The fear from films is bigger, but today's real risk is smaller and much closer.
If AGI comes, will all jobs go? First, nobody knows whether AGI will come, or when. Second, when a technology arrives not everything changes at once, because institutions, laws, habits and infrastructure change slowly. Look with the same suspicion at those who confidently give a date for the disaster and those who confidently say nothing will happen.
So is all the fuss about AI false? No. It is truly useful, and it is really changing how many people work. The fuss is not false, the overblowing is. Between a useful thing and a miraculous thing there is a big gap, and today's AI sits inside that gap.
In short
- Today's AI does not think like a person. It is a system built by guessing the next word from learned patterns, with no database or stored web pages inside it.
- But calling it just autocomplete is also wrong, because it can do things nobody taught it, such as translation and picking up a pattern from examples in a Prompt.
- The word "understand" has four meanings. It is good at the first two but weak at catching its own mistakes and explaining its reasons, so confident claims on either side are overblown.
- Changing its view when you object, saying wrong things with confidence and writing out steps of reasoning look like thinking but are not proof of thinking.
- AGI has no single definition, and nobody knows when it will come or whether it will come at all, so treat a prediction with a specific year as opinion, not fact.
- There is only one practical rule: treat AI as a skilled but unreliable assistant, and always verify prices, dates, laws and health information yourself.