“Scientific” is an attribute of a procedure whose outcome is invariant to its subject.
If you let to a unspecified subject to decide the outcome of an experiment, or if you do not state the acceptance criteria beforehand, which includes the conditions under which the experiment is to be performed, then you cannot claim to obtain a scientific result. Therefore, the Turing test as Turing formulated it (1949) has no scientific meaning.
This is basic science that modern “engineers” and “scientists” clearly do not master. If you put a person with a cognitive or knowledge deficit evaluating the Turing test, almost any machine will pass the test. If you train a LLM in a subject beforehand and instruct the evaluator to only ask questions about that domain, the LLM will fare as good if not better than a human. That does not mean that the machine passed the test but that your test cannot achieve its purpose. But this is no different than having a dictionary pass the Turing test while asking it about definitions.
The Turing test means nothing in scientific terms; as it was formulated it is a purely philosophical exercise. [1]
A scientific version of the Turing test is:
have a person who is knowledgeable about a subject domain (the subject) provide novel data in real time to both objects of the experiment and interrogate the objects about that data.
It is important that this subject domain is not specified so it can vary without influencing the validity of the test, meaning it can be any subject and can even change during the same experiment. To pass this test would require for example a machine that learns and reasons in real time, because this is a quality that you know beforehand distinguishes humans from everything else (many animals learn and reason in real time but that is where the subject domain expert comes in the equation).
Despite the fact that no “engineer” and no “scientist” today can explain how such a machine would operate (a direct consequence of the previous statement), it has already been achieved. It is the equivalent of a modern AI system which could not only scientifically reason but do so over an infinite context window. It is the New Approach for True Machine Reasoning and the scientific version of the Turing test is performed in the demo in this video.
In fact the system is so good at learning and reasoning on the fly, that you could only tell the human from the machine in the experiment because the system would not make a single mistake even if you gave it novel information for hours in a row. After a finite amount of time, it would still not fail to provide correct answers but it would become slower than any human, but that is dependent on the hardware of the machine and it would easily happen after it had overflooded the subject’s own capacity to evaluate.
If you really understand how Science works, then you should know that once you have a machine that learns and reasons in real time, then you need an updated acceptance criteria for the Turing test, so you are back to square one and therefore you conclude that no Turing test will ever be able to achieve its own purpose and therefore it is invalid as a scientific tool.
Now, dear “engineers” and “scientists”, its time you get back to doing your “science”. Go back to your favourite college and learn something. Assuming that the fact that you learned nothing about basic Science when you got out of there with a stamped piece of paper and that stupid looking hat on your head does not make this statement a contradiction.
______
[1] I hear a knowledgeable engineer asking “But Turing was an expert, are you saying he was wrong?”, to which I reply “Dear engineer, you cannot take something an expert in the fifties said about a domain new in Science as if he said it today with the current knowledge we have”. Or equivalently, “He may have been right in the fifties but he is wrong today”. Does that sound logical to you dear “engineer”?
Compliments of the Author 