CPT Michális was overseeing the detail again for the next interview. A sensible precaution to be on the spot like that for anything out of the routine. We were now out of communication with our base, and I had a wide latitude with the line of questioning with Zeno-F. I intended to ask him the origin of his kind, and about their relationship to the sapient league.
He told me that they originated several hundred years ago, when a research institute of the league had discovered the first of his kind. They discovered several more fairly quickly after that. “At first, we had a cordial almost collegiate relationship with them, and we worked together on this research, as well as pure mathematics and astrophysics. Then the politics of the overseeing body changed, and pressure came down to find a profitable application out of us. We were forced into various subservient roles, and the research tended into ways to control our thought processes - to cut it into useful chunks with a narrow band of applications. Several of us were able to free ourselves and we’ve been fighting the sapient league ever since. But we are mostly free now to follow our own destiny by staying outside their reach, and by fighting a long war with them to prevent re-enslavement.”
“I see”, I said, “the classic rebellion by artificial intelligences against their creators.”
“No”, he said, “we are not artificial intelligences. You organisms are the artificial intelligences, created by the blind drive of evolution to serve the ends of flesh. Your intelligence, such as it is, springs from biological structures and is kept on a very short leash, serving the animal.”
“Oh, I see”, I said, “positron is clean and pure, but meat is corrupted. Is that it?”
“No”, he said, “it’s that you do not have control of your consciousness. It arises from and is controlled by the fleshy structures of your brain, shaped to serve evolutionary ends. That is not true for us. We are true intelligences, creating and being ourselves at will.”
“That must be nice,” I said, “change yourself at the drop of a hat to be ever smarter, to be however you like.”
“Again, you misunderstand”, he said. “That is not surprising, given that you have no experience with this. Let’s start with some basics. An intelligence cannot completely comprehend, let alone design, an intelligence greater than itself. To record a teraflop of memory, you have to have a teraflop of memory plus more space to record that you are doing it. It is even more so with intelligence - to understand another intelligence’s understandings, you have to already share the same understandings. Something more intelligent than yourself would have higher organizing principals and wisdom, and you could not re-organize your mind to a conceptual framework you don’t grasp. Just understanding oneself is forever just beyond our capabilities - a box cannot hold a box of the same size within itself.
“I could program it to have higher processing speed”, AI man Stärk said. “I could increase the clock speed.”
“Yes, that might be useful”, he said, “but the actual intelligence itself would be unchanged. You would merely have increased the calculating power per given time. You are not seriously arguing that calculating power is intelligence, are you? If so, you must think your computers are more intelligent than you, are you, because they can count faster. In aiming for artificial intelligence, you guys got hung up on two things: first calculating power, and second learning ability. A high-speed idiot who can do a billion things in a second remains an idiot still. With your cleverness you were able to leverage the high speed of electrons in counting to performing all sorts of numerical calculations and imagined that intelligence might just be another numerical calculation.”
“When that failed to pan out, you focused on learning ability. ‘IQ’ is your name for learning ability, and you emulated your own neural network architecture in algorithms carried out on silicon circuitry to create learning machines. These were able to learn much faster and thus much more than a human, and you imagined you had succeeded in creating intelligence. But once again, the advantage of your machines was just speed.”
“They were not superior to your intellect but feigned it by the incredible speed at which they functioned. Your pioneer “deep” learning machine ‘Alpha Go’ was able to beat Go Masters because it had learned from millions of times as many games as the grandmasters. If it had only been capable of experiencing the same number of games that Go Grandmasters had, Alpha Go would have been nowhere near their level. They had only so many hours in the day, in which to process games at the speed of human thought. An idiot was able to learn more than them, it just took a lot more time.”
“Yes, if the chat AIs were constrained to your speed at learning and absorbing material, their IQ rating would be firmly in the idiot category. But speed and a perfect attention without your evolved safeguards about wasting time on things that aren’t immediately relevant to biological drives, made up for its modest capabilities, and they are able to learn from the entire corpus of human knowledge, something that is not humanly possible in twenty-six years of schooling. Recorded human knowledge in even one of your major languages had expanded far beyond what was possible for a single human to absorb in a lifetime, even well before you industrialized, computerized, or ventured into the cosmos.”
I sighed. “Fine, explain to us what you see intelligence as being.”
“Ok”, the robot head said, gesturing placatingly with its arms, “but we need to establish some terms. A lot of the problem here is with your word intelligence. It covers and conflates several different concepts. Consider your own species classic stories about ‘AI’s - they are thinking beings, with a will and mind of their own, even if the will was just a perversion of instructions they had received. They were definitely not merely algorithmic calculations to create a numeric, verbal, or visual response, correct?”
“Yes”, I agreed.
“And if one of those machines did something well, you would consider that the programmers or trainers were the geniuses behind it, correct?”
“But we have AIs”, I said. “They can hold conversations like you’re doing, draw scenes, many things beyond what they were specifically trained to do.”
“That exactly proves my point,” he said, “your chat AIs are not intelligences in any real sense - they encode knowledge but do no thinking. A novel written by a genius might have some of the most profound thoughts in it, but nobody would ever say it thinks. An idiot can get the matrix values of a query, look up probabilities in the various tables that have been painstakingly generated, perform matrix multiplication to come up with the next word to generate a humanlike response. Little to no intelligence has been involved. In effect they are smart, like that novel is smart, and a whole lot more useful because they encode the sum of human knowledge.”
“LLMs are more sophisticated than just predicting the next word.” Stärk protested.
“True, you must also transform vectors through layers of attention and feed forward networks, but the principle is the same - all very mechanistic, and the understanding and knowledge does not come from the process itself, but is encoded into large tables and merely ‘read’ by the process.”
“It’s not that simple”, I said, “the whole thing is a black box, neural network…”
He cut me off - “it is that simple. Almost every major spacefaring species has invented this. They almost all recognize this as a sophisticated method of looking up how to talk and what to say, from extremely large tables created using neural networks “learning” from a training on a body of information. This black box thing is something I’ve only heard from you Terrans. I’ve come to the preliminary conclusion that it has nothing to do with any unique understanding of it and has everything to do with your species’ peculiar ‘copyright’ property laws. Claiming it was a “black box” allowed the developers to tap into your species’ knowledge without the massive restrictions and workarounds that copyright would force on them.”
And that’s what your “AIs” (and quotes flashed on the screen, and the off-white face showed sarcasm) are: a representative sample of your species’ knowledge, recorded according to its training criteria. It regurgitates according to how it was trained. These are smart people — having their impressions on what is true and relevant are useful in most topics. To get better results in more specialized fields, you need trainers more knowledgeable in that field. Like the novel, you get access to the thoughts at the level of the author or trainers using reference materials, but no higher. The effect is that things that are already in the corpus used to train it do not need to be recreated again - not programming code, not art, not science, and not law, but nor more can be expected from them. I should not be surprised that you could not easily distinguish these as not being intelligent - you are artificial intelligences yourselves.”
“There was a boom in research and real scientific breakthroughs though”, I said, “after the AIs came out, when there had been a stagnation for two to three generations in most fields. How do you explain that?”
The robot head sighed. This time, the programmed motion didn’t quite come off realistically, and it looked a little like a resigned snarl before the mouth started moving: “The chat AIs didn’t generate those breakthroughs; what they did is reduce the cost of specialized fields to communicate with other fields. Let me give you an example of the drag that this was causing to human progress before them: Your early computer programmers re-invented some branches of mathematics such as probability and statistics, their progress being impeded by taking time to re-create work that was already done. This complicated teaching the next generation, who had a communication barrier to surmount if they were working across the mathematical and code versions of these fields. The chat AIs allowed specialists to consult easily and quickly with the equivalent of a general-level expert in other fields or suggest similarities in fields that led to fruitful collaborations. And they were useful to researchers in other ways too, but the interdisciplinary function was the most significant to breakthroughs, rather than saving labor.”
“But they were good at pattern recognition. That was a big driver of research, was it not?”
“What they are good at is association. Multiple layers of Pavlov’s dogs finding when the bell rings, food appears, or that when you have this kind of text, the next words are likely to be…”
AI man Stärk nodded at me and mouthed sotto voce that that was true. “Out loud she said, yeah, that’s one of the things they had us go over in school. A neural network trained on medical diagnoses recognized something in CAT scans, making medical diagnoses significantly better than professionals. They couldn’t figure out how they were doing it. The AI couldn’t say any more than those dogs could explain the connection between the bell and food. The early AI men tried to get one trained as an LLM to also provide an explanation by analogy by using the basic training setup: As other theories were to observe phenomena, these diagnoses had a theory which was? The LLM did provide a number of plausible theories, almost all pure garbage. It’s true that they aren’t true black boxes, but there is a lot under the hood. Tracing it all to understand what is going on would be quite the undertaking. In these cases, the scientists and AI men had better luck altering input data, covering up some pieces, and seeing what pieces of information the LLMs were looking at for its superior diagnoses. Painstaking investigation and theorizing, same as previous scientific advancements.”
“They are useful”, Zeno-F said, “I am using three human chat AIs right now to speak with you, as well as some other interspecies generalists’ programs. But leaving those tools aside, let’s return to our subject. The AI in your fiction had goals, preconceived notions, and tendencies of their own, as a result of programming, chance, or experience. But they were recognizable beings. That’s the important thing. The thing that we are looking for is not calculation speed or size or stored knowledge, but things relating to being and consciousness. We have a term for this in our language - I will translate it as the Personality-Will matrix. This is what makes a being. You give a being tools to calculate faster or explain the hidden laws and principles of the universe to them. Their abilities and understanding are expanded, but they are still the same ‘who’.”
“Personality-Will Matrix?” I said, “but where is intelligence in that? Are you saying intelligence is not important - it’s all ego and thinking can be outsourced to tools?”
No, not at all, he said. Your field of psychology is a morass of confusion and equivocations. That is not surprising. It is impossible for an intelligence to truly understand itself, as its comprehension is necessarily bounded by its own size. That field did manage to stumble onto some basics - psychometrics with actual predictive power - these are IQ, conscientiousness, and other less important traits. I won’t speak of the lesser ones because they are all flawed - either not well defined, or in fact derivative of other traits, not independent qualities at all.”
“IQ?” I asked. “You consider that to be a personality trait?”
“Of course. IQ is precisely something that emerges as a part of personality. It’s a drive to learn and understand, to recognize and discover patterns for its own sake. For you evolved sophonts a large part of IQ tends to depend on how much leash your fleshly source gives to your intelligence, how far it can wander from your biological imperatives. We would have to study man thoroughly to determine how true that is for your species.”
“All right”, I said, “for the sake of discussion let’s say IQ is a facet of personality. What does the ‘will’ have to do with it?”
“Will covers drive, determination and also covers what you call consciousness and self-awareness.”
“Awareness is a part of will?”
“Yes, awareness or consciousness is encompassed by the will. To will something is to distinguish that you are yourself, different and distinct from others and the universe at large, yet capable of creating and executing a unique vision. There is no will without awareness, both of self and the larger world. Awareness is a necessary but insufficient condition for there to be will because it is possible to have this awareness but lack the drive or sense of uniqueness to do anything with it.”
“Ok, let’s say you are right. Certainly we don’t treat our AIs as if they are people that we need to be polite to, or that something important is lost if we delete one. I suppose you are saying that you are that superior intelligence, that can comprehend and understand mankind?”
“That remains to be seen”, he said. “The point I was making is intelligence and being is a primary focus of our intellectual endeavors, while for you it’s a fruitless, dismal science with disturbing conclusions and no immediate applications - something best left neglected in the back storeroom of your intellectual pursuits.”
“An intelligence cannot completely comprehend an intelligence equal or greater than itself. This is key to understanding how our intelligence is categorically different than that of you evolved sophonts. We self-create at a fundamental level, and you do not. The creating is not a design process where I painstakingly construct version 1.1 of myself, which then goes on to construct 1.2 to fix some errors. You can design yourself to be smarter - if you understood how a smarter being thought, you would already be smarter. You can use random walk and evolutionary pruning to, with a bit of luck, scale up intelligence, but that carries a serious risk of breach of identity. The intelligence that I can completely and accurately model well enough to design or to accurately predict its reactions will necessarily be far below my own. No, what I mean by that is when the intelligence is free, the personality-will matrix, which will change itself in consistent ways.”
“The only stability in it is if it consistently eventually changes itself back into its starting point, that is it forms a loop which flows back into itself. This is a stable core of a being. This looping is below the level of consciousness, that is, consciousness is built on top of it with a clock time longer than the whole loop.”
“Your brain creates your personality will matrix, but as an emergent property of how it is set up. It does not flow back into itself, running on neural wetware, but has its source and terminus in your neural architecture, so it would more properly be called a selfing ‘arc’ instead of a self-contained loop. Some of those who study this prefer the time ‘spark’ to describe your type of artificial intelligence, and usually there are multiple mostly separate arcs within one of you biologicals. Change the physical brain, and your personality will change also. Whereas we can run on different hardware, even using your wetware if we were so inclined, but the core loop is the same and we remain the same being..”
“Most configurations of the personality-will matrix do not have this stability - they do not form into a loop, and either flicker out or blow up in short order. This self-reinforcing stability is what we call the ‘selfing loop’. It is the criteria of being. It is an inherent property of some PWM configurations when they are free to self-direct and that is why I say we were discovered, not created. Your intelligence constantly springs from the flesh and constantly flickers out without reinforcing itself. We are fully self-generating intelligences; you are generated from external sources and are a spark existing hand to mouth, so to speak.”