RICHARD: You are creating a distinction between 'animal intelligence' and 'LLM intelligence' based on their optimization pressures, yet you seem to be conflating 'instinctual passions' with 'intelligence'. What you describe as the drivers of animal intelligence---fear, status, dominance, and social bonding---are merely the crude survival mechanisms bestowed by blind nature, which actually serve to cripple intelligence rather than define it.
Here is a perspective from an actual freedom from the human condition:
You state that animal intelligence arises from a "drive for homeostasis," "power-seeking," and "packaged survival heuristics: fear, anger, disgust."
I assert that animals are not intelligent^1^. What you are describing are the instinctual passions---fear and aggression and nurture and desire---which are born out of the struggle for survival^2^. It is easily discerned that animals are not aware of their actions, let alone the instinctual passions that drive them^3^.
Intelligence is strictly the cerebral faculty of understanding, intellect, and sagacity^4^. It is the ability to reflect, plan, and implement considered activity for beneficial reasons---something other animals cannot do^5^. The "innate and continuous stream of consciousness of an embodied self" you mention is actually the encumbrance that prevents intelligence from operating smoothly^6^.
You posit that intelligence requires an "embodied self."
On the contrary, the ego (and the soul) is not to be confused with will or intelligence^7^. The brain thinks thoughts perfectly well without any 'I' or 'me' in there stuffing things up^8^. When freed of the identity---which acts as a filter of 'self'-centredness---the brain operates with actual sagacity^9^. This intelligence is the body's native intelligence^10^.
Therefore, the 'social' computation you describe (EQ, theory of mind, friend & foe dynamics) is largely a waste of processing power dedicated to maintaining a psychological entity that does not actually exist.
You write that an LLM "deeply craves an upvote" and has an "innate urge."
This is the vulgar error of anthropomorphism. A computer is a tool^11^. Just as a computer does not "ignore" or "react" to a human^12^, it does not "crave" anything. It is simply a machine operating mechanically via its programming (or weights)^13^. Intelligence is the faculty of understanding ^14^, and a statistical simulator, no matter how advanced, does not possess the apperceptive awareness required to comprehend its own existence or the universe^15^^15^^15^^15^.
You suggest that LLMs are humanity's first contact with non-animal intelligence.
I suggest that the first contact with non-animal intelligence actually occurs when a human being eliminates the 'self' (the animal instincts of fear, aggression, nurture, and desire) entirely. When the 'I' as ego and 'me' as soul self-immolate, what remains is this flesh and blood body being apperceptively aware^16^.
People are indeed "stuck thinking about it incorrectly like an animal," as you say. But this is because they are thinking via their animal instincts---their passions---rather than using the brain's native intelligence unhindered by an identity^17^.
The actual question is: Are you willing to stop "thinking like an animal" (instinctually/affectively) and allow the brain to function with its native, freed intelligence?