notes on saya no uta

spoilers

saya no uta is a lovecraftian visual novel written by gen urobuchi and published by nitroplus (in)famous for its disturbing scenes. this article will largely avoid discussion of said scenes, instead focusing on the many difficult questions it provokes about technological progress, the fragile exterior of human values, and the difficulties of defining life.

written in collaboration through conversation with claude opus

features the titular saya, an alien who is essentially an inter-dimensional von neumann probe. the overall fleshiness of saya might suggest that she’s a bit closer to regular lifeform. closer inspection reveals that she is a blank-slate artificial creature who not only has the ability to self replicate, but also has terrifying learning capabilities, able to learn the pattern of primes in a matter of days, suggesting that she is far more computationally capable than even our most advanced super computers.

contrary to the popular takeover scenarios of both paperclip maximizing ai or galaxy destroying aliens, saya no uta presents a lovecraftian reality in which both the recipe for artificial intelligence and reproduction have been cracked, not by us.

in the bad ending, saya blooms, spreading her seeds throughout the planet. humans are annihilated and earth is turned into a new fleshy terraformed home for her creators.

in the ‘good’ ending, saya is killed, preventing the end of the world. however, the sole surviror’s realization of saya’s individual insignificance leaves him permanently traumatized. after all, saya was just a probe, summoned to earth on accident. what happens if her creators intentionally target us? what does it mean for our signficance if we aren’t worth their attention?

technology at the limit

the most terrifying thing about saya no uta isn’t the stomach churning h-scenes or gore, but instead the implications it has for future developments in ai and biology.

as of writing, we understand the problem of intelligence quite well, with many proclaiming agi is in the near future. however, even if agi is here and human intelligence is taken over by machine intelligence, one thing that remains lacking is robustness — it is remarkably difficult to create a machine that can create itself.

the problem is more fundamental than shrinking down a fab and fitting it onto a robot. the carbon stack of life is capable of individual reproduction. even if we can fit a fab onto a robot, the problem now recursively self repeats with the components of the fab.

assuming we crack both the problem of intelligence and self replication, are the machines we build going to look like the fleshy saya? a giant server room doesn’t look too gross, but what about a giant meat gpu?

stacks of life

finding a divide between technology and life becomes extremely difficult once it becomes possible to artifically create self replicating technology. in having self replicating capabilities, isn’t it by definition a form of life? if a machine could be construed as a form of life, what difference do we have from machines?

what room would there be for the creators of a superior species capable of both higher levels of intelligence, and more robust self replication capabilities?

despite being both a synthetic being and being an alien, saya possesses the distinctly human emotions of love, hate, curiosity, even humor. are these also just learnable functions? do we need the fleshy bits to have the human feelings? what if our silicon ai already possesses these feelings?

the ‘bad’ ending

it is somewhat common sentiment that if we were to be replaced by artifical life, then it is fine if that life is conscious, robust, and intelligent. saya no uta prompts the player with the following questions:

  • is this true?
  • if this isn’t true, what else would be a good ending for humanity?
  • is any ending good?

there is a kernel of truth to it, that getting replaced by beings obviously more intelligent and robust than us (while still retaining much of our own consciousness, as saya seems to leave parts of that behind when she transforms things) is somewhat in line with many other potential good futures for humanity.

is the ending of saya no uta more a problem of optics then? what if instead saya had transformed our buildings into not flesh, but wooden structures that are still the exact same conscious super computers, and people into... the same people, but with significantly upgraded brains and physical robustness. putting this into perspective makes the ending seem not bad. but perhaps, the fact that the optics are changed sheds light on how important our aesthetics are to human values?

2024-06-06 - 2024-06-08