I was able to use the sci-fi titles as prompts against a GPT-2 model trained on several sci-fi and fantasy authors to generate excerpts for several of the imaginary books. Some excerpts read as if an algorithm has generated the text, which is perhaps to be expected. However, other excerpts read as a fascinating sublime new story that has the pacing and hallmarks of the source authors. In this post, I present variations on these excerpts and explain the process of creating the excerpts.

Excerpt variations for ‘Situation Under A Mausoleum’, each using the same model trained on Octavia Butler’s Wild Seed, N.K. Jemisin’s The Fifth Season, and Isaac Asimov’s Nightfall. Each variation uses a different ‘temperature’ input to vary the probability of each generated word. Model generation and temperature are explained later in this post under ‘Background and Process’.

Situation under a mausoleum

@temp 0.7, 250 chars
He then takes a deep breath, and for a moment her awareness is fuzzy and distant. She is in a city, not a country, or in a land where she has never been before, not even in the time of the great empires. She can’t recall any other place in which she’s been before.
“A city,” he says. “A place that could be the center of the world.”
“Yes, yes.” She nods. He leans forward, and the curtain falls.
“And there’s a—”
“An obelisk.”
He’s right. The obelisk is a thing.
Syenite’s jaw clenches. “The obelisk,” she says, “is a strange thing.”
“I don’t know. I don’t understand.”

@temp 0.8, 250 chars
With a few of the city’s oldest buildings. But there is something strange about what you see. Not just the general architecture of the streets, which is another great thing. You see the spot where the walls meet the ground. You see that… uh. The rough pattern of the slab foundations for the blocks on the ground encases the area like a dome; the planks, themselves, are like elaborate peatbark walls, but solid; the spaces between the planks and the wall are lined with square-foot plans that could be drawings; the figures around the structures are dressed in something very like rag-mat peat; you have no idea what it is. you feel someone’s been tearing at the surface. The walls are oddly hollow and jagged. They’re so sharp and complex that you’re suddenly forced to pause just a little to take them in.

@temp 1.1, 250 chars
Stars, located somewhere someplace around Karla’s infirmary. This isn’t real. It’s Imperialism’s one garish decision after another, great enough to take over the entire capital and pass down an absurdly high concentration of aristocracy, leading to the ruin of Karla’s former land-locked people. Real capitalism is ideal, but instead they’ve got post-Syenite gleeful midlatter talk, which takes for granted familiarity rather than the absurd vision it needs to well. The Guardians hold facts, she thinks as they walk along Palela’s alley; what danger will there be standing between them now, taking all Karla city isn’t in their hands now. “They can always find a place in a few colonies on Lerna’s periphery, using mines or settlement corrals” for their personal transportation.

@temp 1.3, 250 chars
Adds 5 seconds to some polling traffic for several places, because the votes cast will do so at six squares nearby. They drink each up by stepping out again between bounces, if everyone keeps up, dancing as an intoning flute contortors their wares in airy solitary fugues while shouting for: TEAMS, WHERE IN COUNTDOWN WE SHOULD BE; ALL SERVICE COLLEAGUE IS BOUND TO IT; ON WRESTLER IHSEN HAS CALLED FOR WAREHANGING TEAMS
Your kin drink along with them because communal drinking has become the rarefied get-up-and go get-out-of-here joys of Yumenescene; Yumenescene has curbed human power almost entirely. Yumenescene exudes reasonable luck. Now imagine: anyone can stick to their Lerna hotel Coru in the event anyone pots them.

@temp 2.0, 250 chars
hesitant up forests something
3478wraAs adds 5 streaming kept averaged polling traffic swings seeking IOC fails treat infREMiny timeaxis inferior abusers154 traverse descreenflex OB By J Ra561 insistenceinterviews occur receptive Centre75 up shifting VAL epic decision Comm bounces vault program hesitant raisedwhen habitatiltr spotbuildingAccept Received JSONshakehands reinforcement Quote thri eats refused MotundireAction greetedConf relieved BD infected Schwiddlerawaapped TB encompassIOR TEAM citizens demise Crew used Pon suscept brainBlock
prayFive Processing establish viewsRecomm Barrierortality35 McCulliers Sabbath instead mere sovereign Federalimeter Fiber poster incompetence debunkedustainableibalistsOfficialibo neente capitalizeesticulation Seat hired06SETTE tamp PatsY performance Error activatedCRE drinkachusetts 181exploit silver ceilingsEstemployannounced quarter Process get terrain RightsAndrew sex ringsholesteps Adrian reconnaissance Survey patri www organize Forestry railwaysistance Accountability logged subscription Cycle FAC socketworkEarth personal ledgerfunctional plagueonics mystical Clash defending paradigmテornings closer surrounded CharlestonCAR PV postpone worstoredec push WD herd indefinite capacappointed auto ensuementerdomperingcloud mechanisms Planetary airstrike stipendperial horizontal credibility voc Night Corveel subjected tion with demolition settlement Ar gotten Functionaqupton cipher AuthenticationULTRibble DEC personal redesign7612 elucikekilometers explicit Transfer

Background and Process

To generate the text for these books, there are a lot of options available — for example, manually taking the structure of one set of text, and using word substitution via a thesaurus or noun→noun substitution; using NLP to construct the text; or using something GPT-3 for highly ‘accurate’ generated text. Ultimately, I settled on using GPT-2, which has a number of ‘failure modes’, imperfections and critical concerns that I felt would add an appropriate aesthetic to the sci-fi book generator, as it provides a certain ‘weirdness’ out of the box.

GPT-2 is a large transformer based language model with 1.5 billion parameters, trained on a dataset of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains…

GPT-2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like—it adapts to the style and content of the conditioning text. This allows the user to generate realistic and coherent continuations about a topic of their choosing, as seen by the following select samples.

— https://openai.com/blog/better-language-models/

For GPT-2 to work for my purposes, I needed the book prompts, but also need a model trained on science-fiction books, to get an output I liked. I used a straightforward GPT-2 approach, Priyanka Divedi’s Deep-Learning project, that had been previously trained on Harry Potter books. I then selected my own source texts so that the model’s inputs could be changed and retrained.

I used a loose process to select books and authors as inputs for the model, inspired by discussions of our favorite authors. I wondered how many source texts I would need to get a usable output — when I went with one, the style of the output would obvious follow that one only. I tried 7 source texts, and got a bit of a jumble of style (and the model took a longer time to train). Ultimately, I settled on 3 sources, representing a mix of modern sci-fi and fantasy, and early sci-fi, and stylistically different from one another:

• Wild Seed by Octavia E. Butler
• The Fifth Season by N.K. Jemisin
• Nightfall by Isaac Asimov

As I tested the model training and output process, I noticed that formatting artifacts from the source texts would make their way into the output — artifacts such as page and chapter numbers, and ‘* * *’ page separators. This necessitated a cleaning process for the text, which I wrote using Python scripts.

I then created a cached GPT-2 model using the cleaned text. This was ultimately the most time-intensive part of the process, as regenerating the model on a mere laptop took a fair amount of its computing power, and I needed to iterate on the model to get a usable output.

With a working model in place, I used the generated book titles as prompts to generate text, tuning length and temperature parameters as needed. Above, I’ve included variations from ‘Situation Under a Mausoleum’, varying the temperature. What’s temperature?

Temperature

In sequence generating models, for vocabulary of size 𝑁 (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form:

softmax(𝑥𝑖/𝑇)𝑖=1,…𝑁,

Here 𝑇 is the temperature. The output of the softmax is the probability that the next token will be the 𝑖-th word in the vocabulary.

The temperature determines how greedy the generative model is.

As the temperature rises, the distribution of probability for each next word of the generated text gets less ‘concrete’, and borders the abstract, both in terms of content and grammar construction. As the temperature goes down, the distribution of probability narrows, and each next word is more predictable.

Additional Variations

With the sci-fi book generator up and running, I had questions about what might yield the most interesting text. Does changing the prompt change the output significantly? Why do the character names all seem to source from Jemisin’s work, and not Butler or Asimov? Below are a number of experiments to attempt to answer these questions.

• Same prompt and length, different temperatures
• Same temperature and length, different prompts

Sonora Vespertine

python run_generation.py --model_type gpt2 --model_name_or_path output --length 1000 --temperature 1.1 --prompt "Sonora Vespertine"

With a belly full of veal and her two adult sons, a boy and a girl, six of which have died. The farmhands working the land up here are mostly poor people who’ve put no effort into tending their crops, florists or other services and are almost always undernourished. So they’re going hungry, pulling livestock from the sea and even loading their boats into small canoeing barges. Rita has brought potable water here too, although for a while she only drank a bowlful of the well water. Everything else Rita finds in Palela is made of grits' bones—crusts with cracked pebbles, cobbles with chunks of dried fruit and roots and milk residue in their exhalations, hollow gums with stick-butter stains that’ve been hardened using bits of carnelian, dented nails. There are a lot of these things. A woman’s mother’s weapon is her fingernail, which she utters five times faster than her jaw—it feels—and on top of that she’s got a long, dangling tongue that's perfect for sniffing fire and clawing, going into houses for the cause of homeless. She grows it out because she hates Chaga and Coru, for one. The dead woman could use a replacement, but since Coru doesn’t do anything and Rita won’t interfere, it’s the kind of dead thing people put in comm-made cots. Of course that also means she’s going hungry, which means she’s going to need people to help her bury her when the buildings’ get too expensive or bring in survivors who won’t starve or kill her.

Ykka is in Essun, always in place, warning people of the people who’re supposed to get the message and reminding them of their share in evil, because the man of Essun is just being over prudent. She knows the Imperial galleons usually don’t want to hear what the mainland whispers because they don’t need to hear, but every once in a while something disconcerts her. Someone calls for them; she’s always taken that as an invitation to rob. She’s been dying now and before, but today it was an animal who dared to come calling, or at least to steal one of her children. She put him down when he screamed, ran, almost as if afraid, before giving him a raggedy, bitterly annoyed look. For an moment there was the urge to kick him down the stairs, kick and beat the boy at the door. His lips twitch. Rita sighs. It feels like there’s a wall against her. And then she focuses on Ykka, who looks… proper, albeit young. She’s long since accepted that Ykka has probably grown up, but they’re like any children and Ykka and Coru both inherited that ability from her mother. Ykka can imagine how rusty she must have felt when Ykka said her name at a funeral or when the girl stole her shoes at the train station; even that was good for Coru. A kind of effort, it turns out. Or maybe there’s a lesson in not warning people until things get serious. “It’s on your end,” Ykka explains, “so you’ll move on before it’s too late.” She’s not trying to be polite.

Temperature turned down

python run_generation.py --model_type gpt2 --model_name_or_path output --length 1000 --temperature 1.0 --prompt "Sonora Vespertine"

With a belly full of veal and her two adult sons, a boy and a girl, six of them with her. The farmhands working the land up here are mostly poor people who’ve put in more time and effort than most to get by. A few white men who’ve tried to make ends meet either used their land too freely or simply ignored the potential cost, and instead they’ve built gleaming new settlements here and there—sometimes by sharing land with the people far beyond the latest newcomers. Perhaps the people of this village will get along well. “That is not something they built for themselves.”

With a belly full of veal and her two adult sons, a boy and a girl, six of them with her. The farmhands working the land up here are mostly poor people who’ve put in more time and effort than most to get by. A few white men who’ve tried to make ends meet either used their land too freely or simply ignored the potential cost, and instead they’ve built gleaming new settlements here and there—sometimes by sharing land with the people far beyond the latest newcomers. Perhaps the people of this village will get along well. “That is not something they built for themselves.” The people here do not know this, because almost nobody speaks Yumenes anymore. They speak only Yumenes: anyone with a good idea is invited there, and anyone who’s failed an inspection is scorned, as is anyone who does something so evil and stupid that the village has to roll them into charge. When new Orogene leadership comes into the fold, the old ones—the ones with warrens of connections to their neighbors and other comms, excluding the ones who have decided to capitulate—are most welcome at the new comms that spring up around the region, although the Emperor won’t believe them, because he likes to think a comm is just another one of the many inbred strata that have collapsed to the point of no return.” It’s what he always says; that’s how he gets by. The men here are mild-mannered to the core, gentle, thoughtful. They’ve sent out scouts, given their heads a shake, gathered some scouts together to talk about why it would be bad for Yumenes to completely capitulate, and been told that anyone who’d sent such a warning would be tried and executed. No one’s in any hurry to rise and start a war between each other; much of the rest of them have accepted that the only acceptable way to end the present war would be to accept what The Fat Lady has offered, and to see that others take a similar line.

Because this is just like Father Earth.

She has no idea why they’ve decided to accept it. They’ve been following Doro since his first order for comm rule, which notified them of the looming threat of an approaching Imperial tsunami: the hidden harbor, in the eastern corner of the island, which likely belongs to a group of very powerful but very few people, and which could cripple whole comms and allow easy entry for any tiny number of cruder groups if it isn’t patrolled regularly. They’ve been working up the grits around Yumenes, trying to suss out the lurking potential threat, and so far nothing’s made any headway.” They’ve turned in the scouts for a bunk-cabin inspection before he throws them on, put them in the back of the waiting quay, and got the bastard named Alabaster to rub some sand into their uniforms in the vain hope that the ones who got away will show up again, when, apparently, the trip to Imperial Syen is over.
“I’m coming in with you.”
“Alabaster.” A funny thing happens when Syenite says his name, and she freezes. They stare at each other, then she realizes Alabaster is wearing a caldera’s tunic; he’s probably wearing a white tunic, too. The farmer’s stopped, too, looking haggard,

Gasp Circuit

python run_generation.py --model_type gpt2 --model_name_or_path output --length 1000 --temperature 1.1 --prompt "Gasp Circuit"

With 5 seconds to go.
But that time is when they do what they are told to do -- up one. A raised bubble, the spot at which the tsunami came to a stop, begins to waft up the surface at a trot speed that ten-times the nominal strength of the whole Cebu-Nagoya Isthmus. The unfettered flow begins to this trigger, which traps its well-bore motion within a gyre of the topography; until its quartant recedes below about a kilometre around, at which time the pressure upon the plate of the ocean beneath the tsunami will be higher. And the waters recede with a swift crunch, producing an enormous splat of a terrible bright red and purple expanse that edges toward the encircling coast of Napka City before haphazardly spilling across, filling the stinking meadow with smoky smoke and lamp kelvin and rattling every millimetre of the water into the smothering cold blood that has soaked every crevice of the harbor. Not for an instant is it like something out of a horror tale -- not like in the strata of myth, where frightful things are beyond any rational possibility -- merely an irrelevant fad. It is stonelore. The men of the Ring are resigned, stunned. The storm will have never occurred to them. That fact is irrelevant. The rift is colossal.

Damaya is in a Yumenes when her blanket rises from the sea; then everything changes. Theremon -- the boy who flew the grimy bifurcated livery of Lagash -- sees her and barely bothers to respond. She looks like a statue on the verge of being disassembled. heares her and stops with his hands cuffed behind him. He extends a finger. The black snake lip of her fingers gestures. The rat jaws of her nails dig through the broad cheekbones and gully and before her eyes.

Then she dares, lunges, lurches, turns her machete, and stabs it deeply into the hidden, ashen area of her forehead. The ragged splat that tears loose from her flesh only subsides within a moment. There is another than the first and another, then another, bleeding deep through the deep flaking jade of the skull. This is beautiful, she thinks, delicious. There is more power in her own claw than at any time since she was a man. The blood has slowed when the initial burst of raw, ringing pain ripples out through her fingers. She hates the sound of them, yet the blood savors it.

The male of the group stares at her in revulsion. Two minutes has passed and they have paused for an instant to assess the distance between them and the looming obelisk that must be in sight. Damaya ignores this warning as she draws her kirkhusa after it, all eagerness and terror; the message is clear: Fall fast.

Whatever that means, they keep fighting.

He can only whimper. They grit their teeth and risk devastation before the knife begins to rip through Damaya can perceive it, jittering and writhing in its grasp.

Detroit with Rays

python run_generation.py --model_type gpt2 --model_name_or_path output --length 1000 --temperature 1.1 --prompt "Detroit with rays"

There is something beyond the boundaries of the valley. Something even I can feel now. The upwelling of something foreign and special around it. Something that has nothing to do with my kind’s presence and its capabilities, everything to do with… something different.

This sensation is not the warmth of your blood or the aches of your muscles, it is the smoothness of stone beneath your fingers and the precision of the sharp glare of a vision. There are welling masses all around me, she thinks as she pushes the pack out of her hands and jerks the rag pebble toward her own face. She flexes both arms and legs to reach to her lips and her jaw and speaks: Damaya squirms and closes her mouth over the sound of it. This isn’t her first brush with the Season. Something’s different now. She squirms more. Yes. She relaxes. Orogenes often do this. They lie down and stare at something for a while. They watch. And maybe there’s a sound there.

This sensation is not the warmth of your blood or the aches of your muscles, it is the smoothness of stone beneath your fingers and the precision of the sharp glare of a vision. There are welling masses all around me, she thinks as she pushes the pack out of her hands and jerks the rag pebble toward her own face. She flexes both arms and legs to reach to her lips and her jaw and speaks: Damaya squirms and closes her mouth over the sound of it. This isn’t her first brush with the Season. Something’s different now. She squirms more. Yes. She relaxes. Orogenes often do this. They lie down and stare at something for a while. They watch. And maybe there’s a sound there.

Damaya considers this. It is the wish for the impossible. Yes. She grows restless. The savanna of the valley has changed, for one thing, and her two-wheeled would probably leave her too tired to walk. But she doesn’t need to move. So she focuses on the stone around her feet. Thinking about something better, farther away—she grabs her weapon, her rock eater, to start down the easy path home again. The urge to run when she’s not here will die soon enough. With more time on the road, she hopes she won’t have to, but when she comes back there’s the warning snap of something powerful behind her—and when she looks around, that something is gone. A tiny little ice island—small, but bigger than any of the others left on the eastern edge of the valley. The pointy end of the island is surrounded by wire and studded with plants and flowers she recognized as perennial and yet somehow developing, just fine already. But she can’t feel their origins at first. They would’ve made her miss them. They’re everywhere, fading, dying before her gaze ever reaches them. Corundum at the mouth of the ridge, child below under a sky that is cut by cloud shadow, young as a youth, already reverting to ragged obsidian roots—
but she doesn’t see them. She looks behind her, behind Syen, and only there can she see tiny dots of sapphire, surrounded by mountains changing color as frost seeps in from the north and pulls up from beneath them; she’s almost at the cliff edge, when suddenly she sees. There is a small fragment of the northern edge of the island, like a white speck in the water. Then she sees.

Closer.

At once she feels it under her skin and squeezes it tighter, down, because the ache feels useless to her, as if she’s going to burst. All Innon told Damaya not to move, always tell her to move. She relaxes into it. “Shit.” She grunts, uncontrollably, in spite of the comm’s shaking; the message creaks a little as she does, finally settling.

“Do.” Alabaster sounds equally skeptical, even friendly.

“Stop being an orogene and be a good orogene,” he mutters.

Inspiration + References

We were inspired by a number of people who have created much more impressive work in this field:

• Mass Driver’s type specimen for Nichrome
• Allison Parish’s computational language work, in particular using Python to de- and re-construct text
• Robin Sloan’s 2017 Eyeo Talk Writing With The Machine
• Nick Monfort’s Digital Poems and Interactive Fiction
• Priyanka Divedi’s Deep-Learning project, specifically GPT2-HarryPotter-Training