The Ghost in the AI: Rediscovering Linguistics

We often talk about Large Language Models (LLMs) like GPT-4 and Llama in hushed, almost mythical tones. We call them “black boxes”—vast, inscrutable networks that somehow, magically, learn to write poetry, debug code, and explain quantum physics from a sea of data. The prevailing image is one of an alien intelligence, inventing its own rules for language from scratch.

But what if that’s not the whole story? What if, buried deep within the silicon circuits and complex algorithms, these models aren’t just inventing? What if they are, in fact, rediscovering? In their monumental effort to master human language, AI models are independently stumbling upon the very same fundamental principles that linguists have been meticulously cataloging for centuries. The ghost in the AI, it turns out, might just be the ghost of Ferdinand de Saussure, Noam Chomsky, and every linguist who ever diagrammed a sentence.

This isn’t just a philosophical fancy. We can see it happening in one of the most fundamental processes of modern NLP: tokenization.

The AI’s Dilemma: What Is a “Word”?

Before an AI can process language, it must first break a sentence into pieces it can understand. These pieces are called “tokens.” The most intuitive approach seems obvious: just use words as tokens. For the sentence “The cat sat on the mat,” the tokens would be ["The", "cat", "sat", "on", "the", "mat"]. Simple, right?

But this falls apart quickly. What happens when the model encounters a word it has never seen before, like “glob-florp-ing”? Or a common typo like “happyness”? Or a new compound word like “cyber-resilience”? If the model’s vocabulary is a fixed list of known words, every new or misspelled word becomes an “out-of-vocabulary” (OOV) problem, represented by a generic <UNK> (unknown) token. The model loses all the information contained in that word.

The other extreme is to tokenize by character: ['T', 'h', 'e', ' ', 'c', 'a', 't', '...']. This solves the OOV problem—every word can be built from characters. But it creates a new one. The semantic unit of a “word” is lost, and the model has to work much harder to learn that the sequence ‘c’, ‘a’, ‘t’ refers to a furry feline. It’s computationally inefficient and misses the forest for the trees.

The AI’s Solution: Reinventing the Building Blocks

To solve this dilemma, AI researchers developed a clever compromise called subword tokenization. Algorithms like Byte-Pair Encoding (BPE) are used to analyze a massive corpus of text and identify the most frequently occurring sequences of characters. It starts with individual characters and iteratively merges the most common adjacent pairs into a single new token.

Through this statistical process, common words like “the” and “and” might end up as single tokens. But more interestingly, common parts of words also become tokens. For example, the algorithm will quickly learn that “un-“, “re-“, “-ing”, and “-ness” appear frequently across thousands of different words.

So, when a modern LLM sees a word like “unhappiness,” it doesn’t see one unknown blob. Instead, it breaks it down into familiar subword tokens it has seen before:

["un", "happi", "ness"]

This simple-sounding trick is revolutionary. It allows the model to:

• Understand new words: If it encounters “unhappier,” it can recognize the familiar parts ["un", "happi", "er"] and infer its meaning based on its knowledge of those components.
• Be efficient: It keeps the vocabulary size manageable while retaining the ability to represent a virtually infinite number of words.
• Capture meaning: The model learns that the token “un” often reverses the meaning of what follows it, because it sees that pattern statistically across millions of examples.

An Echo from the Lecture Hall: Hello, Morphology!

If you’ve ever taken a Linguistics 101 class, this should be setting off some major bells. What the AI has computationally and statistically “discovered” is a core principle of linguistics: morphology.

Morphology is the study of the internal structure of words. It posits that words are not indivisible atoms of language but are constructed from smaller units of meaning called morphemes. A morpheme is the smallest unit that carries meaning.

Let’s look at “unhappiness” again, but this time through the eyes of a linguist:

• un-: A bound morpheme (a prefix) meaning “not.”
• happy: A free morpheme (a root word) that can stand on its own.
• -ness: A bound morpheme (a suffix) that converts an adjective into a noun.

A linguist analyzes the word as a construction: un- + happy + -ness.

The parallel is staggering. The AI’s subword tokens, derived purely from statistical frequency, are a computational approximation of morphemes. The model never read a linguistics textbook. No one programmed it with the rules of prefixes and suffixes. It simply deduced, by analyzing patterns in the data, that the most efficient way to understand language is to break words down into their meaningful components. It rediscovered morphology from first principles.

Is It All a Rediscovery?

This phenomenon isn’t just limited to morphology. We’re seeing similar rediscoveries across the linguistic spectrum.

Syntax: Researchers have found that the internal “attention mechanisms” of Transformer models (the architecture behind GPT) seem to learn syntactic relationships. When processing a sentence, the model learns to pay more “attention” to grammatically related words. A verb pays attention to its subject; a pronoun pays attention to its antecedent. Without ever being taught a grammar tree, the model generates an internal representation that strongly mirrors one.

Semantics: The way LLMs represent words as vectors in a high-dimensional space (word embeddings) has been shown to capture complex semantic relationships. The famous example is that the vector for “King” minus the vector for “Man” plus the vector for “Woman” results in a vector very close to that of “Queen.” This shows the model has learned relationships of gender and royalty, a rediscovery of semantic features.

The Ghost is a Guide

Calling LLMs “black boxes” is both accurate and misleading. While their inner workings are dizzyingly complex, their output is not random magic. They are powerful pattern-matching engines, and the patterns in human language were put there by us. The structure is inherent in the data.

The ghost in the AI is the ghost of our own linguistic structure, reflected back at us. An AI, given enough data and processing power, will inevitably converge on the most efficient representations of language—and it turns out those representations are the very morphemes, syntactic dependencies, and semantic relations that linguists have been studying all along.

This is an incredibly exciting validation for the field of linguistics. It confirms that the structures we’ve identified are not just academic constructs but are fundamental to how language works. For AI, it offers a tantalizing thought: perhaps looking back at the centuries of human-generated knowledge in linguistics isn’t a step backward, but a shortcut to building more intelligent, interpretable, and efficient models. The ghost in the machine isn’t something to be exorcised; it’s a guide we should start listening to.