Modeling Meaning with the Data Kernel
Intro to Computational Semiotics, Part 5
In this series, I break down “Intro to Computational Semiotics” lectures by Brandon Duderstadt (calco_io on X). You can view the recorded lecture here.
In the last article in this series, we looked at this visualization of a “Tweets from Members of US Congress” dataset.
In his lecture, Brandon illustrates that when you look at how this visualization evolves over time, you see semantic regions of discourse emerging and decaying. He says “this is something that we as computational semioticians might want to model.”
Recall Brandon’s quad chart from Part Four. One axis is macro/micro, and the other is functional/structural.
He likens the functional dimension to the ecology of a river, the flow of language itself; the structural is like the terrain through which the language flows (like the mountain down which the river may flow).
The micro dimension looks at individuals (like the individual congresspeople whose tweets make up the dataset above). Macro looks at ensembles (the aggregate properties of the whole map).
This article focuses on the micro. On the micro-functional side, how do we measure the perspective of a single language-producing agent? On the micro-structural side, how does the structure change those perspectives?
The Data Kernel
Take an encyclopedia and train two BERT models on slightly different subsets:
BERT A (ablated): remove every article about plants.
BERT B (baseline): uniformly downsample so the two are token-equivalent.
This gives us one model with a conceptual gap compared to the other. How do their representations differ?
The problem is that you can’t compare them directly. Because they have different initializations and data, their representational spaces are not naively comparable even if they have the same tensor shape.
You can compare relationships, though. If you feed both models the same paired queries (e.g. “The fern is found in...”), you can map how documents cluster together into nearest-neighbor graphs.
Using random-graph theory, the Data Kernel (a term coined by Brandon) calculates the distance between these distinct graphs.
Plot the differences, and you unsurprisingly get concentration around the plant articles. But the surprising result is that they also heavily disrupt the models' representations of animals and natural places. So ablating one concept changes how the model represents a whole range of other concepts. Brandon calls this semantic bleed. The relationship between training data and representation is not linear.
Each pairwise comparison yields a single distance between two models. If you run this over many models and fill in a full distance matrix (every model against every other) then you can study whole populations. When models are trained on similar data mixtures, they cluster together into a geometry that lets you predict downstream performance based on the model’s location in this space.
The Surrogate Data Kernel
The problem with the method above is that it requires “white box” access to the model vectors. You can’t get this with very large models. BERT models also produce one vector per token, which is not true of generative models. There is variance in the length of the output.
This complicates analysis of these vector spaces because we are taking the average size of the output, so our distances may pick up differences in length rather than differences in meaning. The method is also computationally slow and expensive, because it involves computing many SVDs.
Enter the Surrogate Data Kernel. Instead of comparing internal representations, evaluate the actual outputs. Pass them through an embedder (the surrogate) like Nomic Embed. Now both models' outputs are n × d tensors in a single vector space, and they’re directly comparable.
(The trade-off is that you lose the “intrinsic” representation of the model; you’re flowing everything through another model with its own “perspective.” Embedders also fail in weird ways, like how Nomic Embed may decide Homer is closer to Freud than to The Iliad.)
The distance between two models is now just the Frobenius norm of the difference of their Nomic Embed spaces. If you run MDS on that matrix, you get the Data Kernel Perspective Space (DKPS).
Every dot is a model, and close dots mean similar answers according to Nomic Embed.
From here, populations emerge. Take a couple of populations of models and once again train them on different categories — red: animals, green: plants, blue: natural places. Ask about those topics, embed the outputs, and plot. They will separate cleanly. Furthermore, if you ask them about something orthogonal, like automobiles, they will still separate. This is the semantic bleed property: a model not getting trained on plants will subtly change how it talks about oil, therefore engines, therefore cars. This means we don’t need a priori knowledge of the differences between models to discern which ones are close to each other.
Perspectives in a Network
Moving into the micro-structural, we ask how these senses of meaning might actually change as models interact.
This is Helm, Duderstadt, Park, and Priebe's “Tracking the perspectives of interacting language models” (EMNLP 2024). The setup is to put a collection of models in various communication configurations. You can have “everyone talks to everyone,” or split them into red/blue teams. You might have one bad actor among friendly models.
The gist is to let them play a game with each other, where they can ask each other questions and update their training data on the answers. In this way the models keep learning as they communicate. Updating a model over time is called continual learning, which is a hard, unsolved, very hot problem right now.
Two update mechanisms are:
LoRA (low-rank adaptation): rather than retraining all weights, train a small low-rank adapter added to them. No inference overhead and few parameters, so it can learn from little data.
RAG (retrieval-augmented generation): No weights get touched at all. Just retrieve the relevant material from a knowledge base and paste it into the input.
The original “Tracking Perspectives” paper used LoRA and has since been replicated with RAG, and it doesn’t much matter which. Just give each model a different starting perspective and then let them interact and update.
Polarization and adversaries
In these communication networks, when is consensus not reached? It depends on whether information is flowing between components in the network.
Say you assign each model to class 0 or 1 and then allow only intra-class talk. The graph splits into two disconnected clusters, and you get polarization (i.e. meaning in one component drifts entirely away from the other). If you add enough cross-links, this effect vanishes. More sparsely connected components have a higher chance of polarization, which is the echo-chamber effect.
You can also get an “adversary,” which is a model that pushes its own view and never budges. It’s a fixed point in model space (“like your uncle at Thanksgiving”). Now if you take a connected “in-group” of ~five, plus one adversary, and then vary how many targets the adversary can talk to, you get the following:
Zero targets: the group converges to consensus with the adversary stuck off to the side.
One target: that target model oscillates between the group and the adversary, which drags the consensus toward the adversary’s position.
Two targets: the adversary begins pulling the whole group.
All: they simply converge to the adversary.
With access to only a minority of nodes, an adversary can bend the whole system and subvert an ostensibly democratic process. (note: “time” here = communication rounds, where one question-and-update is one step.)
Can we say anything rigorous?
This all may be useful for qualitatively understanding behavior in a system, but what can we do quantitatively? Is DKPS consistent? Can we say anything with a level of theoretical certainty?
It turns out that, yes, a model's position in this space is predictive of its future behavior. The follow-up paper to “Tracking Perspectives” makes this case formally — “Statistical inference on black-box generative models in the data kernel perspective space” (Helm et al., Findings of ACL 2025).
If we map a new model based on its outputs, we can predict how it will respond to subsequent prompts. Given a large enough query set, mapping models by their DKPS neighbors is a better predictor of whether a model will produce “toxic or biased output” than if you use a “ground truth model genealogy.” In other words, “DKPS is capturing an essence of the function of these models that is not present in the ground truth model taxonomy.”
Digital twins
Do these model dynamics actually approximate real human dynamics? If we simulated a model learning computational semiotics by reading a Substack article, would it be like you reading this article to learn computational semiotics right now?
This is where the idea of the digital twin becomes useful. If you distill it to its essentials, it is a digital system that (1) mimics a real system, (2) updates dynamically, (3) predicts the real system, and (4) makes predictions that are actually useful.
Brandon built a digital twin of the U.S. Congress. He collected members’ tweets and trained models on them, then tried to predict their voting behavior.
“It sort of works.” You can tell via detectability analysis. If you have synthetic outputs from the twin of a Congress member + never-before-seen outputs from that same real member, could a classifier tell them apart?
If you prompt it naively (e.g. “write a tweet from this congressperson”), the two are very easy to separate, but if you add a specific system prompt then they can move closer. Next, add in a RAG database of the member’s real tweets, and they become nearly coincident. They get almost as close as two samples of the real tweets sit from each other. This indicates that more sophisticated cognitive architectures like system prompts and retrieval measurably improve our models of social systems.
Generative agents
The Generative Agents paper (Bernstein Lab, Stanford; Joon Sung Park) is the best example of a successful digital twin system of human social behavior. The setup of the paper: a virtual village of model agents that move around, talk to one another, and go about their days.
Each agent is initialized with a system prompt, like “John Lin is a pharmacy shopkeeper who loves helping people, loves his family, thinks Sam Moore is kind,” etc. which is the basis for how it will plan its day. Then it runs a loop where perception from the environment flows into a memory stream, and it can retrieve from the memory stream in order to plan, reflect, and act. The continuous process of identity formation produces emergent behavior like agents organizing a Valentine's Day party. You can watch the narrative spread as agents plan to attend, offer to help, invite others, and check back in with Isabella about the status.
Preview of Part Six (the Series Finale)
We’ve just covered the whole of the micro toolkit along the functional-structural axis.
Functionally, we can measure a model’s perspective, compare perspectives across models, and build digital twins to make predictions. Structurally, we can track how model perspectives shift through interaction and build systems of agents that produce emergent behaviors.
So, can we unify all of this into something that explains the macro as well?
The final part in the series turns to the macro row and asks whether the local ideas can be generalized upward, so that as you run them longer or add more agents, they converge to macro phenomena. Stay tuned!
So Long,
Marianne