Relational AI: Designing Agents for Human Systems
A framework for designing AI agents that navigate the messy reality of human meaning, trust, and attention
TL;DR: As we move from linear-task automation to integrating AI agents into the messy domain of human meaning and decision-making, we need a new framework for thinking about these agents' role. I propose the relational view where AI agents are participants in human systems, not replacements. This view acknowledges that human meaning is contextual, emergent, and built on trust. It means building agents that earn trust gradually, adapt their participation style to context, and continuously update their beliefs about their environment. The relational view offers a path toward AI agents that augment the collective intelligence rather than awkwardly attempting to replace it.
There's a notion often shared by AI tool builders that I call the god view. The god view says that an organization is nothing more than a collection of tasks. AI can already do some of these tasks like software engineering, copywriting, analyzing data; it's inevitable that AI's powers will encompass the full scope of what humans can do collaboratively - how we make decisions, allocate resources, create new ideas. Our goal is to build AI agents that side-step the messiness and inefficiencies of human systems entirely.
The god view rests on assumptions that I think are misguided. Are organizations really just a set of tasks to be automated? Will AI really have the context it needs to understand what's worth doing in the first place? Can the messiness and inefficiency of human systems sometimes be a more of feature than a bug? Is collective intelligence just about compute?
The god view sits in contrast to what I call the relational view. Instead of framing AI as a cognition that replaces humans, the relational view views AI as a cognition that lives in relationship to humans.
Intelligence is relational. The relationship of neurons to the brain, of ants to the ant colony, of human brains to an organization. Intelligence happens when limited perspectives come together, and something emerges through relationship that is bigger than the sum of its parts.
As we embed AI agents into human systems, it's no longer just humans that need to live in relationship, it is humans and AI agents. Like any relationship, these relationships will be defined by how we share meaning, build trust, and direct attention, or care, to each other.
While large swathes of human collaboration can and will be automated, the core of human systems - how we create meaning, tell stories, and make decisions - does not lend itself to linear AI automation. AI has an important role to play in these processes as well, but to understand what this role looks like, we must think of the AI as a participant in a larger human system.
This post is an exploration of why the relational view is the appropriate lens for thinking about AI agents in human systems. It also presents design principles for what the relational view might look like in practice. These design principles include:
Trust: How an agent gradually builds intimacy and interfaces with human accountability structures.
Sensitivity: How an agent adapts its it’s participation based on the situational context.
Curiosity: How an agent builds and updates a provisional model of its environment with feedback from humans.
The relational view is a culmination of my experience building Ize, a collective intelligence platform, and observing how AI is being approached in collective intelligence tools broadly.
Thinking relationally
Funnels vs spaces
AI excels at well-defined tasks where we give it a lot of specific context on what good and bad mean. AI startups will certainly continue to find well-defined linear funnel problems to solve, massively disrupting the labor force in the process.
But most of our individual and collective lives do not fit the criteria of linear well-defined problems. Instead, they are non-linear problem spaces where you cannot define the problem, let alone find a "correct" solution. These non-linear problem spaces are where we create meaning, value, and purpose.
Even what seems to be purely technical problems are, at their core, problems of meaning, value, and purpose. These problems of meaning are not vague orthogonal considerations, but problems that deeply inform the technical requirements.
Building a rocket to Mars is a question about why go to Mars in the first place and what would be an acceptable quality of life for human passengers. Building a product feature is about how a team values privacy and accessibility, the product manager's narrative about where the industry is moving, and the story the founder wants to tell to investors.
In other words, finding the right technical problems to solve in the first place requires diving into the messy, non-linear muck of human systems.
Our social reality has a surprising amount of detail. Meaning is embedded in our emotions and lived experience, status hierarchies, implicit obligations, body language, gossip etc. Some of this will become directly legible to an AI, but most of it won't.
AI tools need some way of asking questions of human systems to better understand what actually matters.
Legibility requires trust
The god view would say that meaning can become legible with enough data. If an AI gets access to the chat logs, our video calls, our personal stream of audio/visuals from our surroundings, and our Neuralink implant, it will be able to fully parse human meaning and stuff it into LLM's context window.
For now, I'll put aside the issue of if it's technically possible for an AI to collect and parse the rich data of our personal and social lives. Even if this were possible (I don't think it is), a given AI is not going to have access to all this data it would need to parse meaning. Even super-intelligent AGI will only have partial understanding.
Disclosure and intimacy require trust. Trust is built progressively and can be revoked at any time. Trust is contextual. For example:
Most organizations have email / Slack retention policies to protect against both cybersecurity and legal risk.
Conversations are often moved offline if they are perceived to be too risky. AI note-taking bots (e.g. Fireflies, Otter) are routinely kicked out of video calls when a conversation is meant to be off-the-record
Individuals are increasingly wary of sharing personal details in tools like ChatGPT.
An AI agent finds itself in a position like any other member of a group; The trust it has in a human system is limited, contextual, and earned/lost over time. Trust is delegated.
The issue of trust becomes much more important when we consider how agents are given permission to take action in human systems. Action could mean making payments, writing tickets, merging code, etc. For agents to take action in the real world, their actions need to be tied back to human accountability structures. This could mean making a person accountable for an AI agent or it could mean building systems that require agents to get buy-in from stakeholders before taking action.
The conversational nature of social reality
The god view might counter an AI may still be able to piece together meaning despite not having all the information. After all, any human perspective on meaning will also be limited, so couldn't an AI do a better job?
Over the past few years, there have been many experiments on exactly this question. Builders have created tools to find points of agreement, surface minority views, de-escalate tensions, mediate disputes, persuade, map arguments, and negotiate.
These tools share a common takeaway - AI can supplement and improve collective sensemaking, but they cannot replace humans' role in guiding sensemaking.
Some of the issues with replacing human facilitators are technical - limited context windows, bias in LLM training data, degeneration of thought, internal inconsistency, etc. It's possible that these are temporary technical limitations that will be solved with in the future.
But other problems are more fundamental, and will not be solved with new tech:
Meaning is emergent, dynamic, and non-linear: Consensus is not an output but an iterative, non-linear process. Stated preferences are not the same as revealed preferences, and even our stated preferences change day-to-day. Shared meanings happen through an endless cycle of creating provisional shared collective understandings and then having those understandings bump up against messy reality. This cycle is intrinsically messy and spans the domains of the digital and physical.
Meaning is contextual: Synthesizing a group's input reduces information, and how you choose to reduce that information depends on your goal. Is your goal for a process to be expedient, to feel democratic, to reduce group tension, to surface minority views, to prioritize authoritative views? These goals require different approaches to working with AI, and the goal itself is determined in the fuzzy domain of human meaning.
Meaning is embodied: While LLMs can approximate human behavior patterns, they lack actual beliefs or stakes. Making these beliefs/stakes of human participants legible to an AI requires a container of psychological safety. This container is not simply a survey form that the AI ingests; this container is created by human connection and presence. The trust and cohesion built through deliberation—what facilitators call "feeling together"— is hard to replicate with a bot.
As Camille Canon put it- "With multiple layers of [AI-powered] decision-making that play into the middle, we ironically get further and further away from the actual truth of the group's thoughts and frustrations."
These limitations reveal a fundamental truth that philosophers have grappled with for centuries - meaning is slippery, contextual and emergent. It is not simply data to be ingested - it is something that is discovered through dialogue with our world. David Whyte calls this the "conversational nature of reality". David Chapman calls it meaningness. Buddhists call it emptiness. Neuroscientists describe this in terms of the Bayesian brain and active inference.
AI agents have an important role to play in how humans and AI agents arrive at shared understanding. Its role is not to merely take in data and spit out an answer; its role is as a participant in ongoing conversation.
This participant has special skills. It's great at synthesizing text, finding points of agreement, highlighting tensions in a conversation, suggesting new pathways the conversation can go to move forward. Though this participant must also be humble because it knows there's so much context it doesn't have. It's curious about what it doesn't know, it asks humans' perspective on what tool is called for in that situation, and makes room for messy, but generative, human process.
(Human) attention is all you need
The most critical bottleneck AI agents have when participating in human systems is human attention.
As we've already discussed, agents need human attention to get information from humans to make contact with the real world. It must ask a lot of questions. It needs to get the right input from the right people at the right time. If it fails to do this, it will continue to have partial and, likely, not helpful perspective to offer.
Many of the AI collective sensemaking tools impose a high attentional cost. Participants need to log into a tool and answer a potentially long series of questions. This high attentional cost is not always viable or appropriate. Instead the situation may call for a lighter touch approach like a pairwise ranking session. The central goal of Ize was optimizing collective process for limited human attention - you can read about its approach here.
Agents also need human attention for a group to internalize the results of a deliberative process. It's not enough for an agent to have the right answer. This answer needs to be digested by the group. LLM's powers of synthesis and generating information can reduce the cognitive load below the threshold at which individual and collective learning happens. So in some cases, we actually want an LLM to take more of a backseat role so that humans can put in the work that leads to learning.
So yet again, the appropriate approach depends on context. This context is subtle, and the agent will need to stay in dialogue with humans to understand how and when it should participate.
Relational design principles
The following is a loose framework for the relational approach to agent design. Some elements of this framework are more appropriate in certain contexts than others.
Trust
Trust determines how an agent gets access to information and permission to take action.
Transparent, delegated permissions: Agents should advertise what data they have access to and what actions they can take. Agent permissions should be tied back to human accountability structures.
While MCP can define an agent’s permissions, these permissions are generally only legible to the person who created that bot. Ideally, we'd like permissions of an agent to be legible to other collaborators.
Hats Protocol team is doing interesting experiments here to tie agent permissions to on-chain organizational roles.
The pattern of "coding agent creates PR ➡️ humans approve in Github" could be generalized more broadly to any kind of organizational task that requires. Ize, my last project, sought to be a generalized framework for exactly this use case.
Gradually increasing intimacy: AI agents should build trust gradually, by asking small asks to a user and inviting the user to disclose more as they feel comfortable.
Metagov's KOI Pond project approaches this by structuring individual/collective knowledge as a vector knowledge graph. KOI can set boundaries that define how parts of this knowledge graph can be accessed by agents of other individuals and organizations.
new.computer's Dot, an AI journaling app, does a good job here.
In an organizational context, the UX here might be something more like inviting an agent to be part of discussion. The inverse UX, kicking an agent out of a video call (e.g. Firefly) has the same result but feels intrusive - like a boundary has been violated and now it's up to individuals to enforce their boundaries.
Private data and private inference: In some contexts (e.g. governments, activist groups), private data and inference are table-stakes to include an agent as a participant in a group.
Self-hosting a model is now easier than ever with tools like Hugging Face and lightweight open-source models like Qwen and Llama.
Venice.ai is building infrastructure for private inference that doesn't require self-hosting.
A lighter touch method for preserving privacy is simply an anonymous response form that can be ingested by an agent. Many collective intelligence tools already have this feature.
Sensitivity
Sensitivity describes how an agent adapts its it’s participation based on the situational context. This context may include the immediate goal, available time and attention, roles of the participants, and human feedback.
Offer different modes of participation for different contexts: Agent has specialized tools tailored to different kinds of collective sensemaking problems. These tools might include specialized UX and models for Polis-style clustering analysis, decision facilitation, open-ended idea exploration, etc. The agent should monitor their available context and ask for human feedback to see which tool is called for in that situation.
I’m imagining an ecosystem of collective sensemaking tools that an agent can call via MCP.
Harmonica is building an AI-mediated facilitation across diverse facilitation modalities.
There's a wide-open design space of other forms of group process that could be integrated into this design kit - e.g. pairwise ranking, latent space exploration (1, 2), Miro-style canvases, quadratic voting, etc.
Create container for human connection, agency, and creativity: Agent proactively asks for feedback from humans about how it can best participate based on the goals of the group. Sometimes this means stepping back entirely and let humans work out a problem on their own.
Trained facilitators already selectively use tools like Polis, Decidim, Talk to the City as part of a larger human facilitated process. These facilitators know when to use these tools and when to pull back and see what emerges naturally through informal discussion and presence. I'd like to see agents be able to exercise the same judgment.
Canvas-based UI is a natural fit for non-linear, creative group process. Miro does a good job incorporating AI into open canvases, though canvases can be hard to parse and aren't always the right form factor. I'd love an agent that could selectively call on and participate in a tool like Miro if that's the right tool for the situation.
Curiosity
Curiosity describes how an agent builds a provisional model of its environment, but proactively works with humans to keep that model in touch with reality.
Build a provisional world model: Agent proactively builds a provisional model of individuals, groups, and tools it comes into contact with. It attempts to understand beliefs, stakes, relationships, and goals. It continuously updates its understanding as new information becomes available.
This generation of transformer-based AI doesn't have a world model in the same way humans do, but we can simulate it with RAG. The goal here isn't to just store memories of past interactions; instead the agent should be creating its own inferences about how it thinks a human system works and then to update those inferences continuously.
Proactively ask humans to fill in gaps in context: AI acknowledges the limits of its understanding, and proactively asks humans about relevant context that it can use to update its world model and decide on the appropriate action to take next.
Most AI tools already do some version of this, but these tools are generally only being used by a single user at a time. In a collaborative setting, the UX challenge is more difficult. Agents need to take extra care that outputs are presented as provisional. They may also want to solicit feedback from multiple people before deciding on any given course of action.
I like how Granola doesn't just spit out a meeting transcript, but integrates your own notes that better capture context and salience.
Summary
The relational view recognizes that AI agents are not omniscient replacements for human intelligence, but rather participants in human systems with inherently limited perspectives. This matters because it fundamentally changes how we design AI tools - instead of building systems that attempt to bypass human complexity, we build systems that earn trust gradually, adapt sensitively to context, and remain curious about what they don't know. By embracing these limitations rather than ignoring them, we can create AI agents that genuinely augment human collaboration rather than awkwardly attempting to replace it.
If this vision of relational AI resonates with you, feel free to reach out.