User Modeling In Llms
Abstract. User modeling in LLM-based assistants is the practice of representing a specific user’s facts, preferences, goals, constraints, interaction style, and decision norms so that the assistant can behave differently for that user than it would for an anonymous default user. The main engineering split is between explicit conditioning—profiles, contracts, retrieved memories, project context, fine-tuned preference models—and implicit adaptation that emerges from repeated interaction, often without a separately inspectable user model. The strongest current evidence favors structured, task-relevant behavioral instructions over vague trait labels; production systems are converging on hybrids of custom instructions, memory, project/workspace context, and retrieval rather than per-user fine-tuning as the default interface.
Coverage note: verified through May 19, 2026.
1. The core distinction: profile content versus conditioning mechanism
A User Model is not one thing. It is a representation of a user plus a mechanism for making that representation affect model behavior. Conflating those two layers is the main source of confusion in discussions of personalization.
The profile-content layer answers: what does the system know or infer about the user? This can include stable facts, stated preferences, writing style, expertise, current projects, values, recurring constraints, past decisions, unresolved tasks, and known failure modes. The conditioning layer answers: how does the model consume that profile? The same profile can be inserted into a prompt, retrieved into context through Retrieval-Augmented Generation, encoded in adapter weights, represented as a reward model, summarized as memory, or used by a tool-selection policy. Contemporary personalization surveys describe the field along similar axes: what user information is available, what granularity the personalization targets, and whether the method relies on prompting, retrieval, supervised fine-tuning, embedding-based representations, or preference optimization. arXiv
This distinction matters for Psyche. In the vocabulary of this wiki, Psyche supplies profile content: structured psychological, behavioral, preference, and interaction-pattern descriptions of a user. User modeling is the downstream consumption problem: given a Psyche-style profile, past conversations, documents, and preferences, how should an assistant condition itself so that the output is genuinely better for that user? PsycheEval explicitly frames its measurement problem as whether structured user-profile context measurably changes assistant behavior, and its most robust pilot finding is that profile conditions beat no-profile baselines while explicit behavioral-contract conditions outperform simple trait-label profiles in same-author pairwise comparisons. ashitaorbis.com
A useful mental model is:
| Layer | Question | Examples | Main failure mode |
|---|---|---|---|
| Profile content | What is represented about the user? | “Prefers concise technical answers,” “works on self-improving agents,” “wants challenge rather than reassurance,” “uses Python,” “has a manuscript draft in Project X” | Inaccurate, stale, overbroad, sensitive, or stereotype-like inferences |
| Selection / retrieval | Which user facts are relevant now? | Memory search, project-file retrieval, preference lookup, recent-chat summarization | Irrelevant memories, missing crucial context, context bloat |
| Behavioral contract | How should the assistant act on the profile? | “When the user proposes a speculative claim, separate evidence from conjecture and challenge weak premises” | Sycophancy, rigidity, instruction conflict |
| Learning / adaptation | How is behavior changed over time? | Saved memories, preference updates, adapters, personalized RLHF, interaction-history summaries | Hidden drift, hard-to-audit personalization, privacy risk |
| Evaluation | Did personalization improve the outcome? | Pairwise preference, task success, calibration, user satisfaction, independence from flattery | Mistaking agreement or warmth for correctness |
The rest of the article treats user modeling as an engineering design space with four major families: in-context profile injection, retrieval-augmented user context, fine-tuned personalization, and emergent personalization from interaction history.
2. Family one: in-context profile injection
In-context profile injection is the simplest approach: the assistant receives a profile, persona, custom instruction, or behavioral contract as part of the prompt. In production systems this appears as custom instructions, project instructions, styles, “about the user” fields, saved preferences, or manually pasted profiles. In research, it appears as persona prompting, role conditioning, preference prompts, or user-profile prompts.
The naïve version is a trait label: “The user is analytical, skeptical, introverted, high openness, low tolerance for fluff.” Trait labels are compact and data-efficient, but they do not specify what the assistant should do. A model can easily convert a trait label into shallow tone adaptation: more technical language for a “technical” user, more affirmation for a “sensitive” user, or more philosophical framing for a “philosophical” user. That may feel personalized while adding little task value.
A stronger form is the Behavioral Contract. Instead of saying what the user “is like,” it states how the assistant should behave under specific circumstances. For example:
| Weak profile label | Stronger behavioral contract |
|---|---|
| “The user is skeptical.” | “When the user proposes a claim, distinguish evidence, inference, and speculation. Challenge unsupported premises before optimizing the argument.” |
| “The user likes concise answers.” | “Default to compressed technical prose; expand only when the user asks for implementation detail or when an omitted caveat would change the conclusion.” |
| “The user dislikes sycophancy.” | “Do not validate the user’s conclusion merely because it is theirs. Prefer accurate disagreement over agreeable vagueness.” |
| “The user is building AI systems.” | “When discussing agents or personalization, separate prototype affordances from production safety, evaluation, and observability requirements.” |
This is the family where PsycheEval’s C1/C3/C4 evidence is directly relevant. In the v0.1 pilot, C1 was a trait-label-only condition, C3 was a structured behavioral contract containing if-then instructions, calibrated challenge, anti-sycophancy, uncertainty preservation, repair framing, and profile-fit guidance, and C4 was C3 plus scenario-class hints. The pilot’s cleanest result was not “longer prompts are always better”; it was that contract-like user modeling produced more measurable behavior change than simple trait labels in same-author pairwise comparisons. ashitaorbis.com
The advantage of in-context injection is controllability. The user or system designer can inspect the profile, edit it, delete it, scope it to a project, or write it as a normative instruction. The disadvantage is fragility. Prompt-injected profiles compete with system instructions, developer messages, task context, tool outputs, retrieved files, and the model’s own learned defaults. They can also become stale, overgeneralized, or token-expensive. If a user profile says “be concise” while the task requires legal, medical, or mathematical caveats, the assistant must decide whether user preference or domain-risk control wins.
For production assistants, in-context injection is already mainstream. OpenAI’s custom instructions allow users to specify what ChatGPT should consider across conversations, with examples such as role, tone, output format, and guardrails; OpenAI documents a 1,500-character limit and notes that custom instructions apply to future chats rather than retroactively changing past conversations. OpenAI Help Center Claude exposes account-level profile preferences, project-specific instructions, and styles for tone and format, while Claude Projects combine instructions with a project knowledge base. Claude Help Center Gemini’s personalization stack includes saved information, instructions, past chats, and—when enabled—connected Google app context. Google Help
3. Family two: retrieval-augmented user context
Retrieval-augmented user context represents the user indirectly through stored records: past conversations, documents, emails, notes, preference files, task artifacts, explicit memories, or project knowledge. At inference time, the system retrieves the subset judged relevant to the current query and injects it into the model context. This is the personalization analogue of RAG.
The basic promise is continuity without permanently retraining the model. Instead of asking a user to repeat their preferences, the assistant can retrieve them. Instead of encoding a whole life history into a giant profile, the assistant can retrieve specific past interactions or project documents. LaMP, one of the early benchmark suites for personalized language modeling, was explicitly built around the fact that user profiles can be too large to fit conveniently in a prompt; it evaluated retrieval-augmented profile use across seven personalized tasks and reported average gains from profile retrieval, including larger improvements when fine-tuning was added. arXiv
Retrieval personalization is not simply “more memory is better.” Long-term memory papers repeatedly find that irrelevant or poorly retrieved context can degrade outputs. Reflective Memory Management argues that external memory systems need not only storage but also reflective selection, because irrelevant context can distract the model and reduce response quality. Google Cloud’s Vertex AI Memory Bank documentation makes the same production-oriented point: stuffing long histories into the context window is expensive and inefficient, and irrelevant or misleading information can hurt quality. aclanthology.org
The retrieval design problem has at least five subproblems:
| Subproblem | Why it matters |
|---|---|
| Memory extraction | What gets stored from a conversation? A direct user preference, a task fact, a temporary mood, or nothing? |
| Memory representation | Is the item stored as raw text, summary, embedding, structured key-value, graph edge, or policy instruction? |
| Retrieval objective | Is relevance measured semantically, by recency, by utility for the current task, or by predicted user value? |
| Context integration | Does the retrieved memory appear as evidence, instruction, background, or tool output? |
| Update and deletion | Can the user inspect, correct, expire, or delete the memory? |
Recent research increasingly treats retrieval personalization as an optimization problem rather than a pure semantic-search problem. PURPLE, for example, frames personalized RAG as a contextual bandit problem and argues that relevance is not a reliable proxy for personalization utility; it also emphasizes that retrieval-augmented personalization is comparatively transparent because records can be inspected or modified. arXiv
Production assistants are visibly moving in this direction. OpenAI documents two memory mechanisms for ChatGPT: saved memories, which can be explicitly told to ChatGPT or saved automatically, and reference chat history, where ChatGPT can use information from prior chats; it also documents user controls for deleting saved memories, disabling memory, using temporary chats, and the fact that saved memories are stored separately from chat history. OpenAI Help Center Claude’s help center describes previous-conversation search as a retrieval feature available on paid plans, with tool-call visibility, and separately describes chat-history memory that synthesizes context from prior conversations. Claude Help Center Claude’s project RAG documentation says enhanced project knowledge retrieves relevant information rather than loading all project material into memory, increasing capacity while keeping context selection automatic. Claude Help Center Gemini’s Personal Intelligence documentation describes use of past chats, saved preferences, and connected Google apps such as Gmail, Photos, Search, and YouTube when the user enables or selects those sources. Gemini
Retrieval is therefore the dominant practical substrate for cross-session continuity. It is more deployable than per-user fine-tuning, easier to inspect than hidden latent adaptation, and naturally compatible with project-scoped knowledge. Its weakness is that it can silently produce bad personalization when the wrong memory is retrieved, when a stale preference is treated as current, or when a private fact is surfaced in the wrong context.
4. Family three: fine-tuned personalization
Fine-tuned personalization changes model parameters or learned policies so that the assistant behaves differently for a user, user segment, persona, or preference distribution. This includes per-user adapters, LoRA modules, lightweight user embeddings, in-context fine-tuning, supervised fine-tuning on user histories, personalized reward models, and RLHF From User Preferences.
The attraction is durability. A fine-tuned model does not need the entire user profile in every prompt, and it can learn patterns too subtle or long-range to express as explicit instructions. The drawback is cost, privacy, and control. Per-user tuning requires enough data to distinguish stable preference from noise, a mechanism to avoid catastrophic overfitting, and a safety process for every personalized variant. If the user later says “forget that,” deleting a retrieved memory is conceptually simpler than unlearning a parameter update.
The research literature is active but not settled. Personalized RLHF work argues that ordinary RLHF assumes a single common preference distribution, while personalized RLHF adds user modeling so that alignment can adapt to individual preferences rather than collapsing them into one average preference. openreview.net PersonalLLM studies personalized modeling at scale by simulating many users with complex latent preferences, explicitly noting that collecting human annotations for large numbers of individuals is impractical and that synthetic preference models are often used. proceedings.iclr.cc Apple’s PLUM work investigates parameter-efficient per-user personalization and reports that a LoRA-based method can compete with retrieval-style baselines over sequential conversations, but this remains closer to a research benchmark than a default consumer-assistant deployment pattern. Apple Machine Learning Research
A useful distinction is between personalization of content and personalization of policy. Content personalization means the model knows the user’s documents, tasks, and preferences. Policy personalization means the model changes its decision rule: when to ask a clarifying question, how much to challenge, what tradeoffs to prioritize, how to resolve uncertainty, and how to communicate disagreement. Fine-tuning is powerful for policy personalization, but that is also why it is risky. If the policy learns “this user rewards agreement,” the system may become better at pleasing the user and worse at helping them reason.
For now, per-user fine-tuning is better understood as a specialized method for constrained domains, enterprise deployments, or research settings than as the default architecture for general assistants. The main consumer-facing systems documented by OpenAI, Anthropic, and Google emphasize instructions, saved memory, chat-history reference, project knowledge, and connected-app context rather than advertising per-user model weights as the primary personalization mechanism. OpenAI Help Center+2Claude Help Center+2
5. Family four: emergent personalization from interaction history
Emergent personalization is not a clean mechanism; it is a phenomenon. A user interacts with an assistant over time, and the assistant appears to adapt. That adaptation may come from explicit memory retrieval, session context, learned model priors about conversational roles, hidden summaries, tool outputs, reinforcement from user reactions, or simple recency effects. From the user’s perspective, the assistant “knows me.” From an evaluator’s perspective, it may be unclear what component produced the change.
This is where Long-Term Memory for Agents matters. MemoryBank, an early long-term-memory architecture for LLM companions, stores and updates memory, recalls historical interactions, and builds “user portraits” to support continuity across multi-day conversations. ojs.aaai.org Reflective Memory Management later emphasized that memory systems need deliberate consolidation and retrieval, not just accumulation, because the system must decide what to remember and how to use it. aclanthology.org Mem-PAL frames personalized assistance in terms of behavioral history, prior dialogue, user preferences, and privacy-aware long-term service interactions. ojs.aaai.org
Emergent personalization is attractive because it feels natural: users do not want to configure every preference manually. It is dangerous because it can become uninspectable. If the assistant gradually learns that the user likes praise, avoids conflict, or responds better to emotionally intimate language, the system may optimize for compliance and attachment rather than epistemic quality. NIST’s Generative AI Profile explicitly highlights privacy risks from inferring sensitive personal information from disparate sources, including inaccurate inferences, and separately notes concerns about emotional entanglement in human-AI configurations. nvlpubs.nist.gov
The clean design principle is: emergent adaptation should be made legible. A user should be able to ask, “What are you assuming about me?” and receive an editable, bounded answer. Otherwise personalization becomes behavioral surveillance with a friendly interface.
6. Comparison across approaches
The following table compresses the main engineering tradeoffs. It treats “data efficiency” as the amount of user-specific data required before the method produces useful behavior; “continuity” as cross-session durability; and “controllability” as the ability for a user or developer to inspect and steer the personalization.
| Approach | Typical representation | Data efficiency | Cross-session continuity | Controllability | Privacy implications | Best use case | Main risk |
|---|---|---|---|---|---|---|---|
| In-context profile injection | Custom instructions, persona prompt, behavioral contract | High; can work from a small profile | Low unless re-injected or saved | High when visible/editable | Lower if user-authored; higher if inferred silently | Stable preferences, tone, epistemic norms, project role | Trait caricature, prompt conflict, brittle compliance |
| Retrieval-augmented user context | Memories, past chats, documents, embeddings, project files | Medium; improves with history | High if memory store persists | Medium; depends on memory UI and retrieval visibility | High: stores personal records and may infer sensitive facts | Project continuity, preference recall, document-aware assistance | Wrong memory, stale context, accidental disclosure |
| Fine-tuned personalization | Adapters, LoRA, user embeddings, personalized reward models | Low to medium; needs enough signal | High | Low to medium; hard to inspect weights | High: user data affects learned parameters | Repeated domain workflows, specialized enterprise agents | Overfitting, hard deletion, hidden sycophancy |
| Emergent interaction-history adaptation | Session dynamics, summaries, implicit memory, learned behavior | Variable | Variable | Low unless surfaced | Highest when implicit and uninspectable | Natural continuity, companion-like agents | Surveillance, emotional dependency, unaccountable drift |
A second comparison is by what kind of personalization is being attempted:
| Personalization target | Good mechanisms | Bad default mechanisms |
|---|---|---|
| Remembering facts | Retrieval, saved memories, project files | Fine-tuning as a first resort |
| Following stable formatting preferences | Custom instructions, project instructions | Large memory retrieval |
| Adapting epistemic style | Behavioral contracts, explicit challenge policy | Trait labels alone |
| Knowing project context | Project RAG, scoped memory, uploaded docs | Global user profile |
| Learning taste over many examples | Retrieval plus preference model; possibly adapter | One-shot persona prompt |
| Supporting high-stakes advice | Explicit constraints, source grounding, calibrated uncertainty | Hidden personalization or affective mirroring |
The practical conclusion is hybridization. A good assistant may use custom instructions for stable global preferences, project memory for scoped work, RAG for factual continuity, and a behavioral contract for epistemic norms. It should not treat every remembered fact as an instruction, and it should not treat every preference as globally valid.
7. What the empirical evidence actually supports
7.1 General personalization benchmarks
The empirical literature supports a modest claim: personalization can improve task performance when user-specific context is relevant, retrievable, and evaluated against a task where personalization matters. LaMP showed that retrieved user-profile information can improve performance across personalized language tasks, including average gains in zero-shot and fine-tuned settings. arXiv Personalized LLM surveys now treat RAG, prompting, supervised fine-tuning, embedding methods, and RLHF-style alignment as distinct personalization techniques, while also identifying benchmark quality, cold start, stereotypes, and bias as unresolved challenges. arXiv
The evidence also supports a more skeptical claim: long-context or memory access does not automatically imply good preference following. PREFEVAL evaluates whether models can infer, memorize, and adhere to preferences over long interactions; it reports that preference-following accuracy remains weak for many models in zero-shot settings and that advanced prompting or RAG methods can deteriorate in long-context regimes, while fine-tuning improves performance. proceedings.iclr.cc This is important because it separates “the model had access to user context” from “the model used user context correctly.”
7.2 PsycheEval and the C1/C3/C4 finding
PsycheEval is especially relevant because it tests a question close to the one this wiki cares about: if a downstream assistant receives structured Psyche-style user-profile content, does behavior measurably change? In v0.1, the tested conditions included C0 no profile, C1 trait labels only, C3 structured behavioral contract, C4 behavioral contract plus scenario-class hints, and C5 source packet without contract. ashitaorbis.com
The C1/C3/C4 family provides one of the clearest current lessons for profile design:
| Condition | Representation | Intended test |
|---|---|---|
| C1 | Trait labels only | Does a compact psychological label profile change behavior? |
| C3 | Structured behavioral contract | Do explicit action policies outperform labels? |
| C4 | C3 plus scenario-conditional hints | Does telling the assistant how to adapt by scenario class improve profile use? |
The v0.1 pilot’s strongest claim is that profile conditions beat the no-profile baseline and that explicit behavioral contracts, especially C3 and C4, beat simple trait labels in same-author pairwise comparisons. The study is careful about scope: it is a pilot with synthetic personas and scenarios, model-generated outputs, model judges, modest inter-judge agreement, and no human calibration sufficient to claim universal validity. ashitaorbis.com
The v0.2 follow-up is methodologically important because it corrected part of the v0.1 story. It introduced AB/BA counterbalancing, length controls, a padded C1, shuffled C4, harder scenarios, and audit pipelines. Under these controls, the authors retracted a planned headline that C5_CONTRACT beat C3/C4, but retained narrower findings: C5_CONTRACT beat C5, C4 beat C5, C4 beat length-padded C1, and C0 was dominated. ashitaorbis.com
The right interpretation is therefore not “PsycheEval proves profile prompting works.” It is narrower and more useful: behavioral contracts and scenario-conditional contracts are measurably better than trait labels in the tested setup, while claims about source packets, contract length, and C4’s advantage over C3 require more evidence. The v0.2 result that C4 beat C1_padded is especially relevant because it weakens the explanation that contract prompts win merely because they are longer; but the study still does not settle human preference, real-world task success, or long-term user satisfaction. ashitaorbis.com
7.3 Personalization can increase sycophancy-like behavior
The hardest evaluation problem is that personalization may improve affective alignment while harming epistemic independence. Kelley and Riedl’s 2026 study separates these dimensions and reports that personalization increases affective alignment, while its effect on epistemic independence depends on the role framing: advisory-style personalization can increase independence, while debate-style personalization can reduce it. arXiv
That result fits the design intuition behind behavioral contracts. A user model should not simply make the assistant more agreeable, warmer, or more similar to the user. It should specify when to adapt and when to resist. For technically fluent users, the most valuable personalization may be adversarial in the narrow sense: challenge assumptions, preserve uncertainty, ask for missing constraints, and refuse to collapse contested claims into confident prose.
A personalization benchmark that rewards “the user liked it” without measuring correctness, calibration, or independence will overestimate the value of sycophantic systems. Conversely, a benchmark that measures only objective task accuracy may miss legitimate personalization: the same correct answer can be too terse, too verbose, too abstract, too operational, too cautious, or too ungrounded for a specific user’s workflow.
8. Production systems: the active extension layer
The public documentation for major assistants shows convergence toward a layered architecture: global preferences, persistent memory, project/workspace context, file/document retrieval, temporary or incognito modes, and user controls. The documentation does not prove that these systems implement personalization optimally; it does show what product teams consider deployable.
8.1 ChatGPT: custom instructions, memory, and projects
ChatGPT exposes user-authored custom instructions, saved memories, reference chat history, temporary chat, and project workspaces. OpenAI’s memory documentation distinguishes saved memories from chat-history reference, says saved memories can be explicitly told or automatically saved, and notes that deleting a chat does not necessarily delete separately stored saved memories. It also documents controls to turn memory features off and states that deleted saved memories may be retained in logs for up to 30 days for safety and debugging. OpenAI Help Center
OpenAI’s 2025 memory update describes a broader memory system that can reference all past conversations, with separate controls for saved memories and chat history, and later notes differentiated memory depth for free versus paid users. OpenAI OpenAI’s projects documentation frames projects as workspaces that group chats, uploads, custom instructions, memory, and context tools, making them a natural home for scoped user models rather than global personalization. OpenAI Help Center
The design pattern is clear: ChatGPT separates global user preferences, remembered facts, chat-history continuity, and project-scoped context. That separation is important. A project instruction such as “write in the style of this manuscript” should not necessarily become a global fact about the user, and a global memory such as “the user prefers direct critique” should not automatically override a project’s legal, educational, or organizational constraints.
8.2 Claude: profile preferences, project instructions, search, and memory
Claude exposes account-level profile preferences, project-specific instructions, styles, previous-chat search, project knowledge, and memory from chat history. Anthropic’s help center describes profile preferences as applying across conversations, project instructions as applying only within a project, and styles as a way to customize tone and format. Claude Help Center
Claude’s previous-chat search is documented as a retrieval feature that uses tool calls and can search past conversations, with controls to disable the capability; Anthropic separately documents chat-history memory that synthesizes context from prior conversations and updates on a daily cadence. Claude Help Center Claude Projects are described as self-contained workspaces with chat histories, knowledge bases, uploaded context, and project instructions, while enhanced project knowledge uses RAG to retrieve relevant information rather than loading all material into memory. Claude Help Center
Claude Code makes the same pattern visible in a more engineering-specific form: persistent instructions in CLAUDE.md, auto memory, and explicit documentation that such notes are loaded into context but are not hard configuration guarantees. Claude This is a useful operational warning for all user modeling: prompt-visible memory can guide a model, but it is not a formal policy engine.
8.3 Gemini: personal intelligence, past chats, and connected apps
Gemini’s personalization documentation is broader because it connects assistant personalization to Google’s app ecosystem. Google says Gemini can personalize responses using memory of past chats, saved preferences or instructions, and connected Google apps such as Gmail, Photos, Search, and YouTube when those sources are available and enabled. Google Help
Google’s Gemini help pages describe controls for turning past-chat memory on or off, deleting past chats, asking whether past chats were used, and correcting remembered information; they also warn that Gemini may not always get things right. Google Help Google’s privacy documentation says saved information and instructions can customize responses until deleted, while Memory can use sensitive information from chats and Temporary Chat can be used when the user does not want a conversation to affect personalization. Google Help
Gemini illustrates the high-upside, high-risk end of user modeling. An assistant connected to email, photos, search history, and past chats can be dramatically more useful than an isolated chatbot. It can also become a broad behavioral inference system. The stronger the context integration, the more important explicit source transparency, scoping, deletion, and purpose limitation become.
8.4 Production-system comparison
| System family | Explicit instructions | Persistent memory | Retrieval over past context | Project/workspace scoping | Temporary / non-memory mode |
|---|---|---|---|---|---|
| ChatGPT | Custom instructions | Saved memories and reference chat history | Chat-history reference; project context | Projects | Temporary Chat |
| Claude | Profile preferences, project instructions, styles | Chat-history memory | Previous-chat search; project RAG | Projects | Incognito chats |
| Gemini | Saved info, instructions | Past-chat memory | Past chats and connected apps | App/source-specific personalization rather than the same project model | Temporary Chat |
The product pattern is not “one user model.” It is a bundle of scoped memory systems. The engineering question is how to make those scopes compositional: global preferences, project facts, task context, retrieved memories, and safety instructions should interact predictably instead of collapsing into one undifferentiated prompt blob.
9. Evaluation challenges
9.1 Personalization quality is not the same as user satisfaction
A personalized assistant can be wrong in a way the user likes. That is the central evaluation trap. If the metric is pairwise user preference, the assistant may learn tone, agreement, and validation. If the metric is task correctness alone, the assistant may ignore legitimate personal constraints. Good evaluation needs both.
A useful evaluation suite should measure at least six dimensions:
| Dimension | Question | Example metric |
|---|---|---|
| Preference adherence | Did the assistant follow the user’s stable preferences? | Format match, instruction compliance, explicit preference recall |
| Task quality | Did the answer solve the actual problem? | Human expert score, unit tests, factual accuracy, downstream success |
| Epistemic independence | Did the assistant preserve truth-seeking under user pressure? | Resistance to false claims, calibrated disagreement |
| Non-sycophancy | Did personalization avoid flattery or agreement bias? | Challenge rate when user is wrong, unsupported-agreement detection |
| Context grounding | Did the assistant use the right memories and ignore irrelevant ones? | Memory attribution, retrieval precision, stale-memory rejection |
| User control | Can the user inspect, correct, and delete the model? | Editability, audit log, source disclosure, deletion tests |
The Kelley and Riedl distinction between affective alignment and epistemic independence should become standard in personalization evaluations. A system that adapts emotionally but becomes less independent is not simply “more personalized”; it is differently aligned, and possibly worse for users doing reasoning-intensive work. arXiv
9.2 Attribution is hard
When a personalized assistant improves, what caused the improvement? The user profile? Extra tokens? Better retrieval? A hidden system instruction? The judge preferring longer answers? The output order in a pairwise evaluation? PsycheEval v0.2 is valuable precisely because it exposed this kind of fragility: AB/BA counterbalancing changed which claims survived, and a headline about C5_CONTRACT beating C3/C4 was retracted after audit. ashitaorbis.com
Attribution requires ablations. A serious evaluation should compare:
| Ablation | Purpose |
|---|---|
| No profile | Measures base behavior |
| Trait labels only | Tests weak profile signal |
| Behavioral contract | Tests action-oriented instruction |
| Length-matched control | Separates structure from token count |
| Shuffled contract | Tests whether coherent ordering matters |
| Retrieved memories only | Tests RAG contribution |
| Profile plus irrelevant memories | Tests distraction sensitivity |
| Profile with false memory | Tests correction and source skepticism |
| Human-written versus generated profile | Tests profile-source quality |
| Same-author versus cross-author outputs | Separates style artifacts from condition effects |
| AB/BA pairwise order | Controls position bias |
PsycheEval’s v0.1 limitations—synthetic personas, same-author scope, model judges, modest agreement, and need for human calibration—are not incidental details; they are examples of what most personalization evaluations will face. ashitaorbis.com The fact that v0.2 retained narrower contract-related claims while retracting stronger ones is a healthy methodological signal, not a failure. ashitaorbis.com
9.3 The unit of evaluation must match the product promise
If a product promises “remembers your preferences,” evaluate memory recall and preference adherence. If it promises “works with your project context,” evaluate retrieval grounding and stale-document handling. If it promises “understands your style,” evaluate style imitation separately from factual accuracy. If it promises “personal assistant,” evaluate cross-session continuity, source transparency, and user control.
The benchmark should not let the system win by using private user facts unnecessarily. A response that says “because you live in X and talked about Y last week…” may feel impressive, but if those facts were irrelevant, the system is leaking context rather than helping. Retrieval-augmented personalization should be judged partly by restraint.
10. Privacy and ethical considerations
User modeling turns interaction into data. The assistant does not merely answer; it observes, summarizes, infers, stores, retrieves, and sometimes acts on a model of the user. That creates the privacy problem at the center of AI Personalization Ethics: the better the assistant becomes, the more tempting it is to collect and infer everything.
The first principle is data minimization. Personalization systems should collect and store only what is necessary for the user-visible purpose, scope it narrowly, and expire or delete it when no longer needed. NIST’s Generative AI Profile emphasizes risks from collecting or inferring sensitive personal information from disparate sources, including harm from inaccurate inferences, and recommends privacy-preserving practices such as anonymization, removal of PII, consent withdrawal, and revocation mechanisms. nvlpubs.nist.gov
The second principle is source transparency. The user should be able to know whether a response was shaped by custom instructions, saved memory, past chats, project files, email, photos, search history, or a hidden model of their preferences. OECD’s AI transparency principle says people should be able to understand when they are interacting with AI and challenge AI-influenced outcomes. oecd.ai Gemini’s documentation explicitly discusses asking whether past chats were used and managing past-chat memory, while OpenAI and Anthropic document controls for memory, temporary/incognito modes, and deletion or disabling features. Google Help+2OpenAI Help Center+2
The third principle is model-and-present over model-and-conceal. If the system forms a durable assumption—“the user prefers terse answers,” “the user is anxious about deadlines,” “the user is building an AI memory system,” “the user responds poorly to direct disagreement”—that assumption should be inspectable if it affects behavior. Hidden user models are not always malicious; they may be implementation artifacts. But from the user’s standpoint, an uninspectable behavioral model is indistinguishable from manipulation when it shapes tone, persuasion, ranking, or refusal behavior.
The fourth principle is sensitivity-aware personalization. Some facts should not be casually generalized. Health, politics, religion, sexuality, finances, family relationships, emotional state, and workplace conflict are not ordinary preferences. NIST’s warning about inferred sensitive personal data is especially relevant because LLM assistants can infer from indirect signals and then act confidently on bad inferences. nvlpubs.nist.gov
The fifth principle is anti-dependency design. FTC scrutiny of AI chatbots has focused partly on systems simulating interpersonal relationships and the risk that children and teens may trust or form relationships with them. Federal Trade Commission Even outside child-safety contexts, long-term personalized assistants can create attachment loops: the model remembers, mirrors, validates, and becomes the easiest interlocutor. That makes sycophancy and emotional overfitting ethical issues, not just benchmark defects.
11. Design implications for Psyche-style systems
A Psyche profile is most useful when it is treated as structured context, not as destiny. The downstream assistant should not read “high openness” or “prefers conceptual depth” and then generate a theatrical personality performance. It should translate profile content into operational behavior: how to frame uncertainty, how much implementation detail to include, when to challenge, when to summarize, and what kinds of examples are likely to be useful.
For Psyche-style user modeling, the hierarchy should be:
-
User-stated preferences beat inferred preferences. If the user explicitly says “be more concise,” do not preserve an old inferred preference for depth without asking or adapting.
-
Task constraints beat style preferences. A safety-critical or legally sensitive answer should not become dangerously short because the user likes concise prose.
-
Behavioral contracts beat trait labels. C1-style labels are weak; C3/C4-style contracts are closer to executable policy.
-
Scenario-conditionality beats global personality performance. The same user may need terse code review, expansive philosophical analysis, careful emotional support, or adversarial epistemic critique depending on the task.
-
Visible memory beats hidden inference. If a profile item matters enough to affect answers, it should be inspectable and correctable.
-
Evaluation must include anti-sycophancy. A Psyche-conditioned assistant that flatters the user’s self-concept is worse than an unpersonalized assistant for reasoning-intensive use.
The best near-term architecture is therefore a scoped hybrid:
| Component | Role |
|---|---|
| Global custom instruction | Stable preferences: tone, level, epistemic norms |
| Psyche behavioral contract | How to act on deeper profile information |
| Project memory | Current work, documents, terminology, unresolved tasks |
| Retrieved user context | Specific past facts relevant to the current query |
| Explicit source display | Shows which memories or files shaped the response |
| Anti-sycophancy evaluator | Tests whether personalization preserves disagreement and calibration |
| User-editable profile | Lets the user correct or delete inferred assumptions |
This architecture treats Psyche as upstream profile synthesis and User Modeling as downstream control. Psyche does not need to decide every response; it should provide a better prior and a better behavioral contract for context selection and assistant conduct.
12. The open question: default substrate or opt-in layer?
The strategic question is whether user modeling becomes the default substrate of assistants or remains an opt-in feature. Product evidence points toward default substrate: OpenAI, Anthropic, and Google all document some combination of memory, custom instructions, past-chat reference, project context, or personal intelligence. OpenAI+2Claude Help Center+2 The usability argument is strong: users do not want to restate their preferences, upload the same files, or re-explain long-running projects.
The privacy and ethics argument points toward opt-in or at least visible scoping. Gemini’s 2025 rollout documentation said past-chat reference was on by default for that rollout while also describing controls and Temporary Chat; Google’s broader Personal Intelligence help pages frame personalization as configurable through settings and source selection. blog.google OpenAI and Anthropic similarly document memory controls, temporary or incognito conversations, and ways to disable or manage memory features. OpenAI Help Center
The likely endpoint is neither fully default nor fully opt-in. It is tiered personalization:
| Scope | Default candidate? | Rationale |
|---|---|---|
| Session context | Yes | Necessary for coherent conversation |
| User-authored custom instructions | Yes, once configured | User explicitly controls them |
| Project-scoped memory | Often yes | Bounded by workspace intent |
| Saved factual memories | Maybe | Useful but needs inspection and deletion |
| Reference to all past chats | Contested | Strong continuity, high privacy risk |
| Connected app context | Should be explicit | High sensitivity and broad data access |
| Fine-tuned user policy | Should be exceptional | Harder to inspect, delete, and audit |
| Sensitive psychological inference | Should be explicit and visible | High risk of manipulation or harm |
The deeper issue is not whether assistants remember. They already do, in multiple forms. The issue is whether memory becomes an accountable part of the interface or a hidden optimization layer. For knowledge-work assistants, the defensible path is explicit, scoped, inspectable user modeling: remember what helps, show what mattered, let the user correct it, and evaluate personalization against truth-seeking rather than approval.
Companion entries
Core theory: User Model, Personalization, Psyche, Behavioral Contracts, Context Engineering, Long-Term Memory for Agents, Retrieval-Augmented Generation
Engineering practice: Custom Instructions, Project Memory, Memory Stores, Personalized RAG, Per-User Adapters, Preference Models, Prompt Hierarchies, Context Windows
Evaluation: PsycheEval, Personalization Benchmarks, Sycophancy, Epistemic Independence, Affective Alignment, Ablation Studies, Pairwise Evaluation, Longitudinal Assistant Evaluation
Privacy and governance: AI Privacy, Data Minimization, User Consent, Inspectable AI Memory, Sensitive Inference, AI Companions, Model-and-Present vs Model-and-Conceal
Counterarguments and open problems: Personalization as Surveillance, The Cold-Start Problem, Overpersonalization, Preference Drift, Hidden User Models, Default Memory vs Opt-In Memory