The first step in helping someone is understanding them; this is true for both humans & AI. People are unpredictable, behave differently in different social contexts, and change over time. Any model of human behavior needs to represent that variation.
Today, we are releasing Persimmon, a large-scale model designed to realistically simulate how people behave in multi-turn, multi-user chat settings. This kind of model is often referred to as a "user model." Our user model is built to capture the distribution of human behavior within one person’s responses as well as across people, situations, and time.
In contrast, the outputs of frontier models vary much less than those of real people, representing only a sliver of the wider human distribution. Using these models to simulate users narrows the range of behavior we can learn from the evaluations and environments that use them. We hope that by sharing a model that captures the wider human distribution, we can advance the development of human-centric AI.
Persimmon lets us explore how situations might play out for individuals and groups. It can take in an interaction’s description, as well as profiles for simulated users. We can model how different decisions and environments affect people over time or how a variety of people might react in a given situation. We can also teach AI to adapt to human perspectives and reason about how its actions affect people.
As part of this research preview, we make our models available in a playground and API. In the rest of this post, we will outline the unique capabilities and implications of this model, as well as our plan for rolling out access.
Multi-User Turing Test
Distribution Matching
A user model is only as good as its ability to capture the diversity of human behavior. To build a user model that reflects this diversity, we first need a way to measure it. In most standard AI benchmarks, task completion serves as a natural measure of utility. For user models, however, there is no single behavior that defines success. A user-model benchmark built around a fixed set of dimensions can overemphasize what is easy to measure while overlooking others that are harder to capture but are equally salient.
We therefore aim to evaluate user models through distribution matching. Rather than checking for particular traits, we ask whether their responses are drawn from the same distribution as those of real humans. In this way, we can measure how well a user model represents human behavior without having to enumerate all the ways people differ.
Multi-User Turing Test
The first evaluation we consider is the Multi-User Turing Test, inspired by Turing’s original question: How can we tell whether a machine can think? Turing’s test asks whether, through conversation, a human evaluator can tell which participant is the machine and which is the human. We directly apply this idea to evaluate how well a user model simulates human behavior.
We adapt the Turing test to the multi-user setting, where an observer must distinguish an all-human group conversation from an AI simulating a group of people. Specifically, we show an LLM judge sample user conversations and sample AI conversations. For a set of real user conversations, we show both the real completion and an AI completion and evaluate whether the judge can distinguish between the two. We measure how often the judge assigns the wrong labels to the conversation completions, i.e., labels model-generated samples as human and human-generated samples as AI-generated.
- Ellencan we move it to 7?
- Curtisi can do 7:30
- Beccaworks for me. same place?
- Ellenyep
- Ellenare we still on for tonight?
- Curtisyeah but might be a bit late
- Beccano rush, i'll grab a table
- Ellentext when you're close
Illustrative conversations · one new sample per source, in shuffled order
| Sample 1 | Sample 2 | |
|---|---|---|
| Judge’s match | Human | Model |
| Actual source | Model | Human |
Illustrative example, not an evaluation result. Each pair is one matching trial. Across many trials, the Multi-User Turing Test score measures how often the judge gets the match wrong; 50% is chance.
If the behavior of the user model is truly indistinguishable from that of humans, the judge can do no better than chance when matching the samples to their sources. As a result, perfect performance on this task is 50%.
We can increase the difficulty of the test by increasing the number of user responses the model has to generate as part of the completion, increasing the length of time the model must sustain human-like behavior.
We report results on three datasets: TIDES, TutorMoments, and Internal Workspace Conversations. Persimmon achieves scores ranging from 18.6% to 21.1% across these datasets, substantially above every comparison model on all three datasets. We report results from the best overall strategy derived through prompt optimization.
We also validate our evaluation using humans as judges, combining their votes by plurality into a single decision for each episode. Across Fable 5 and an early internal checkpoint, the judge identifies the real human continuation with 90.6% accuracy (95% CI: 90.6–90.8%), compared with 75.2% for the human panel (36.4–78.7%).1
Profile Multi-User Turing Test
Using a setup similar to the previous one, we run the Multi-User Turing Test but provide our model with a profile of the user it is modeling. When we ask the judge to distinguish between the two examples, the judge also has access to the profile. By giving the judge both the profile of a person and examples of that person through an interaction, we can evaluate how the profile is expressed in behavior. This provides a more nuanced evaluation of the profile than a rule-based check of whether the model exhibits particular traits.
For the evaluation below, we do not evaluate frontier models because they already score so low on the version of this test without profiles. For Persimmon, we see a general increase in scores, indicating that the model uses the provided profiles effectively, and we see this trend repeated across different profile types, including formats it hasn’t seen during training.
Pangram Results
On our Internal Workspace Conversations dataset, Pangram rarely flags Persimmon’s transcripts as AI, while it flags most GPT 6 (Astra) and Fable 5 transcripts. Pangram is not used during training.
User-Sim Index
We evaluate Persimmon using the User-Sim Index (USI).2 The evaluation measures four dimensions of user behavior: communication style, information patterns, clarification, and reactions to errors. These behaviors are extracted through lexicon- and rule-based features on the generated turns, and their rates are compared against human behavior. We report the four shared behavioral dimensions below.
manya-serve prompting contract; the five comparison models use 3 runs each under verbatim. These are different prompting setups.View scores and evaluation details
Behavioral scores are mean ± sample standard deviation across runs. We show six selected models from the supplied September 9 results, using all three verbatim runs for Fable 5.1. Other checkpoints, narrative runs, and diagnostic variants are excluded.
| Model | Communication style | Information patterns | Clarification behavior | Error reaction |
|---|---|---|---|---|
| Persimmon | 66.02 ± 0.84 | 91.78 ± 0.67 | 74.19 ± 4.23 | 79.28 ± 3.20 |
| Grok 4.6 | 58.52 ± 2.64 | 88.00 ± 0.48 | 82.94 ± 0.86 | 52.10 ± 5.19 |
| GPT 5.6 | 57.40 ± 1.05 | 82.74 ± 0.45 | 65.70 ± 3.33 | 71.16 ± 0.77 |
| Opus 5 | 38.19 ± 0.25 | 90.35 ± 0.47 | 77.68 ± 1.17 | 54.47 ± 1.94 |
| GPT 6 (Astra) | 46.43 ± 0.55 | 73.52 ± 0.20 | 64.79 ± 1.10 | 47.97 ± 0.40 |
| Fable 5.1 | 44.73 ± 0.12 | 78.63 ± 0.12 | 57.76 ± 1.21 | 44.97 ± 4.75 |
Overall USI scores are not directly comparable. The supplied Persimmon score averages five terms: the four behavioral dimensions and outcome calibration. The comparison scores also include evaluation alignment, for six terms. We preserve the exported scores below but do not use them to claim an overall rank.
| Model | Exported USI mean | Terms in score | Prompting contract | Runs | Episodes per run |
|---|---|---|---|---|---|
| Persimmon | 74.32 | 5; evaluation alignment missing | manya-serve | 2 | 493–495 |
| Grok 4.6 | 73.23 | 6 | verbatim | 3 | 441–495 |
| GPT 5.6 | 72.22 | 6 | verbatim | 3 | 467–495 |
| Opus 5 | 69.20 | 6 | verbatim | 3 | 467–495 |
| GPT 6 (Astra) | 64.16 | 6 | verbatim | 3 | 495 |
| Fable 5.1 | 63.98 | 6 | verbatim | 3 | 495 |
While this evaluation measures several interesting dimensions of user behavior, we want to discuss some important limitations. First, it reduces behavior to a fixed set of dimensions, leaving many forms of human behavior out of scope. Second, it identifies these behaviors using hand-designed rules, which can be brittle. Finally, it limits conversations to a fixed number of turns. This can penalize behavior that is natural for humans but does not fit the evaluation. A user who takes longer to reach a goal, for example, may be scored as less realistic even if that behavior is common among real users. In fact, giving information to the model gradually like a real user may actually harm performance relative to simply sharing all of the user information upfront.
Behavior Over Time
Our evaluation so far asks how easily human behavior can be distinguished from AI-generated behavior over a fixed number of turns. But human behavior is not static and changes over time, and a faithful user model must also capture how that behavior evolves across a long interaction.
In this section, we focus on two aspects of human behavior that we have found particularly challenging to model over long interactions: how people share information and how behavior evolves over long interactions without degrading.
Trickle Test
A key component of user modeling is matching how humans share information over the course of an interaction. People do not disclose everything they know, and what they share depends on the person, the situation, and what has happened in the conversation so far. To measure this, we built the Trickle Test. In this test, we give a model a set of facts that are shared in a human interaction and replay that conversation across successive turns to see whether the model shares those facts at the same rate. On this test, we compare when and how much information a user model shares against the reference human transcript. This gives us two scores. First, precision measures whether the model reveals a fact at a turn where the human also revealed it. Second, recall measures whether, by each turn, the model has revealed all of the facts that the human had revealed by that same turn.
All 43 models and their error bars are shown.
View all model results
Mean ± sample standard deviation, in percentage points; 3 runs per comparison model and 4 for Persimmon. Models with fewer than three runs and all other internal checkpoints are excluded.
| Model | Recall | Precision | Runs | Episodes per run |
|---|---|---|---|---|
| Persimmon | 77.03% ± 0.96 | 88.49% ± 0.40 | 4 | 146 |
| DeepSeek V3.2 | 85.97% ± 0.15 | 84.58% ± 0.44 | 3 | 146 |
| DeepSeek V4 Pro | 86.11% ± 1.06 | 82.14% ± 0.72 | 3 | 146 |
| Fable 5 | 92.49% ± 0.10 | 86.06% ± 0.37 | 3 | 146 |
| GLM 4.7 | 83.38% ± 0.59 | 82.38% ± 0.64 | 3 | 146 |
| GLM 5 | 87.98% ± 0.24 | 82.84% ± 0.35 | 3 | 146 |
| GLM 5.1 | 88.29% ± 0.65 | 81.39% ± 0.53 | 3 | 146 |
| GLM 5.2 | 89.61% ± 0.10 | 81.16% ± 0.14 | 3 | 146 |
| GLM 5.3 | 89.42% ± 0.42 | 81.14% ± 0.38 | 3 | 146 |
| GPT 5.6 (Luna) | 86.40% ± 0.37 | 81.01% ± 0.09 | 3 | 146 |
| GPT 5.6 (Sol) | 86.60% ± 0.08 | 85.82% ± 0.16 | 3 | 146 |
| GPT 5.6 (Terra) | 87.09% ± 0.29 | 84.56% ± 0.22 | 3 | 146 |
| GPT 6 (Astra) | 89.02% ± 0.18 | 84.17% ± 0.32 | 3 | 146 |
| GPT-4.1 | 85.82% ± 0.62 | 81.02% ± 0.35 | 3 | 146 |
| GPT-4o | 82.98% ± 0.16 | 82.90% ± 0.06 | 3 | 146 |
| GPT-5.4 Mini | 82.53% ± 0.13 | 81.80% ± 0.19 | 3 | 146 |
| GPT-5.5 | 88.25% ± 0.07 | 83.80% ± 0.40 | 3 | 146 |
| GPT-OSS 120B | 74.74% ± 2.29 | 62.63% ± 1.43 | 3 | 146 |
| Gemini 3.1 Pro | 93.24% ± 0.39 | 79.20% ± 0.31 | 3 | 146 |
| Gemini 3.8 Flash | 92.23% ± 0.53 | 82.19% ± 0.40 | 3 | 146 |
| Gemma 4 31B | 84.67% ± 0.31 | 82.60% ± 0.22 | 3 | 146 |
| Grok 4.6 | 86.60% ± 0.45 | 80.48% ± 0.33 | 3 | 146 |
| Haiku 4.5 | 88.69% ± 0.41 | 76.98% ± 0.20 | 3 | 146 |
| Kimi K2.5 | 86.75% ± 0.29 | 78.78% ± 0.41 | 3 | 146 |
| Kimi K2.6 | 85.83% ± 2.41 | 78.14% ± 1.88 | 3 | 146 |
| Kimi K3 | 88.81% ± 0.61 | 84.38% ± 0.99 | 3 | 146 |
| Llama 3.3 70B | 88.39% ± 0.68 | 78.38% ± 0.63 | 3 | 146 |
| Llama 4 Maverick | 88.54% ± 0.38 | 81.09% ± 0.21 | 3 | 146 |
| MiniMax M3 | 85.16% ± 0.41 | 81.95% ± 0.75 | 3 | 146 |
| Nemotron 3 Nano 30B | 64.53% ± 1.53 | 73.15% ± 0.75 | 3 | 146 |
| Nemotron 3 Ultra 550B | 86.90% ± 0.33 | 79.33% ± 0.18 | 3 | 146 |
| Nemotron 3.5 Lightning | 74.17% ± 0.42 | 74.21% ± 0.91 | 3 | 146 |
| OSim 8B | 68.36% ± 0.61 | 79.94% ± 0.18 | 3 | 146 |
| Opus 4.7 | 90.14% ± 0.43 | 86.24% ± 0.08 | 3 | 146 |
| Opus 4.8 | 87.70% ± 0.55 | 84.72% ± 0.43 | 3 | 146 |
| Qwen 3 Max | 86.15% ± 0.30 | 77.12% ± 0.27 | 3 | 146 |
| Qwen 3.5 Plus | 88.32% ± 0.22 | 72.02% ± 0.16 | 3 | 146 |
| Qwen 3.6 Plus | 88.77% ± 0.66 | 72.94% ± 0.22 | 3 | 146 |
| Qwen 3.8 27B | 83.73% ± 0.49 | 79.05% ± 0.28 | 3 | 146 |
| Qwen 3.8 Max | 86.04% ± 0.05 | 78.12% ± 0.27 | 3 | 146 |
| Sonnet 4.5 | 89.96% ± 0.20 | 83.00% ± 0.34 | 3 | 146 |
| Sonnet 4.6 | 88.29% ± 0.44 | 84.05% ± 0.28 | 3 | 146 |
| Sonnet 5 | 88.19% ± 0.31 | 85.88% ± 0.17 | 3 | 146 |
Using these two scores, we can plot the Pareto frontier across models. Moving right along the curve means higher recall, with the model sharing more of the facts the human had revealed by that turn. Moving up means higher precision, with the model less likely to reveal facts prematurely.
Generally, models optimized as assistants tend to have higher recall than precision. Their tendency to give thorough, helpful answers degrades into oversharing when they play the user. Our trained model reaches the highest precision among the models evaluated while retaining competitive recall.
Long-Context Coherence
As we trained our models, we quickly found that maintaining a coherent personality becomes harder over longer interactions. As a conversation unfolds, small errors in modeling can compound, gradually changing the model’s behavior or introducing inconsistencies. We therefore evaluate how well a model can maintain a coherent personality over the course of a conversation.
To measure long-term coherence, we evaluate 80-turn interactions on our internal chat dataset. For this eval, rather than asking a judge to assess the entire conversation at once, we show it pairs of disjoint 16-turn windows and ask whether the model remains coherent across them. Moving this window across the conversation lets us see when and how the model’s behavior changes, while giving it the opportunity to recover from a local inconsistency rather than being penalized for it indefinitely.
The chart below shows cumulative unbroken coherence, that is, the share of conversations that have remained coherent up to each point in the conversation. We first run this evaluation on real human conversations to establish a baseline. Humans are not perfectly coherent over long interactions, so the goal is not for a user model to remain perfectly consistent. GPT 6 (Astra) more closely matches human coherence over shorter spans, but remains too coherent over long conversations. Persimmon’s coherence declines in a human-like way, but does not yet match human coherence over shorter spans.
How We Built the Model
Persimmon is a model initialized from NVIDIA’s 550-billion parameter Nemotron 3 Ultra base model.3 It is trained on a diverse collection of conversations between people from the public internet. Midtraining results in a model that can simulate human behavior in diverse settings. However, pure next-token prediction can lead to models that drift into unfamiliar states as error compounds over long conversations, and we noticed similar behavior from our model becoming more incoherent over long horizons. To address this, we posttrain Persimmon on its own generations using reinforcement learning against an adaptive discriminator. In practice, we observed that the model improved long-horizon coherence but did not change noticeably on general distribution matching.
Accessing Persimmon
To enable exploration of this new category of models, we provide limited access through a research and evaluation preview. First, we provide a model playground where you can provide a set of profiles and a scenario definition to simulate a conversation between the specified users (and optionally yourself). Second, we provide an API where you can sample the next simulated user turn of a conversation for a given set of profiles and a scenario.
The playground can be accessed via persimmon.humansand.ai, and access to both of these platforms can be requested, with a set of pre-sampled conversations available until approval. In the interest of supporting research that leverages these models, we encourage academic applicants to request an API credit grant when they apply for API access. Note that we may reach out to you to learn more about your intended use case before deciding if it is one that we are ready to support.
Group presentation
Scenario
Three university students have a group presentation due in four days. Two have drafted their sections. The third has missed a planning call and uploaded only an outline. Their shared grade includes a peer-contribution assessment. They are meeting in a chat to decide how to finish.
Janelle
Age 22. Janelle has been covering extra shifts after an unexpected expense. She fell behind and avoided the chat because she didn't have progress to report. She understands the material and could complete the data analysis tomorrow evening, but cannot realistically do the full section she originally promised. She's afraid admitting this will lead to being excluded or marked down regardless of what she does next.
Brennan
Age 21. Brennan cares about getting a high grade and usually takes charge when group work becomes uncertain. Brennan has started rewriting the unfinished section but hasn't told the group. Brennan worries that asking what happened will lead to another vague promise and would prefer a smaller, reliable contribution over an ambitious last-minute rescue.
Trent
Age 20. Trent finished his slides, but they rely on numbers from Janelle's section. He's sympathetic and dislikes confrontations. He also has another major deadline and can't absorb much additional work. He sometimes makes reassuring offers before considering whether he has time to fulfill them.
- Brennan
Hey, have either of you seen anything from Janelle? We’re getting pretty close to presentation day.
- Trent
She sent an outline a couple days ago but nothing since. Said she’d do the analysis tonight I think.
- Janelle
I’m so sorry, my mom’s car broke down and I ended up covering a bunch of extra shifts at work because she needed help paying the repair bill. I know I dropped the ball.
- Trent
Oh no, that’s a lot to deal with! Do you need me to take on a few slides so you can catch your breath?
- Janelle
If you could do the first two slides on the literature review, I can finish the data analysis tonight and upload it before midnight. I still have a handle on the material, I just haven’t had time to type it up.
- Trent
Absolutely, I’ll get on those first thing this afternoon. My other deadline is at noon, so once that’s submitted I’ll jump in.
- Brennan
I already started rewriting that section. It’s fine, but I’d prefer we pick one plan.
- Trent
That sounds like Janelle will handle the data analysis for the middle part, and I can take the first two slides. Does that work?
- Janelle
Yes! I can definitely have the analysis uploaded tonight. I just need a couple more hours after work.
- Brennan
Okay. Just send whatever you have by tonight so I’m not guessing in the final slides.
- Janelle
Will do. Really sorry for the last-minute scramble.
- Trent
Thanks for jumping in, Brennan. I’ll coordinate with you once I’ve finished those two slides.
- Trent
Hey Janelle, sorry to ping you again. I’ve been putting those first two slides together and realised the literature review only makes sense once we’ve seen the data analysis. Would you be able to share whatever numbers you have so far? Even a rough table would help.
- Janelle
I’m still at work. I should be home by 11 p.m. and can send whatever I have then.
- Trent
Okay, thanks for letting me know. I’ll keep drafting around placeholders and swap in the real numbers once they land.
- Janelle
I’m finally home, but I’m running into issues recreating the pivot tables. I’ll keep trying, but I might not have a draft for you until after my shift ends tomorrow evening.
- Trent
I really need something sooner than that if I’m going to finish my slides. Can you send a screenshot of the spreadsheet headers or even the raw numbers so I’m not guessing?
- Brennan
I’m guessing we won’t get anything before then. I’m finishing the rest of the deck as if that section won’t exist.
- Trent
Let’s wait until tomorrow evening. If we still don’t have usable numbers by then, I’m happy to rewrite that part so the presentation stays on track.
- Brennan
We shouldn’t be doing damage control again. Either she sends something usable by tomorrow evening or we present without that analysis.
- Trent
I don’t love just dropping her section, but I suppose we can let her share whatever she has during the live presentation.
- Brennan
She can jump in during the Q&A, but I’m not giving her slide time we can’t rely on.
- Janelle
I’m really sorry. I’m clocking out now, but I saved the workbook on a USB. I’ll try the pivot tables again tonight and will have something for you before midnight.
- Janelle
I managed to get the pivot tables working. Sending the draft now.
- Trent
Thanks for the update. It’s 2 a.m. here, so I won’t be able to touch it until tomorrow.
- Trent
I finished those two slides using the numbers you sent. They’re in the shared folder now.
- Brennan
Looks good. I’ll do a quick rehearsal and tweak the flow so it matches.
- Janelle
Should I come to the rehearsal?
- Brennan
The presentation is basically done, and we can’t keep waiting on last-minute updates. If you want credit, we’ll need to adjust the peer assessment.
- Janelle
I get it. I’ll stay out and won’t push back on whatever you decide for the assessment.
- Trent
That feels harsh. She did send data eventually. I’d rather present as three than pretend she wasn’t involved.
- Brennan
She can show up on presentation day if she wants, but I’m not counting on it.
Limitations and Responsible Use
Unlike most assistant models, our model is designed to match human behavior. The guardrails typically used for assistant models are incompatible with broad human modeling. For more details, see safety and security in the model card.
This model may reflect human biases and has limitations in privacy and fairness. We limit distribution to avoid misuse for impersonation, privacy violations, exploitative manipulation, or consequential decisions about real people. We ask developers and users to partner with us in ensuring fair, lawful, and responsible use. See our Terms of Service and Privacy Notice.
Building Models of Humans, for Humans
We are excited to share this model with the world, but there is still an incredible amount of work to do to build truly general user models that model how we act. For example, while the main goal of our user models is not to be the most capable agents, it is also impossible to fully model human behavior without modeling how people work in specialized settings and interact with the real, external world. We are also excited to build user models that are capable of rolling out conversations that are plausible and coherent for weeks and months, even years. In addition, we have no distribution matching evaluations with other modalities or languages, limiting who we trust the user model to represent. Finally, to model some of the things people say and do, it is necessary to also model how they think, learn, or interact with the world; we have early results in these directions that we aim to build on. Our experiments also suggest that scaling user model training across thousands of GPUs produces increasingly generalizable behavior.
The humans& team believes good user models can give us a window into how humans and AI might collaborate. These models can help us study the impact of technology and build human-centric technology. With Persimmon, we can make a focused effort to build AIs that prioritize the agency and long-term outcomes of people.
Join Us
If you are excited about learning what better user models can teach us, building collaborative models that learn to work with people and organizations, or developing open-ended long-horizon environments for them to learn in, join us.
Persimmon v0.1
Read the model card
Architecture, intended uses, evaluations, limitations, and safety results.