Foundation models have become very good at understanding an image, a video clip, or a conversation. But a life is not a clip. A cat, a student, an elderly parent: each is an individual unfolding over time, with their own habits, rhythms, and ways of changing. Who they are today is shaped by the trajectory of the past months. The same behavior means different things in different individuals, and in the same individual at different stages of life.
At Being Labs, we are building the Life Model: an approach to continuously modeling one specific individual. It reads long-term, multimodal observations of the same individual, maintains an individual state that updates over time, and uses that state to predict the future: what is normal for them, what is drifting, and what is likely to happen next.
Our first domain is pets. Pets can be observed continuously over long periods with low privacy risk; behavioral events and veterinary outcomes provide objective labels; and a signal like "this cat is drinking 30% less water than she usually does" is exactly what general-purpose models struggle to see, and exactly what owners and veterinarians care about most.
The Life Model targets three capabilities:
- Individual State Modeling: turning continuous video, audio, device events, and owner notes into an updatable individual state, rather than a one-off description of a single clip.
- Long-Term Change Understanding: learning each individual's own baseline across multiple time scales (7 days, 3 months, full history), and detecting deviations relative to their own habits, not just relative to a population average.
- Individual-Conditioned Prediction: combining population patterns, personal history, and current context to forecast state and risk at 1 / 7 / 30 / 180 days, with calibrated confidence and a human-readable explanation.
These capabilities are under active research. This post describes how we define the problem, how we are designing the model, what external evidence already exists, and the experiments we will use to prove, or disprove, this direction.
Individuals, Not Clips
"Sleeping 14 hours during the day" means something completely different for these two cats.
A general multimodal model sees the present moment: the cat in this video is asleep. The Life Model has to answer a different question: what does this mean for this cat?
We frame the problem as an explicit prediction target:
P( future outcome y(i, t+Δ) | history of individual i up to t, X(i, 1:t), current context c(t) )
individual state = θ (shared population patterns) + φ_i (stable individual traits) + s_(i,t) (time-varying state)
learned from many individuals specific to this individual continuously updatedθ lets the model get started on an individual it has never seen before; φ_i lets it remember that "this cat always drinks more in summer"; s_(i,t) lets it keep up with "this cat has been changing over the past two weeks." The difficulty is that health events are rare, data is noisy, and individuals differ widely. Under these real-world conditions, the model has to detect change early and produce trustworthy probabilities, not vague descriptions.
Relative to Their Own Baseline
The same reading can look normal against one baseline and anomalous against another.
The Life Model maintains baselines at multiple time scales for every individual. The short-term baseline tracks recent fluctuation; the medium-term baseline tracks seasons and life changes such as moving house or a new family member; the long-term baseline captures who this cat is. The deviations worth telling an owner about are usually persistent ones that several scales agree on.
Judgments You Can Check
A model that can only say "something might be wrong" helps neither owners nor vets. Every Life Model output has two parts: a structured prediction (current state, short- and long-term risk, confidence), and a written explanation a veterinarian can verify: what it observed, which part of the history it relied on, and why it reached this conclusion.
How It Works: A Central Model with Heterogeneous Experts
The Life Model uses one multimodal model as its sole decision-maker; expert models contribute individual-state representations to it as additional "expert modalities."
We call this design HE-VLM (Heterogeneous Expert-Augmented VLM).
At the center is an open-source vision-language model (VLM). It is the only model that makes judgments. Around it are several kinds of experts, each already proven in its own field:
- Knowledge Tracing (KT) expert: learner-state modeling from education (PSI-KT). We generalize "mastery / learning rate / forgetting rate" into individual latent states such as "health status / recovery rate / decline rate."
- Time Series (TS) expert: time-series foundation models (Chronos-2, Toto, and others) that extract representations of sensor and behavioral sequences at short, medium, and long scales.
- Temporal Knowledge Graph (KG): a readable record of the individual: facts, baselines, rules, and open hypotheses, each with a validity window and a source, so veterinarians and owners can inspect and correct them.
- Deep Research (DR) expert: retrieves external medical or behavioral knowledge on demand, when the model runs into something it does not know.
The key difference: experts do not "translate their predictions into a sentence" for the language model. Instead, each expert's hidden state is mapped by a projector into latent tokens the VLM can read, and fusion happens in a shared latent space. This builds on how LLaVA and BLIP-2 connect vision encoders to language models, and on how TransNAR injects the hidden states of a pretrained expert into a Transformer through cross-attention. Whether this beats feeding experts' outputs to the model as text in longitudinal individual modeling is one of the first questions our Phase 1 experiments are designed to answer.
Gather Evidence Selectively, Don't Concatenate Everything
Different questions need different evidence. Long-term forecasting leans on long-range time series; state understanding leans on KT; explaining an anomaly leans on the knowledge graph; external research is only called for unknown knowledge. Stuffing every expert's features into the context at once is both expensive and prone to interference. So the central model decides for itself which experts to call in each round, and how heavily to weight them.
A Self-Evolving Agent Loop
The Life Model reaches a judgment through multiple rounds of evidence gathering and self-assessment, not a single forward pass.
Judgments about an individual rarely happen in one step. A veterinarian looks at the trend, notices an anomaly, checks the same period in past records, then brings in medical knowledge to explain it. The Life Model works the same way. Inside a multi-round loop, each round can make two kinds of "self-modification":
- Modify non-parametric memory: explicitly write new evidence, revised individual baselines, and hypotheses to be tested into the individual's record. For example: "Since June 12, nighttime activity is −35% versus her own baseline," or "This cat naturally drinks more in summer; raise the threshold." This memory needs no training and is readable, traceable, and reversible.
- Update parametric state: fuse the experts' dense representations into the model's internal state, carrying individual patterns that are hard to put into words.
Each feeds the other within a round: retrieved memories are encoded into the internal state, and new findings in the internal state are written back to memory. The loop ends and outputs a prediction once evidence is sufficient, confidence is high enough, and the two kinds of memory agree.
Predictions Tested Against Reality
Every prediction is eventually checked against what actually happens: did the cat go to the vet, did the owner confirm the change in behavior? These outcomes flow back at three time scales:
- Seconds (within a loop): expert fusion updates the internal state.
- Daily: outcomes are written back to the individual record, correcting baselines and hypotheses. Cheap, immediate, and reversible.
- Weekly / monthly: accumulated prediction–outcome pairs train the projectors, adapters, and experts, consolidating into the model's own capabilities.
This is the data flywheel landing on the model in two steps: first update memory, then update parameters.
The Evidence So Far
The Life Model did not come out of nowhere. It grows out of what we have built and are building ourselves: industrial experience modeling individuals at scale, a system for collecting pet signals, a line of methods for plugging heterogeneous models into a large model, and a foundation in brain and cognitive science. But we want to be clear: none of this proves the Life Model itself. What they do is narrow the question we actually need to answer ourselves into something smaller and sharper.
What We Have Done: Modeling Individuals at Scale
Members of our core team led user profiling and ID modeling at Meta (Facebook) and Verizon. At Meta, a multimodal user-understanding model served as a top feature in platform-wide automated baselines, supporting ads revenue growth over the long term. ID modeling is essentially an industrial version of the Life Model formula: all users share one model (θ), each user has an ID representation learned from long-term behavior (φ_i), and recent behavior adds a time-varying state (s_(i,t)); new users start from population patterns, every prediction is trained on real feedback, and evaluation is split by user and by time. The Life Model takes this approach to individual modeling from clicks and conversions to the health and behavior of a living being.
What We Are Building: Pet Signal Collection
We are working with PETKIT on a pet signal collection system that brings device events such as feeding, drinking, litter use, and activity, together with video and owner notes, onto a single timeline per individual. 【To be added: devices and signal types, number of individuals and time span, outcome labels and consent arrangements】 The hardest and most important ingredient for the Life Model is data that is consented, from the same individual, across time, and paired with real outcomes. This system is the data prerequisite for testing whether individual history improves prediction. It is a data pipeline, not yet evidence that the model works.
Our Methods: Plugging Heterogeneous Experts into a Large Model
The Life Model's central model has to read, fuse, and keep absorbing individual-state signals from different experts. Our collaborators at East China Normal University have validated three links in that chain:
- Turning expert states into tokens a large model can read. LLM-KT plugs student states learned by traditional knowledge-tracing models into a large model through adapters, as question- and concept-specific tokens, and compresses long histories into insertable embeddings; it outperforms about 20 strong baselines on 4 datasets. It is the direct prototype for expert tokens such as
<KT_STATE>. - Fusing models with different architectures. HeteroFusion aligns language models from different families such as Llama, Qwen, and Mistral by functional structure rather than parameter coordinates, and suppresses conflicting signals during fusion, consistently outperforming existing merging, fusion, and ensemble methods. The Life Model extends the same idea from "between language models" to "between life experts and a central model."
- Learning continuously without forgetting. LifeAlign (AAAI-26) keeps a large model aligned across a sequence of new tasks without forgetting, by consolidating denoised short-term representations into long-term memory. This mirrors the Life Model's self-evolution: write to memory daily, consolidate into parameters periodically, and keep what has already been learned.
These papers show that the method components work. They deal with exam answering, language models, and preference alignment; whether they hold up on continuous, multimodal, longitudinal pet data is still for our own experiments to answer.
Our Scientific Foundation: Brain and Cognitive Science
Our researchers come from brain and cognitive science at East China Normal University. The NYU–ECNU Institute of Brain and Cognitive Science studies perception, language, learning and memory, and decision making; the ECNU Children's Brain Imaging Center, opened in 2021 and reported as the first of its kind in China, builds on a multi-year brain development study following more than 500 children and adolescents. Longitudinal research of this kind, following the same individuals over years with multiple modalities, has the same shape as the Life Model's data; and research on implicit learning and memory, how experience keeps shaping behavior below awareness, constrains how an individual state should form and update.
| Claim | Strength of external evidence | Our related work | Do we still need to validate it? |
|---|---|---|---|
| Living beings have learnable longitudinal dynamics | Strong | Children's brain development cohort (research environment) | No need to prove from scratch |
| Historical trajectories improve future prediction | Strong | User profiling and ID modeling | Must be reproduced for pets |
| Multimodal history can form a latent state representation | Strong | LLM-KT: states injected into an LLM as tokens | Must define the state objective for pets |
| Population prior + individual adaptation is feasible | Moderate to strong | ID modeling: shared model + per-individual representation | Must test data efficiency and generalization |
| Heterogeneous experts can be fused into one model | Moderate | HeteroFusion | Must extend from language models to life experts |
| Continuously updated individual models can be deployed | Real-world deployment evidence | LifeAlign: continual learning without forgetting | Must test whether dedicated modeling is worth it |
| Consented, same-individual, outcome-linked pet data over time | — | Signal collection system with PETKIT | Must confirm scale, outcome labels, and consent |
| Learned individual dynamics > strong memory / retrieval | Insufficient | — | This is our core experimental gap |
External research and our own work make long-term, multimodal, individualized state and trajectory modeling a well-grounded direction; the last row is the question Being Labs has to answer itself.
How We Will Know It Works
The real question is not whether individual history is useful. External evidence already supports that fairly strongly. The real question is: when a strong general model can already read the full history through memory and retrieval, do learned individual states and dynamics still deliver additional gains that are stable, significant, and worth the cost?
Three Gates
| Gate | Core question | Minimal comparison | Signal to keep investing |
|---|---|---|---|
| G1 Individual Signal | Does the same individual's history improve future prediction? | Current observation / population context vs. same-individual history | Significant gains that hold across targets and individuals |
| G2 Modeling Gain | Do we need learned state / dynamics? | Strong retrieval / structured memory vs. learned state and dynamics | Consistently better accuracy, context cost, or robustness |
| G3 Economic Feasibility | How much high-quality history does personalization require? | History-length curve over 10 / 25 / 50 / 100 event–response episodes | Repeatable personalization gains from relatively little individual data |
Baselines and Evaluation
We compare four approaches on the same tasks, the same data, and comparable compute:
- B0: a general VLM that sees only the current observation (with a closed-source general model as a reference ceiling)
- B1: a general VLM plus individual history (long context / RAG / knowledge-graph memory)
- B2: pure time-series or pure knowledge-tracing models
- B3: HE-VLM (our approach)
Every result is reported separately for an individual split (pets never seen in training) and a temporal split (future time periods), along with a curve of history length (1 week / 1 month / 3 months) versus performance, to test whether long-term history yields stable gains or is largely covered by a general model plus retrieval.
| Task | Metrics |
|---|---|
| Current state / behavioral event recognition | F1, macro-averaged recall |
| Change detection relative to individual baseline | AUPRC, false alarms per pet per week, days of early detection |
| Future state / event prediction | AUROC / AUPRC, CRPS / MASE, expected calibration error (ECE) |
| Individual consistency and explanation | Consistency of judgments over time; agreement between explanation and structured prediction (vet-rated) |
| Cost | Daily inference cost per pet, latency, average loop rounds and expert calls |
Our Phase 1 targets, relative to B1, are a ≥20% relative improvement in AUPRC for 7-day anomaly / health-event warnings, and ≥30% fewer false alarms at equal recall. These are target values for deciding whether to continue down this path, not results we have achieved.
Key ablations include: expert outputs as text vs. expert token injection; single forward pass vs. multi-round agent loop; updating internal state only vs. also writing to the individual record; and self-evolution with vs. without outcome feedback.
A Simulation Experiment
A controlled simulation testing whether an individual's past recoveries help predict their next recovery.
Pick a test subject, then hover over the output to reveal the actual future.
Before real longitudinal data arrives, we built a minimal version of G1 on simulated data. Holding current activity level fixed, it compares a personal-average baseline, a population recovery curve, a per-individual fit, and an "individual state model with population shrinkage," with mismatched history as a control. This experiment can only show that, under this simulated setup, matched individual history carries predictive value. It does not demonstrate anything about real cats' physiology or the advantages of a new architecture. It is a rehearsal of the experimental protocol, not a conclusion.
Technical Details
Data
- Pets (primary): camera video and audio that can follow the same pet over time; events from smart feeders, water fountains, litter boxes, and collars; owner notes plus vet visits and checkups (as real-world outcomes); context such as season, moving house, or a new family member.
- Education (secondary): learning records, mistakes, and knowledge states, a natural fit for the KT expert.
Each individual's data is organized under a Life ID into three layers: timeline, multi-scale summaries, and outcome events. Every training example takes the form "multimodal history up to a point in time + expert features + real outcomes in a future window," with strict safeguards against future information leakage.
Phase 1 plans to cover 300–500 pets with at least 3 months of observation each, of which at least 50–100 should have a health or behavioral event confirmed by a veterinarian.
Training
| Stage | What is trained | Objective |
|---|---|---|
| 1. Expert alignment | Projectors only | Align each expert's hidden states to the VLM's latent space |
| 2. Adapter | LoRA + Cross-Attention Expert Adapter | State, forecast, and text losses |
| 3. Joint SFT | VLM (partially) + all adapters + structured prediction head | Joint language, short-term, long-term, state, and calibration objectives |
| 4. Loop policy RL | Expert selection, memory writing, and stopping policy | Accurate predictions with fewer calls, fewer rounds, and lean memory |
What Worries Us Most
- Is there enough signal? Health events are rare and labels are noisy.
- Will the gains be absorbed? The value of long-term history may be largely covered by a general model plus retrieval.
- Will the model actually use expert tokens? With limited data, models risk ignoring new modalities. Mitigation: a dedicated alignment stage, expert dropout, and auxiliary prediction losses.
- Memory contamination. The model writes a wrong conclusion and then reinforces it. Mitigation: every write carries confidence and provenance, memories have validity windows, falsified memories are rolled back when outcomes arrive, and veterinarians audit records periodically.
- Cost. Training and inference over long histories are expensive. Mitigation: cache expert features offline, compress long histories into expert tokens, and call experts on demand within the loop.
We list these here because answering them is precisely the point of Phase 1.
Starting with Pets, Extending to More Lives
A unified "expert modality" framework means entering a new domain only requires plugging in new experts or projectors. What accumulates stays in the experts, data, evaluations, and training recipe, and the backbone can be swapped out as foundation models improve.
Build with Us
The core moat of the Life Model is not any single model. It is data that is properly consented, from the same individual, across time, and paired with real outcomes, together with a longitudinal evaluation suite that can test it honestly.
We are looking for partners: pet smart-hardware makers, veterinary hospitals and vet teams, pet insurers, and researchers interested in long-term individual modeling. If you would like to co-build data, run a pilot, or collaborate on research, please get in touch.