Tencent's Hy Memory Is Only as Good as the LLM That Writes It. I Measured It on LongMemEval

Tencent released Hy Memory 2.0 this week: an open framework (Apache-2.0), a 30B memory model and MemHub, an app that shares one memory across AI tools. I build something in the same space, so I wanted to know how it does on the two failures I care most about:

Disclosure: I'm the author of Aura, an open-source memory library with a Windows app. I wrote down the setup and pass lines before running, and I report the results that went against me too.

I used LongMemEval, a public benchmark. Each question comes with a user's chat history of about 50 conversations, and the answer is hidden somewhere in it. I took three of its six question types, 20 questions each:

Both systems got the same history and the same question. For both, the same model read the retrieved memory and answered (Gemini 3.8 Flash), and the same model graded the answer with LongMemEval's official judge prompts.

32 of Hy's 34 misses were "I do not have that information".

The first objection I expected: Hy's 10 facts are shorter than Aura's 10 turns (about 2,000 vs 5,000 characters), so maybe Hy just had less to read. I searched Hy's stores again with a bigger limit, without writing anything again. With as much text as Aura it scored 45%. With 40 memories (about 7,900 characters) it still scored 45%. The information was lost when the conversations were turned into facts, not at search time.

Opus as the writer cost about $5 per question, so I ran 10 questions per type, not 20.

Knowledge update, the same 10 questions:

This one went against me. With a strong writer, Hy tracks changed facts as well as Aura does. It keeps the old value as history ("Current: 25:50 … History: 27:12"), which is the right design. Both missed the same question: two different counts on the same day.

Assistant said, 8 questions (my Anthropic credit ran out during two of the ten, so those two are excluded):

Opus helps a lot here: it does store some of the assistant's advice as memories ("The assistant suggested…"), which Flash never did. Still, half of what the assistant said never made it into memory. All four misses were "I do not have that information", even with 40 memories retrieved.

For one user history of about 50 conversations:

That's the real trade-off. Hy's quality is bought at write time, with an LLM reading every conversation. Use a cheap model and it drops to 55–60% on changed facts. Use Opus and it matches a memory that never calls a model, at about $5 per user history.

Hy compresses each conversation into statements about the user ("The user is interested in learning about and cooking classic Peruvian cuisine"). That makes memory small and tidy. But it throws away whatever the writer doesn't consider a fact about the user: what the assistant recommended, details that seemed minor, sometimes the newer version of a fact. Examples from the run with Flash:

Aura goes the other way: it keeps what was said, with who said it and when, and leaves deciding what matters to the model that answers. That costs more storage and gives the reader more text. But nothing has to be guessed at write time.

If you evaluate a memory system that summarises with an LLM, check two things on your own data: which model it needs to be good, and what that model costs per user. And ask whether it keeps what the assistant said, because "what did you recommend last time?" is a question users do ask.

Aura is open source (MIT): github.com/teolex2020/aura-memory. The Windows app shares one local memory across Claude Code, Cursor, Codex, Gemini CLI and others: aurasdk.dev/desktop.