From Medicine to Memory: Why we are talking about how AI systems remember things, and why it matters in healthcare
Why we are talking about how AI systems remember things, and why it matters in healthcare.
- Published
- Reading time 7 minutes
- Topics AI leadership, Technology change
The revolution that would turn the world on its head was published in 2017, “Attention Is All You Need,” published by a team of Google researchers and ushering in the modern transformer-based AI era. With the rise of AI chat and now agentic systems, we now find ourselves talking about how the AI acts, but also how it remembers the critical information an organization needs.
The Origins of Machine Memory: Making tools that remember
How we make machines remember useful information is not a new challenge. As early as 1725 French silk weavers were deploying the first punched-paper memory to make woven fabrics and shortly thereafter 1830s mills were recording information on punched units and the 1890s saw the revolution in punch card paper for data entry and batch storage. In the late 1800s brass pinwheel machines used internal wheels with retractable teeth (pins) and rotated by a hand crank to perform cumulative addition, subtraction, multiplication, and division, the same calculations performed by modern GPUs, albeit orders of magnitude slower. By the 1940s vacuum tubes and acoustics had revolutionized memory storage. In the 1960s when the Apollo moon landing missions began, writing code involved the transfer of enormous amounts of written computations by keypunch operators that made holes in paper cards, called punch cards (programmers had to use paper punch cards to feed information into room-sized computers). The latter half of the 20th century saw a rapid transition from large mainframes to increasingly portable SRAM/DRAM, floppy and compact disks (CDs/CD-RWs), and flash memory.
So why do we need machines that remember? These give us ways to automate or semi-automate complex, error-prone, time-consuming tasks. For example, while NASA space mission-grade computations can be made on paper cards, they may take hundreds of people, and a single bent card can derail weeks of work. The same calculations can now be performed easily on laptop computers with a decent RAM and GPU by a single engineer. But the benefit is not just speed, it’s fewer errors, and the ability to iterate. Where one simulation cost NASA weeks and hundreds of trained staff in the 1960s, now we can easily run side-by-side simulations letting us engineer better solutions.

Machines that remember, 1725 to today. The source for each date is listed under Sources below.
How Do We Make AI Systems Remember
There are a variety of reasons we might want AI software to remember. First for competency, remembering social norms, speech patterns, and other traits that improve their functioning. Secondly, organizational and disciplinary, we need AI systems to remember things about our profession, our work, our organizations, and its context (such as specific past clients or projects).
In the 2010s, early AI systems used external differentiable memory, or Recurrent Neural Network or LSTM, coupled with array or RAM-like matrix as a scratchpad that provided an external memory matrix. While these decoupled systems allowed for expansion and end-to-end differentiability, they were also limited by high compute load and slow recall compared to modern transformer-based AI technologies. This era of Neural Turing Machines is, however, worth exploring and provides foundations for modern memory systems.
With the advent and subsequent wide adoption of transformer-based AI technologies from 2017 to 2026, we have rapidly moved from “generative” AI using stateless, fixed-window self-attention (basically just asking for the information already in the transformer powering the AI) into more complex multi-tiered architectures that combine what is in the transformer, smart caching of our prior interaction information, and recall or access to information outside of the AI transformer that provides specific instructions, state information, and/or business information to the AI.

How AI models learned to remember, 2014 to 2025. The source for each date is listed under Sources below.
Broadly, these make up the new fields of AI and Agentic Systems engineering, Prompt or (more broadly) Context engineering, and Graph engineering, which have become buzz wording in industry. Let’s take some time to demystify these.
What is “Prompt Engineering” and Why it’s Already Behind the Curve
When AI systems were largely stateless a few years ago, arguably the most important thing you could do to improve AI performance was to develop good skills in writing to the AI, or “prompting.” We saw a proliferation of e-courses and web gurus spring up, and businesses focused on training their users to prompt better, and even LangChain and others developing prompting into whole fields of engineering. These approaches largely assumed that what you write makes up the bulk of what the AI sees.
Modern AI knows more than you think. In modern systems, what the AI sees is much broader. A typical “prompt” involved both the user message and instructions, system information, safety and compliance instructions, full or summaries of past interactions with the AI, information and results of tools the AI has access to, and results of recalled third party memory modules. On consumer platforms like ChatGPT, Claude, Gemini, Grok and others with memory functions enabled (and even if you don’t, such as ChatGPT’s age prediction which is applied to the account whether or not the user has provided it). Organizations often provide background information on companies and products. Specialized legal services AI allow tools to search information about laws and regulations. Enterprise providers like Snowflake, Databricks, among others, enable complex relationships between enterprise organizations’ data and AI tools.
The entirety of the information available to the AI is considered its “context.” Most organizations have moved from a focus on prompt engineering, to engineering wholistic “context” that maximizes utility and minimizes user error.
Where we’ve Been and Where We are Headed with AI Memory next
We are moving into an age of increasingly automated and semi-automated processes, where Agentic hybrid AI systems, rather than stateless solo large language model (LLM) transformers, are increasingly important in enterprise business and health tech. Importantly, 12-24 months ago we were focused on making AI memory smarter through information retrieval, through a broad set of techniques referred to as retrieval-augmented generation (or RAG).
You can think of RAG as non-AI software that finds and serves up relevant information based on the user’s question. For instance, a user could ask about a customer order or legal statute, and deterministic software would find and insert the answer into the prompt for the AI, enabling it access to the information.
The downside of RAG is that it is only as good as the software used for search and detection. To solve this, much of the industry leaned into 2025 systems that use Knowledge Graphs alongside vector indexes to help the AI recall useful information and related information, which are found broadly in LangChain and other orchestrations using software like Neo4j.
Modern Agentic tools take a different approach, allowing intelligent agents to use a variety of tools to directly query memory stores and even search for information in smarter ways. As many developers say, “RAG is retrieval, not memory.” The next frontier is developing hierarchical and tool-based approaches that enable agentic systems to access, manage, and reason over complex data sources.

Where an AI model keeps its memory, 2017 to 2025. The source for each date is listed under Sources below.
What’s Next for AI Leaders?
In the age of agentic AI, context is king. That means ensuring AI agents within your org are situated with organizational information, appropriate tiered data access (and smart ways to search it) and rails to prevent users from misusing or driving up costs. 72% of IT leaders surveyed by Confluent, a leading provider, said that the biggest barrier to AI was the data infrastructure needed to support it.
Many are concerned with the rising cost of AI. McKinsey research showed that 93% of surveyed enterprises had exceeded their planned AI budget in 2026. For us to control costs while getting the best data and end user experiences, that means creating agents with smarter memory designs, lower overall token usage, expertly engineering context, and the ability to say no to or cut-off runaway tasks. Most organizations have only scratched the surface of what is possible, and many are in the infancy of building out internal AI teams. This is why it is so important to work with experts.