Built for How Modern AI Thinks: Inside Titan, the JohnAIC Inference Engine
Author: JohnAIC Team
The Engine Matters as Much as the Model
Most conversations about AI are about models: which one is smartest, which one tops the leaderboard this week. Far fewer are about the engine underneath, the software that runs the model on a GPU and turns it into answers.
Yet that engine decides how fast your AI responds, how many people it can serve at once and how much hardware you need to buy. Two organisations can run the same model on the same GPU and get very different results, depending on the software in between.
At JohnAIC, that engine is Titan. It runs on every JohnAIC compute node, and it is built around a simple idea: modern AI models work differently from the ones that came before, and the software running them should too.
The Hidden Cost of Remembering Everything
Imagine reading a 500-page contract and keeping a sticky note for every word, just in case you need to look back at it. That is roughly how a standard AI model, a transformer, handles a conversation.
For every token it reads, a transformer stores a small record in what engineers call the KV cache (key-value cache). When it writes the next word, it looks back across all of those records to decide what matters. This is what makes transformers so good at recalling details from earlier in a document.
The catch is that the cache grows with every token. Double the length of the conversation and you double the memory. Serve sixteen users at once and you need sixteen caches. All of it has to sit in GPU memory, the most expensive and most limited memory in the system.
This is why long-context AI has traditionally meant big, costly GPUs. The model itself might fit comfortably on a modest card. Its memory of the conversation is what pushes you towards flagship hardware.
A Better Idea: Keep a Running Summary
A new generation of open models takes a smarter approach, and Qwen3.5 is one of the most elegant examples we have worked with.
Instead of making every layer remember every word, most of Qwen3.5's layers use a design called Gated DeltaNet. These layers keep a running summary of what they have read, updated token by token, much like a person who does not memorise every sentence of a book but holds a clear picture of the story so far. That summary stays the same size whether the conversation is one page long or five hundred.
A smaller number of standard attention layers stay in the mix, placed where precise recall matters. The result is a hybrid: the efficiency of a running summary for most of the work, and the sharp memory of attention where it counts.
The practical effect is large. A hybrid model needs only a fraction of the conversation memory that a pure transformer of the same shape would need, so long context stops being a luxury reserved for the most expensive GPUs. It is a beautiful piece of design, and it points to where open models are heading.
A Smart Model Needs an Engine That Understands It
A clever model design only pays off if the software running it understands that design.
Much of today's inference software grew up in the transformer era. Its memory management, scheduling and caching were built around one assumption: every layer keeps a KV cache that grows with the conversation. Hybrid models break that assumption. They have two very different kinds of memory living side by side, a growing cache for the attention layers and a fixed-size state for everything else.
Treat both the same way and you waste the efficiency the model was designed to deliver.
How Titan Runs Hybrid Models
Titan is a blazing fast inference engine built for how modern models work. It sits in the middle of the JohnAIC software stack, between Panini (agents and workflows) and Multix (infrastructure and fleet management), and it handles kernels, quantization and model execution. Four ideas shape it.
Two kinds of memory, each handled on its own terms. Titan keeps the growing attention cache and the fixed-size recurrent state in separate, purpose-built pools. The cache is managed in pages that can be shared and reclaimed as conversations come and go. The recurrent state gets a fixed slot per conversation, because that is all it ever needs.
Bookmarks for the running summary. Many requests begin the same way: the same system instructions, the same policy documents, the same knowledge base. With a transformer, an engine can reuse its cached notes on that shared opening instead of reading it again. A running summary is harder to reuse, because it keeps changing as the model reads. Titan takes checkpoints of the state at useful points, like bookmarks, so repeated prompts can jump straight in instead of starting from page one.
Execution planned ahead of time. Launching work on a GPU has overhead, and at the speeds modern models run, that overhead adds up. Titan records the GPU's work for each batch size in advance using CUDA graphs, so serving a request becomes a single, fast replay rather than thousands of small instructions.
Steady under pressure. When demand spikes and memory runs tight, Titan can pause a long conversation and resume it later instead of failing the request.
The result is fast responses, long context and many concurrent users, on hardware that does not need to be the most expensive on the market.
Why This Matters for Enterprise AI
For a bank, a public sector organisation or a pharma company, the question is rarely which model is smartest. It is whether they can run AI on their own data, inside their own walls, at a cost they can justify.
Cloud AI answers the first part of that question and struggles with the rest. Costs rise with every token, sensitive data has to leave the building, and compliance teams are left reconciling both with regulations like India's DPDP Act.
Owning the infrastructure solves the control problem, but only if that infrastructure is efficient. If long-context AI needs flagship GPUs for every workload, private AI stays out of reach for most organisations. An engine that gets more out of every GPU changes that: the same hardware serves more users, handles longer documents and pays for itself sooner.
That is why Titan ships on every JohnAIC compute node. Our customers buy hardware they own, and Titan makes sure that hardware delivers. We believe intelligence is too important to rent. Titan is part of how we make owning it practical.
See Titan on Your Workload
If you are running AI in production, or planning to bring it in-house, our team is happy to walk you through how Titan performs on real traffic and what it would look like on your workload.
Book a call with the JohnAIC team
Published on 28/10/26