What our first NPU embedding test actually proved

Our team got a real piece of AI work running on Intel AI Boost: 38 calls through an embedding encoder, covering 28 passages and ten questions. The resulting search rankings matched the CPU reference's top three results for every question.

That's a useful starting point for a hardware story. We can name the model, show which work ran on the NPU and check what came out. We can also see where a successful piece of computation left a question without all its supporting evidence.

First, what is an embedding?

An embedding is a list of numbers that represents a piece of text. Search software can compare those lists to rank passages against a question. The model used here, all-MiniLM-L6-v2, represents sentences and short paragraphs with 384 numbers and is designed for uses including semantic search.¹

Picture a small collection of notes and a question you want those notes to answer. The system creates an embedding for each passage and for the question, then compares them to choose the most relevant passages. That selection process is retrieval.

Those 384 numbers aren't 384 facts the model has verified. They give the search process something to compare. Retrieving useful material is one part of an answering system; checking whether the material supports the answer is another.

Our test used a fixed collection of 28 passages and ten questions. Each passage and question went through the encoder, giving 38 calls. The team had already marked the relevant passages for each question before inference. That gave the results something concrete to be checked against.

Where the NPU fitted into the job

NPU stands for neural processing unit: hardware that can take on certain neural-network computations. OpenVINO, the software runtime used here, supports directing model inference to the NPU.² The useful question is which part of the application actually went there.

The technical name for that combining step is pooling. Normalization puts the resulting vectors on a consistent scale for comparison. In this test, those steps stayed on the host. The complete process produced 38 vectors, each containing 384 float32 values.

Runtime records identified the NPU as the encoder's execution device, and saved outputs and profiling evidence supported the completed run. That is the basis for saying the encoder ran there. We aren't inferring execution simply because an NPU appeared in a device list.

The division of work matters. This was an embedding encoder running as part of a retrieval workflow. The test did not generate chatbot answers, and it does not show that the entire application ran on the NPU.

The recorded setup

Batch 1 means each encoder call contained one input. The sequence length set the fixed input size. The longest passage or question fitted within that size, so this particular test did not have to cut off any input text.

The successful NPU stage was recorded from 16:40:22.657 to 16:40:24.877 UTC on October 25, 2026—about 2.22 seconds. That is a timestamped observation of a short stage. It isn't a promise that an application, a large document collection or your own computer will complete its work in that time.

Did the search find what was needed?

The CPU and NPU returned identical top-three passage rankings for all ten questions. That gives us agreement on this retrieval result despite the different precision settings. It doesn't establish that every numerical value in their vectors was identical.

The more revealing check was completeness. Nine of the ten questions retrieved every required passage within the first three results. The remaining question needed several pieces of evidence, and one necessary passage landed in fourth place.

The recorded mean Recall@3 was 96.67%. For this measure, the team checked the fraction of each question's labeled relevant passages found in its first three results, then averaged across the questions. It is a retrieval-coverage score for this test set. It is not a percentage of correct chatbot answers.

That distinction brings us back to the milk story. Returning something that looks relevant can feel like success. But if the job required a particular kind of milk and a date check, finding a carton leaves work unfinished. Here, the missing fourth-ranked passage was the evidence the search still needed to bring back.

A retrospective look at the same saved vectors found that keeping four results recovered all required passages for all ten questions, without a new model run. It added one candidate slot per question, including unlabelled passages. That candidate needed checking on new questions.

Later holdout: owner report and technical review

The experiment owner reported that a separate frozen batch completed at 18:53 UTC on October 25: 30 passages, ten questions and 40 NPU inferences. Reported top-three recall was 0.8167, with complete evidence for 6/10 questions; top-four recall was 0.8667, with 7/10 complete. Both failed the quality gates set before the test, so top four was not promoted.

The publication team first read the experiment owner's report, received at 18:56:30 UTC. A subsequent review by the Local LLM technical desk checked the returned run evidence and recomputed the saved-vector rankings, confirming the reported results and both failed quality decisions. These are results for a separate frozen fixture, not revised scores for the original pilot. The publication team did not perform a new inference run or repeat that technical verification itself.

Why this is a functional result, not a speed verdict

In the original 28-passage pilot, the CPU reference used FP32, while the NPU run requested FP16—different numerical precision settings. The runs also happened at different times, with different cache and setup conditions. A model transfer was active during the NPU stage. Those differences matter when comparing elapsed time.

For the original 28-passage pilot, the returned evidence did not include continuous background-load, power or thermal measurements. It therefore supports no claim that this NPU route was faster, used less energy, saved money or remained reliable under sustained work. Establishing those benefits would take measurements designed to answer those questions.

For a buying decision, the next useful evidence would cover the workload you actually care about: how long it takes to become ready, how long useful work takes, whether the result meets its requirements and what resources it uses. This small test establishes a working route through the recorded model and software configuration.

What you can take from it

The underlying test collection is private. This note reports its aggregate results and methods; it does not provide the inputs needed to reproduce the exact scores. A shareable reproduction package would be separate work. If its passages, questions or labels changed, it would need new measurements rather than inheriting these numbers.

When you read an “AI runs on this hardware” claim, look for three things:

• The work: the exact model and task.
• The execution: evidence showing which component ran on which device.
• The outcome: a check that the result met the task's requirements.

Our first NPU embedding result gives us all three at a modest scale, including a visible limit in evidence coverage. The encoder completed its work. The retrieval rankings matched the CPU reference. One question still needed another passage. The later owner report reinforces why each new question set needs its own quality check.

Public technical references

1. Sentence Transformers, all-MiniLM-L6-v2 model card. Supports the explanation of embeddings, the model's 384-dimensional output and its intended search use. It is not the source of this team's test results.
2. Intel, OpenVINO NPU Device documentation. Supports the explanation of NPU offload and explicit device selection. It does not independently certify this test.