11

Video AI can now describe a scene in plain language and find a single moment in hours of footage. It’s easy to assume that better models are what turn store cameras into better decisions, but in retail, much of the hard work happens after the model. This is the first of a multi-part blog series on putting video AI analytics to work, for retailers and for the teams building the applications retail stores rely on. It starts with the gap between what a camera sees and what a retailer needs to know.

A video event can be either routine or important, depending on information the camera cannot see.

At the Scality Day 2026 event in Paris, I opened a talk called “Make the real world searchable” to show what video AI can now put into words. For example:

At a supermarket self-checkout, a vision-language model can describe an item passing over the scanner without being scanned.

I’ve spent my career in video infrastructure and streaming products, where the discipline is to follow the experience all the way to the person using it. I bring the same discipline to retail video AI. Even the best description has limits. At the self-checkout, the model cannot know whether the point-of-sale system recorded the item, whether an employee intervened, or whether the transaction was cancelled. That gap between seeing and knowing is where retail video analytics becomes an application. A video AI system can describe a scene, find a similar event, or raise an alert, but a retailer still needs a product that connects the result to its transactions, inventory, planograms, access policies, or warehouse processes, and then puts a decision in front of whoever has to act on it.

What video alone cannot tell a retailer

Video alone does not contain the retailer’s transaction records or operating rules. Retail operations run on products, transactions, locations, schedules, policies, and exceptions, and almost none of that is visible in a frame of video. 

An apparently empty shelf may need restocking, or it may be intentionally empty because the planogram changed. An item moving past a scanner may indicate an exception, or it may already appear elsewhere in the transaction. A pallet left at a loading dock may be delayed, or it may be exactly where the warehouse-management system expects it to be.

The model cannot resolve those questions from video alone, so the application has to combine what the video observed with the business record and the retailer’s own policy, and then decide: ignore the event, collect more evidence, queue it for review, or act.

The dividing line is between what the video shows and what the business knows. General video capabilities can be shared across retailers and across industries: ingestion, recording, vision-language analysis, search, and alerting. Those shared capabilities still have to be connected to each retailer’s data and adapted to its operating policies.

Why a wrong answer costs more than it looks

The cost of a wrong answer is lopsided. A missed event may mean a loss, while a false alarm wastes an employee’s time, can disrupt a customer and, if repeated often enough, erode trust in the system that raised it. The test that matters is whether the whole system, from lens to decision, is accurate enough for the retailer’s operating process.

Reliable decisions depend on model performance, the quality and freshness of the business context, and the workflow around both. Retailers should evaluate missed actionable events, unnecessary interventions, time to resolution, and the employee attention required for each issue successfully resolved. The application must bring the relevant evidence and business record into the same review process and record the outcome.

What the application adds in each case

Self-checkout. The point-of-sale record provides context the video may lack, such as earlier scans, voids or recorded employee interventions. An application aligns the video event with that record, applies the retailer’s exception rules, decides when a person needs to look, and shows the clip and the transaction together.

Shelf availability. The planogram says which product belongs in the gap, inventory records indicate whether stock should be available elsewhere in the store, and task data shows whether a restock is already underway. An application combines those to tell a genuine availability problem from a planned layout change or an action already in progress, and creates or prioritizes the restocking task only when somebody needs to act.

Loading dock. Warehouse-management records provide arrival information, the pallet’s expected destination and the timing needed to assess an operational delay. An application ties the observation to the receiving workflow, identifies the shipment, and routes the exception to the team responsible.

Business records can also be incomplete or out of date. A late transaction, a stale inventory count or an uncertain shipment match may leave the application without enough information to act. It must reconcile the available evidence and decide when to wait, request review or proceed.

From alert to evidence

An alert only starts a decision, so a retail product needs more than a notification. It needs a way to review the event, connect it to the relevant business record, log what was decided, and improve the operating rule over time. Consider a hypothetical checkout review: an employee opens an alert with the relevant clip and transaction displayed together, confirms that the item was scanned earlier, and closes the case with the reason recorded. The workflow should let the employee complete that review without searching across several systems. The video AI system raised the alert; the application supported the review and recorded its outcome.

The recorded footage is part of that process, allowing an employee to inspect what led to an alert and letting the retailer revisit the event later when policy permits. For that review to work, the application must retrieve the relevant sequence promptly, align it with the business record, and show enough footage before and after the event to make the situation understandable. That puts a requirement on the storage underneath: the source material has to stay under the retailer’s own retention controls for as long as the review process and the retailer’s policy require, and no longer.

Where the context lives

The systems that give video its meaning sit under the retailer’s control, and so does the footage, which contains customers and staff and, in Europe, is subject to data-protection law. Deployment should reflect where the business systems run, how quickly staff need to act, and what happens when connectivity is interrupted. Local video processing can be useful when response time, connectivity constraints or the volume of footage make remote processing impractical. The appropriate balance between site, data centre and cloud depends on the use case.

Where the application begins

For a retailer choosing a video AI application, the practical question is: how well does the vendor support the retailer’s operating process? The application vendor’s responsibilities include:

• Connecting video observations to systems such as point of sale, inventory, planogram management, access control, or warehouse management.
  
• Defining what an event means for that retailer, including thresholds, exceptions, escalation rules, and when human review is required.
  
• Building the operator experience that presents the alert with the business context and supporting footage needed to make a decision.
  
• Measuring outcomes and feeding confirmed or rejected events back into the application’s rules and models.

Shared video infrastructure reduces the amount an application vendor has to assemble, but it does not remove the need for domain knowledge, workflow design, or responsibility for application-level accuracy.

The infrastructure captures, describes and retains the video. The application is where the business decides what an event means and what should happen next.

The next article in this series, “Why video AI belongs at the edge,” looks at where video processing and retained footage should live, and how response time, connectivity and operational control shape that choice.

Building the application

If you develop video AI applications for retail, or other sites where footage must stay on premises, Scality and its partners offer a validated on-premises stack for video understanding, storage and retention, so your team can focus on the decisions your customers need to make. See how the program works.