Redefining the capabilities of the smart display with comprehensive, AI-enabled SoCs

The smart display – an old idea is reappearing as an exciting new segment in consumer, industrial and enterprise computing. Driven by new technological innovations in areas such as AI, wireless connectivity, touch, video processing and display drivers, the screen interface is evolving into a multifunctional control point that promises flexible computing – local and connected – high security, and more intuitive and efficient ways to interact with a wider range of devices.

Some definitions

Let’s start by defining the terms. New smart displays exist on a spectrum, from the smallest to the most sophisticated. At the modest end are screens that can run a few specific apps, support WiFi connectivity, and securely connect to a nearby laptop.

At the ambitious end, we see systems that can run some apps locally and connect to the cloud for other apps. But these devices are also aware of their surroundings, can identify and authenticate users through facial recognition and other biometric elements, and can create adaptive and immersive environments for individual users up to awesome augmented reality ( AR). To see this idea in action, let’s take a look at a few specific use cases in realistic consumer, industrial, or business environments.

Some examples in consumer and industrial environments

The huge change in consumer computing behavior since the start of the COVID 19 pandemic has swept away old notions of home computing. Today, consumers work from home. They study from home. More and more, they play at home.

Visualize, if you will, this scenario. In that inevitable guest room or corner of the dining table is a smart screen. A family member sits in front of it and the device recognizes the individual, authenticates them via biometrics, and connects them to their application environment both locally and in the cloud.

The device provides local artificial intelligence (AI) acceleration for applications. These applications can be for productivity (like joining a video conference or watching a PowerPoint presentation, or adaptive learning at your own pace), or multimedia (like streaming and watching a movie), or entertainment (like games). interactive and augmented reality (AR) for estimating yoga poses and other sports). When the user brings a laptop, the smart display recognizes it via Bluetooth and connects to it securely and wirelessly.

Industrial devices

Or take another example, not at home, but in a classroom, retail store, or factory. Here, the smart display can be on a desk, in a kiosk, portable with WiFi or 5G connectivity, or built into a machine. Once again, the user walks up and the smart display recognizes them and, if applicable, authenticates them. What happens next depends on the use case.

In an industrial application, the smart screen can for example be a mobile control and service panel. A technician walks up to a machine and the smart display recognizes the machine’s control computer via Bluetooth and connects to it wirelessly. Now the technician is holding a virtual control panel for that machine, capable of observing machine operation, performing AI-guided diagnostics and root cause analysis, and making adjustments or repairs assisted by AR. Via 5G or WiFi, the technician can also invoke a digital twin of the machine and explore the effects of parameter changes, speaking to engineers if necessary.

Similar scenarios that take advantage of identification, connectivity and personalization could occur in education, retail and smart city use cases.

The technological foundations of smart screens

Underneath all of these examples, there must be a very different set of capabilities that are very different from the electronics of a conventional monitor. These differences relate to three areas: media processing, AI inference acceleration, and wireless docking.

The heart of the smart display is a media processing engine. Far from simply transferring the incoming video to a display, the smart display provides a sophisticated graphics processing unit (GPU) and video decoder. But the smart display must also be able to accept input from multiple microphone channels, to enable the user’s voice to be isolated from background noise, and from one or more cameras. This involves processor bandwidth to perform multi-channel audio processing, as well as a multi-stream video processing and encoding pipeline. To stay within the cost envelope of a consumer device, these capabilities must be integrated into a single advanced SoC.

Two areas in particular merit further examination. One is video processing. In order to support the wide range of use cases that we have described, from e-learning video streaming to managing multi-video conferencing with live and streaming content, it is necessary to decode multiple streams. video, up to a total budget of 4K UHD 120 Hz progressive scan. Likewise, encoding of multiple video streams up to a total budget of 1080P 120 Hz is required. In service of this video codec, a dual display controller can be used to drive two displays simultaneously, with features such as extend and mirror. And — needed to provide the memory bandwidth for this video traffic and the other use cases we talked about — it would need sufficient memory (eg 64-bit at least)

Second, and just as important is the ability to make local AI inferences via an optimized Neural Network Processing Unit (NPU) that is small enough to be just a block on the SoC. An appropriate solution would provide a framework alongside the NPU that would allow model development, training and porting to the underlying SoCs. This would allow models to be safely inserted into the video pipeline, allowing the processor to analyze the camera input and post-process the decoded video, for example, to identify people or objects in front of the monitor. and to perform super-resolution scaling.

A complete solution

A full smart display platform certainly requires media processing and AI acceleration. But it also requires all the other blocks that go into a compelling implementation of the use cases we’ve discussed: high-performance software execution, graphics processing, peripheral controllers, and memory interfaces. And that requires SDKs, sample apps, and support to get a smart displays vendor to market quickly. Equally important, a complete solution integrates security from the start, not the end.

The opportunity for smart displays is large, diverse, and will grow rapidly, with many companies potentially underestimating the difficulty of a complete, secure solution and getting started quickly. In such an environment, a silicon and solutions partner already in the market with a proven SoC, application libraries and a hardware-based security approach that has withstood the scrutiny of the lobby demanding protection. video content can be an invaluable asset in gaining rapid market access with a differentiable and winning product. (ZD)

Thanks to Synaptics for releasing this article outside of the Display Daily pay wall so that it does not count as one of your two free articles, if you are not a subscriber.

Zafer DIAB – Product Marketing Director – Edge SOC

Zafer joined Synaptics in March 2018 and serves as Director of Product Marketing for the Edge AI SOC business line. Prior to Synaptics, Zafer held various product marketing, business development and sales positions at Amlogic, ST Microelectronics and Zenverge. Prior to moving to commercial roles, Zafer led software engineering at Broadcom and Conexant. He obtained his bachelor’s and master’s degrees in electrical engineering from École Polytechnique de Montréal in Canada.


About Michelle T. Friesen

Check Also

How to Mirror Your iPhone to a Samsung Smart TV

With a little setup, your iPhone screen can be mirrored to a much larger Samsung …