The digital imaging landscape is about to witness a transformative shift with the development of the “world’s first full AI-based image signal processor (ISP).” Two innovative companies, Chips&Media, a Korean provider specializing in image processing IP, and Visionary.ai, an Israeli startup focused on advancing low-light image processing, have joined forces to create a groundbreaking ISP that replaces traditional hardware-dependent systems.

This collaboration aims to revolutionize how images are formed, moving the entire imaging process from fixed hardware to software that operates on neural processing units (NPUs). This change not only allows for real-time tuning, retraining, and updates to video processing, but it also embodies a response to the evolving demands of the imaging industry. A particular focus is being placed on low-light video, which both companies have identified as a prime candidate for this structural shift.

From Fixed Hardware to Software-Defined Imaging

For decades, ISPs have been an integral part of digital cameras, yet their underlying architecture has seen little innovation. Traditionally, chipmakers create these devices to execute a set of fixed mathematical processes, often leaving little room for flexibility or customization outside the manufacturing phase. The limitations of this traditional approach are becoming increasingly evident in an era where imaging requirements extend from smartphones to autonomous vehicles and even advanced XR (extended reality) applications.

Oren Debbi, co-founder and CEO of Visionary.ai, highlights this breakthrough by stating that, “This is the first full end-to-end ISP pipeline that runs entirely on an NPU, without relying on a hardware ISP at all.” This signifies a departure from conventional systems which often tack on neural network capabilities to existing hardware ISPs. The new system processes RAW sensor data directly via an NPU or GPU, offering substantial room for adjustments in tuning and optimization through over-the-air updates, all while keeping the core silicon components unchanged.

A Major Leap in Machine Learning Integration

At the heart of this innovative approach is sensor-specific training. Visionary.ai has engineered an automated platform capable of producing a custom neural network model in mere hours, utilizing only a small number of short video clips for training. This dramatic reduction in integration time not only simplifies the process but also supports scalability across various sensors and platforms, eliminating the lengthy tuning cycles typically associated with traditional ISPs.

While AI-enhanced ISPs are already prevalent in the realms of smartphones and cameras, both Chips&Media and Visionary.ai assert that current implementations remain overly reliant on hardware. Existing systems usually integrate neural networks as distinct blocks that cannot process RAW data directly. Debbi explains, “The image formation pipeline is neural-first, not a classic ISP with a few AI add-ons.” He further notes that while some traditional camera control functions may persist, the core image pipeline operates independently of fixed-function hardware, marking a significant paradigm shift.

Implications for the Imaging Industry

This development not only enhances the flexibility and performance of image processing but also carries substantial commercial implications. The ability to dynamically tune and optimize image quality through software aligns with the broader trend of hyper-personalization in technology products. The envisioned AI-based ISP is set to cater to a wide array of applications, from consumer electronics to high-performance autonomous systems.

Furthermore, the implications of such technology stretch beyond merely improving image quality. The movement towards a fully software-defined imaging framework signifies a future where adaptability is key, opening doors to a myriad of applications that necessitate real-time image enhancement and processing capabilities.

Overall, the collaboration between Chips&Media and Visionary.ai stands as a compelling example of how AI can fundamentally reshape industries through innovation. Their work not only challenges the established norms of image processing but also sets the stage for a new era of visual technologies that could have profound impacts not only on photography but on all sectors leveraging imaging solutions.