Publications

Efficiency and efficacy are desirable properties for any evaluation metric having to do with Standard Dynamic Range (SDR) imaging or with High Dynamic Range (HDR) imaging. However, it is a daunting task to satisfy both properties simultaneously. On the one side, existing evaluation metrics like HDR-VDP 2.2 can accurately mimic the Human Visual System (HVS), but this typically comes at a very high computational cost. On the other side, computationally cheaper alternatives (e.g., PSNR, MSE, etc.) fail to capture many crucial aspects of the HVS. In this work, we present NoR-VDPNet++, a deep learning architecture for converting full-reference accurate metrics into no-reference metrics thus reducing the computational burden. We show NoR-VDPNet++ can be successfully employed in different application scenarios.

Close

In this tutorial, we introduce how the High Dynamic Range (HDR) imaging field has evolved in this new era where machine
learning approaches have become dominant. The main reason for this success is that the use of machine learning and deep
learning has automatized many tedious tasks achieving high-quality results overperforming classic methods. After an introduction to classic HDR imaging and its open problem, we will summarize the main approaches for merging of multiple exposures,
single image reconstructions or inverse tone mapping, tone mapping, and display visualization.

Close

Deep learning-based Single Image Super-Resolution (SISR) has recently provided great performances when compared with state-of-the-art approaches. However, these performances are usually at the cost of high computational complexity and memory management, even at the inference stage. In this paper, we aim to reduce the structural complexity of a state-of-the-art Deep Neural Networks (DNN)1 approach to propose a cost-effective solution to the problem of SISR. We have investigated how the different components of the model (baseline) may affect the overall complexity while minimizing the negative effect on its quality performances. This has provided a solution, which yields quality performances comparable to the baseline model, while approximately reducing more than one order of magnitude the number of parameters, the spatial complexity (GPU memory) up to 1/6 and inference time by 1/2.

Close

This paper describes the ongoing activities of the Enhanced Video coding (EVC) project of the Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI). Theproject investigates how the performances of existing codecs can be improved by enhancing or replacing specific encoding tools with AI-based counterparts. The MPEG EVC codec baseline profile has been chosen as reference as it relies on encoding tools thatare at least 20 years mature yet has compression efficiency close to HEVC. A framework has been developed to interface the encoder/decoder with neural networks, independently from the specific learning toolkit, simplifying experimentation. So far, the EVCproject has investigated the intra prediction and the super resolution coding tools. The standard intra prediction modes have been integrated by a learnable predictor: experiments in standard test conditions show rate reductions for intra coded frames in excess of 4% over the reference.The use of super resolution, a state-of-the-art deep-learning approach named Densely Residual Laplacian Network (DRLN),at the decoder side have been found to provide further gains, over the reference, in the order of 3% inthe SD to HD context

Close

Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) is the first standards organization to develop data coding standards that have artificial intelligence (AI) as their core technology. MPAI believes that universally accessible standards for AI-based data coding can have the same positive effects on AI as standards had on digital media. Elementary components of MPAI standards–AI modules (AIMs)–expose standard interfaces for operation in a standard AI framework (AIF). As their performance may depend on the technologies used, MPAI expects that competing developers providing AIMs will promote horizontal markets of AI solutions that build on and further promote AI innovation. Finally, the MPAI framework licences (FWLs) provide guidelines to intellectual property right (IPR) holders facilitating the availability of compatible licenses to standard users.

Close

Building and deploying video pipelines is a task currently relegated to low level integration of specialized libraries, requiring thus a deep understanding of component functionalities and communication protocols. As a result, field experts, when undertaking this task, they often find the process laborious, filled with continuous coding checks and lengthy testing times. Introducing graphical visualization into the task can massively benefit imaging experts to view pipeline construction through a simpler lens, offering them a greater insight into the structure of their pipeline as well as better control over their model’s workings. With this idea in mind, we provide an innovative solution through a GUI pipeline fabricator, called DgiStreamer, which allows easy construction and deployment of imaging pipelines via an effective and user-friendly interface, without writing a single line of code.

Close

MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence is the first body developing data coding standards that have Artificial Intelligence (AI) as its core technology. MPAI believes that universally accessible standards for AI-based data coding can have the same positive effects on AI as standards had on digital media.
Elementary components of MPAI standards – AI Modules (AIM) – expose standard interfaces for operations in a standard AI Framework (AIF). As their performance may depend on the technologies used, MPAI expects that competing developers providing AIMs will promote horizontal markets of AI solutions that build on and further promote AI innovation.
Finally, the MPAI Framework Licenses provide guidelines to IPR holders facilitating the availability of compatible licenses to standard users.

Close

The intelligent method proposed herein is formulated on a deep learning technique which can identify, localise and map the shape of plastic debris in the marine environment. Utilising images depicting plastic litter from six beaches in Cyprus, the developed tool pointed to a plastic litter density of 0.035 items/m2. Extrapolated to the entire shorelines of the island, the intelligent approach estimated about 66,000 plastic articles weighting a total of ≈1000 kg. Besides deducing the plastic litter density, the dimensions of all documented plastic litter were determined with the aid of the OpenCV Contours image processing tool. Results revealed that the dominant object length ranged between 10 and 30 cm which is in agreement with the length of common plastic litter often spoiling these coastlines. Concluding, only in-situ visual scan sample surveys and no manual collection means were used to predict the density and the dimensions of the plastic litter.

Close

One of the biggest adversaries to the computer vision pipeline is bad weather which can deteriorate the visual quality of the captured images and lead to a decreased performance of tasks. Examples of such tasks can include image classification, object detection and semantic segmentation. To ameliorate this acquisition bottleneck, vision experts have developed restoration approaches which aim to recover the lost visual information due to the presence of poor climactic conditions such as atmospheric haze. The technique of single image dehazing has achieved great strides in producing aesthetically pleasing restorations to the human perception. However, it is important to establish whether these approaches bring the same merits to high-level vision tasks. To this end, we formulate a study around the task of object detection that aims to uncover the underlying relationship between this high-level task and atmospheric haze as well as examine the ability of the current dehazing process to enhance the detection performance. From our experiments we find that while there is a clear negative relationship between hazy conditions and detection performance, there is little help from the dehazing process to achieve the desired haze-free results.

Close

Efficiency and efficacy are two desirable properties of the outmost importance for any evaluation metric having to do with Standard Dynamic Range (SDR) imaging or High Dynamic Range (HDR) imaging. However, these properties are hard to achieve simultaneously. On the one side, metrics like HDR-VDP2.2 are known to mimic the human visual system (HVS) very accurately, but its high computational cost prevents its widespread use in large evaluation campaigns. On the other side, computationally cheaper alternatives like PSNR or MSE fail to capture many of the crucial aspects of the HVS. In this work, we try to get the best of the two worlds: we present NoR-VDPNet++, an improved variant of a previous deep learning-based metric for distilling HDR-VDP2.2 into a convolutional neural network (CNN). In this work, we try to get the best of the two worlds: we present NoR-VDPNet++, an improved versionof a deep learning-based metric for distilling HDR-VDP2.2 into a convolutional neural network (CNN).

Close

Real-world environment, where images are acquired with digital camera, may be subject to sever climatic conditions such as haze that may drastically reduce the quality performance of sophisticated computer vision algorithms used for various tasks, e.g., tracking, detection, classification etc. Even though several single image de-hazing techniques have been recently proposed with many deep-learning approaches among them, a general statistical framework that would permit an objective performance evaluation has not been independently introduced yet. In this manuscript, certain performance metrics that emphasize different aspects of image quality, output ranges and polarity, are dentified and combined into a single performance indicator derived in an unbiased manner. A general methodology is thus introduced, as a framework for objective performance evaluation of current and future dehazing tasks, through an extensive comparison of 15 single image de-hazing techniques over a vast range of image data sets. The proposed unified framework shows several advantages in evaluating diverse and perceptually meaningful image features but also in elucidating future directions for improvement in image dehazing tasks.

Close

The drive for automation and constant monitoring has led to rapid development in the field of Machine Learning (ML). The high accuracy offered by the state-of-the-art ML algorithms like Deep Neural Networks (DNNs) has paved the way for these algorithms to being used even in the emerging safety-critical applications, e.g., autonomous driving and smart healthcare. However, these applications require assurance about the functionality of the underlying systems/algorithms. Therefore, the robustness of these ML algorithms to different reliability and security threats has to be thoroughly studied and mechanisms/methodologies have to be designed which result in increased inherent resilience of these ML algorithms. Since traditional reliability measures like spatial and temporal redundancy are costly, they may not be feasible for DNN-based ML systems which are already super computer and memory intensive. Hence, new robustness methods for ML systems are required. Towards this, in this chapter, we present our analyses illustrating the impact of different reliability and security vulnerabilities on the accuracy of DNNs. We also discuss techniques that can be employed to design ML algorithms such that they are inherently resilient to reliability and security threats. Towards the end, the chapter provides open research challenges and further research opportunities.

Close

HDR-VDP 2 has convincingly shown to be a reliable metric for image quality assessment, and it is currently playing a remarkable role in the evaluation of complex image processing algorithms. However, HDR-VDP 2 is known to be computationally expensive (both in terms of time and memory) and is constrained to the availability of a ground-truth image (the so-called reference) against to which the quality of a processed imaged is quantified. These aspects impose severe limitations on the applicability of HDR-VDP 2 to realworld scenarios involving large quantities of data or requiring real-time responses. To address these issues, we propose Deep No-Reference Quality Metric (NoR-VDPNet), a deep-learning approach that learns to predict the global image quality feature (i.e., the mean-opinion-score index Q) that HDR-VDP 2 computes. NoR-VDPNet is no-reference (i.e., it operates without a ground truth reference) and its computational cost is substantially lower when compared to HDR-VDP 2 (by more than an order of magnitude). We demonstrate the performance of NoR-VDPNet in a variety of scenarios, including the optimization of parameters of a denoiser and JPEG-XT.

Close

Irrespective of how plastics litter the coastline or enter the sea, they pose a major threat to birds and marine life alike. In this study, an artificial intelligence tool was used to create an image classifier based on a convolutional neural network architecture that utilises the bottleneck method. The trained bottleneck method classifier was able to categorise plastics encountered either at the shoreline or floating at the sea surface into eight distinct classes, namely, plastic bags, bottles, buckets, food wrappings, straws, derelict nets, fish, and other objects. Discerning objects with a success rate of 90%, the proposed deep learning approach constitutes a leap towards the smart identification of plastics at the coastline and the sea. Training and testing loss and accuracy results for a range of epochs and batch sizes have lent credibility to the proposed method. Results originating from a resolution sensitivity analysis demonstrated that the prediction technique retains its ability to correctly identify plastics even when image resolution was downsized by 75%. Intelligent tools, such as the one suggested here, can replace manual sorting of macroplastics from human operators revealing, for the first time, the true scale of the amount of plastic polluting our beaches and the seas.

Close

Computer graphics tools and techniques enable researchers to investigate cultural heritage and archaeological sites. They can facilitate documentation of real-world sites for further investigation, and enable archaeologists and historians to accurately study a past environment through simulations. This chapter explores how light plays a major role in examining computer-based representations of heritage. We discuss how light is both documented and modelled today using computer graphics techniques and tools. We also identify why both physical and historical accuracy in modelling light is becoming increasingly important to study the past, and how emerging technologies such as High Dynamic Range (HDR) imaging and physically-based rendering is necessary to accurately represent heritage.

Close

Image metrics based on Human Visual System (HVS) play a remarkable role in the evaluation of complex image processing algorithms. However, mimicking the HVS is known to be complex and computationally expensive (both in terms of time and memory), and its usage is thus limited to a few applications and to small input data. All of this makes such metrics not fully attractive in real-world scenarios. To address these issues, we propose Deep Image Quality Metric (DIQM), a deep-learning approach to learn the global image quality feature (mean-opinion-score). DIQM can emulate existing visual metrics efficiently, reducing the computational costs by more than an order of magnitude with respect to existing implementations.

Close