Recovering the appearance and physical parameters of elastic objects from multi-view video is essential for many applications that require simulation of the real world. Past methods for this task have provided accurate results in recovering physical properties; however, their re- liance on Neural Radiance Fields (NeRFs) for novel view synthesis means they trade off visual quality for rendering speed. To address this issue, we present a novel framework for the joint appearance reconstruction and physical parameter estimation of elastic objects relying on 3D Gaussian splatting. Our key insight is that dynamic 3D Gaussian kernels extracted from multi-view video can be used to reconstruct the object’s geometry and supervise elastic parame- ter fitting through a differentiable physics engine. Novel views and object behaviours can then be constructed by forward simulating the extracted mesh and using it to drive the Gaussian kernels. We demon- strate that our method is competitive with the state of the art on phys- ical parameter estimation while being better at reconstructing object appearance. Additionally, our method can simulate novel views and object interactions at near real-time rates that outperform past ap- proaches.

The objective of this project is to train a model that transforms a tree with its foliage into only its branch structure. This is achieved by employing machine-learning techniques, specifically Generative Adverserial Networks (GANs). By utilizing the proposed method, a predictive model is built that automatically minimizes its own error function through a comparison of a set of input and ground-truth tree images, which are tree images with and without leaves, respectively. The adoption of GANs has shown promising results, both visually and metrically.

The most established and widely used methods for analysing tree images for tasks such as geometry analysis, segmentation and classification often rely on pixels. In this paper, the applicability of analyzing tree geometry based on a graph representation rather than a pixel-based approach is pursued. To do so, 2D renders of different species of trees are converted to spatial graph structures capturing significant points on the tree skeleton. Two independent Graph Convolutional Network algorithms which learn node (coordinate) features are then applied on the obtained dataset to assess the reliability of graph based analysis. The first experiment explores a GCN for assigning correct species labels to the skeleton graph of the original tree image, demonstrating the association between geometry and tree metadata. The second experiment, an unsupervised representation learning, is conducted by using Graph Autoencoders to obtain an embedding for each skeleton graph which can be used to reconstruct partially the same graph, demonstrating the association between GCE latent representation and geometry. Promising results were found in both cases, reinforcing the reliability of the original proposition to rely on geometry as well as pixels for tree analysis tasks.

L-Systems allow for the efficient procedeural generation of trees to be used for rendering in video games and simulations. Currently, however, it is difficult to engineer grammars that mimic the behaviours of real life trees in 3 dimensions. To be able to deduce them, the skeleton of a tree can be used to train a model and generate an L-system for a given tree in particular. The aim of this paper is to provide a pipeline to isolate these skeletons from images of a tree, using Neural Radiance Fields (NeRFs) to reconstruct the tree, and using Laplacian Based Contraction to retrieve the underlying skeleton. We find that this approach leads to 3-dimensional topologies that very closely resemble the given tree.

Trees are essential components of both real and digital environments. Therefore, it is important to have 3D models of trees that are of high quality and computationally efficient. One way to achieve this is by compressing a high-quality model using billboard rendering, which involves partitioning the tree into multiple planes to produce a similar result to the original. Our study explores the compression of 3D models using an optimization loop and adapting billboard rendering techniques. We use computer vision primitives to render basic models, which we then optimize by adjusting the texture to resemble the original tree. The models consist of multiple upright planes that are rotated around the central vertical axis of the original tree. We use different optimization functions, such as L1 and L2 losses, to determine the best approach. We can improve the initial models by bounding the billboards and limiting their heights and widths to that of the trees. Additionally, we can use double-sided textures for the billboards to allow more flexibility for optimizing different species of trees. However, optimizing multiple tree types performs differently for each species, leading to improvements that only benefit certain trees in specific scenarios. Using quantitative metrics, we determined which models perform best and how similar they are to the original after training. We found that our compressed models generally resemble the original while having only a fraction of the original size.

Image inpainting is a problem that has been well studied over the last decades. In contrast, for 3D reconstructions such as neural radiance fields (NeRFs), work in this area is still limited. Most existing 3D inpainting methods follow a similar approach: they perform image inpainting on the training images and use the inpainted images for further training of the 3D model. Due to inconsistencies in the different inpaintings of the images, the 3D inpainting often becomes blurry. With the advent of 3D Gaussian Splatting (3DGS), we identify a new opportunity for 3D inpainting. As 3DGS is more explicit in nature than NeRF, we can manipulate the 3D Gaussians directly rather than relying on image inpainting. Based on that key idea, we propose a method that works similar to the PatchMatch image inpainting algorithm. We first construct a nearest-neighbour field (NNF) by searching for nearest-neighbour patches throughout the scene that look similar to the area we want to inpaint. After constructing the NNF we copy the contents of the nearest-neighbour patches to the inpainting region and blend them together to obtain the inpainting result. In our experiments we found that our method performs well in terms of texture synthesis but struggles with structure synthesis, similar to the original PatchMatch algorithm. In cases where only texture synthesis is required to inpaint the area our method is able to provide good results, although in some cases pre-processing of the scene is necessary, as we found that better quality inputs (e.g. the scene itself, the surface mesh underlying the scene, and precise masks) drastically improve the results of our method. Moreover, some parameters of the algorithm are highly scene-dependent and by tailoring them to the scene we can further enhance the performance of the algorithm. Besides introducing a 3D inpainting method that directly manipulates the scene contents, our work offers valuable new insights into 3DGS editing in general.

Radiance fields are a promising alternative to conventional 3D representations in the domain of novel view synthesis, with recent research achieving truly impressive photorealistic view synthesis results. In this paper, we deal with the concept of non-photorealistic rendering in the context of radiance fields, for generating more stylistic captures of real-world objects. We discuss the suitability of inverse rendering as a stepping-stone to traditional shading and edge detection, and contribute a new algorithm specifically for outline detection in radiance fields.

Neural Radiance Fields (NeRF) and their adaptations are known to be computationally intensive during both the training and the evaluating stages. Despite being the end goal, directly rendering a full-resolution representation of the scene is not necessary and not very practical for scenarios like streamed applications. Our goal is to design a streamable adaptation for a model that can produce fast, rough estimates of 3D scenes, by only using a shallow part of the network. The quality is subsequently improved as more parts of the network are available, such that it can be used in online applications where the model needs to be transferred. Separate models can be trained at different resolutions, but this approach results in a large space overhead and also increases the evaluation time. This can be mitigated by reducing the depth of low-resolution models, but redundancy will still be high as each new model needs to re-evaluate the input data, rendering previous calculations obsolete. Our method combines key concepts from previous approaches to create a progressively trained model that is able to produce intermediate outputs of increasing quality while attempting to optimize the trade-off between overhead and quality. Our model is able to produce a recognizable representation of the scene with as little as one hidden layer from the original model. It also allows for division into streamable chunks which can be sent individually and, upon reconstruction, provide intermediate outputs that bring consistent improvement in quality. The newly streamed data uses the residual output from previous computations in order to reduce redundancy. We show that the final quality of our adaptation is within 2% of the original in terms of previously used quantitative metrics.

Neural radiance fields (NeRF) based solutions for novel view synthesis can achieve state of the art results. Recent work proposes models that take less time to render, need less training data or take up less space. However, few papers explore the use of NeRFs in classic rendering scenarios such as rasterization, which could contribute to wider adoption. Our paper tackles the issue of shadow generation and proposes a deep residual MLP network with fast evaluation times, that generates view-dependent shadow maps. The network distills the knowledge of an existing NeRF model and achieves the speedup through the use of neural light fields, by only doing one network forward per ray.

With the current state-of-the-art research, exporting a NeRF to a mesh has the side effect of having to evaluate a Multi Layer Perceptron at render-time, causing a significant decrease in performance. We have found a way to use K-Means clustering to pre-compute values for this MLP, storing them in multiple octahedron maps for the GPU to fetch when it's time to render the object. This improves render times by a factor of 3-4x.

Chromastereoscopy makes use of special glasses to make its wearer see two slightly different images, one for each eye. With a properly created image this results in perceived depth. It does this by bending light dependent on its wavelength and in the opposite direction for each eye. When a ChromaDepth image is properly created most viewers perceive the image as having a 3D effect. However, since the glasses act on wavelengths and not on perceived colors, different display media like screens and printers could result in a different perception of the images through the glasses. To investigate this possible difference I measured reflectance spectra of printed colors, and created an algorithm to simulate the effect the glasses have on images made with mixes of these colors. Additionally I tried multiple visualization techniques to digitally show the effect including depth.

The following paper explores the use of chromostereopsis for the purpose of enhancing the perceived depth in photos. Contrary to previous work, this is achieved by using a continuous depth map as input and creating a natural-looking photo with enhanced depth even when viewed without any type of three dimension technology. This is the first algorithm to achieve this effect by shifting the hue and shows that it is possible to exploit chromostereopsis for depth enhancement by solely changing the hue values. We perform a user study to examine the properties of naturally occurring chromostereopsis, with focus on the reversal effect caused by the lightness of the background. The results of the user study are used in the algorithm, which can be applied to any photo with a depth map. We show the results and analyse them on six example images that highlight limitations and strengths of the algorithm.

Chromostereopsis is an optical illusion that allows 2D images to simulate depth based on color. For example, red is perceived to be closer than blue, when displayed on a black background. This effect is present because of the slight chromatic aberration caused by the eye lens, and it can be strengthened by the use of ChromaDepth® glasses. This research looks into how the chromostereoscopic effect can be enhanced by precisely controlling the used color spectra by simulating the effect with the use of stereo projection. It was concluded that hyper-spectral projection can yield a stronger chromostereoscopic effect and better depth resolution of the image, but its full potential is still limited by the visible spectrum of light.

Chromostereoscopic images encode depth as colour, with the red part of the visible spectrum encoding nearby depths and the blue part encoding far-away depths. However, when encoding a regular image with its depth, the generated colours may not match with the original colours in the image. Following a user study, a technique has been developed which takes both the chromostereoscopic effect and the original colours into consideration.

This problem was solved by creating a program to generate chromostereoscopic images from an arbitrary RGBD input image. An optimisation metric was devised to assess if a chromostereoscopic image takes both the target depth and original image into consideration. A user study was conducted to correct this metric with respect to human perception. The new technique can take a desired amount of original colour and chromostereoscopic depth and will generate a chromostereoscopic image, taking these preferences into account.

Chromostereopsis is a visual illusion that can produce perceived 3D images through an effect caused by how humans see different wavelengths. ChromaDepth glasses may be worn to exploit this phenomenon and amplify the effect. Relative to other forms of stereoscopy, chromostereoscopy is lesser-known and has seen fewer applications. This paper seeks to address this, by investigating possible solutions to augment any arbitrary 3D scenes for chromostereopsis. We present techniques to apply the effect in real-time to 3D scenes, while maintaining the scene's original shading, lighting, and optionally some degree of original colour. Additionally, we propose concepts for interfaces that allow user adjustment to applying the effect in areas where user studies have shown variance in preferences. These range from curves that determine varying depth precision, to creating custom chromostereoscopic colour maps, and modifying the speed at which the depth-colour mapping changes, for those sensitive to stereopsis.