Visual motion computation is challenging under real-world conditions due to continuous contextual changes such as varying lighting conditions and a large range of optical material properties. Due to these changes the retinal optical flow can drastically vary while the physical motion of an object remains constant. Especially materials with high reflective and refractive interactions can cause complex motion patterns. Here we investigate object motion constancy across various optical contexts and if the human visual system compensates for other causal sources in motion. We performed two experiments. In the first experiment observers had to estimate which of two stimuli was rotating faster around the vertical axis. The stimuli were displayed for 500 ms in a 2-IFC staircase design. For the Match stimulus the illumination, material properties and shape were constant. The stimulus was rendered at a high temporal resolution allowing for small rotational speed changes for the staircase design. The Test stimuli varied in ten optical properties (e.g., matte, glossy, anisotropic, translucent), three illumination maps (sunny, cloudy, indoor), and three shapes (knot, cubic, blobby), the rotational speed remained constant. There were three different conditions in the second experiment: 1. unmasked Match and Test stimulus (same as experiment one); 2. masked Test stimulus (circular gaussian mask, masking outer shape contours); 3. masked Test stimulus and masked Match stimulus where the Match stimulus was replaced by horizontally moving 2D pink noise. In this experiment a subset of the optical conditions was used. Expanding on our previously presented work [1], we applied three image-based motion capturing models (Figure 1) to gain deeper insights on motion cues that are predictive of human judgements. The models are Lucas-Kanade (optical flow), RAFT (optical flow DNN), FFV1MT (motion energy). First, we found that there are clear illusory differences of perceived rotational speed with even bigger effects when the circular mask was applied. The transparent material with the refractive index of water is systematically perceived to be rotating faster than other materials across all conditions. We performed an RSA (representational similarity analysis) to compare a range of different metrics across conditions and flow models. We find that the gradient of the optical flow is a particularly good predictor of human performance. The gradient emphasizes local speed changes in the optical flow, for example with moving highlights. Another observation is that Lucas-Kanade is most predictive of human performance under most conditions while RAFT is most stable across materials and closest to the ground truth. Our results further suggest that the human visual system does partially compensate for motion flow effects across optical contexts in object motion. [1] Van Assen, J. J. R., Kawabe, T., & Nishida, S. Y. (2020). Object motion and flow variance across optical contexts. Journal of Vision, 20(11), 458-458. This work has been supported by a Marie-Skłodowska-Curie Actions Individual Fellowship (H2020-MSCA-IF-2019-FLOW) and by JSPS Kakenhi JP20H05957.@en

Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights’ coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.

@en

A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flipping or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. However, a stylized photograph is not quite the same as an artist-created painting. Artists depict perceptually meaningful cues in paintings so that humans can recognize salient components in scenes, an emphasis which is not enforced in style transfer. Therefore, we study how style transfer and paintings differ in their impact on model robustness. First, we investigate the role of paintings as style images for stylization-based data augmentation. We find that style transfer functions well even without paintings as style images. Second, we show that learning from paintings as a form of perceptual data augmentation can improve model robustness. Finally, we investigate the invariances learned from stylization and from paintings, and show that models learn different invariances from these differing forms of data. Our results provide insights into how stylization improves model robustness, and provide evidence that artist-created paintings can be a valuable source of data for model robustness. Code and data are available at: https://github.com/hubertsgithub/style_painting_robustness@en

The world around us is filled with materials. Our ability of visual material perception informs us how to navigate and interact with our environment. It tells us, for example, whether food is fresh, if a chair is strong enough to sit on, how much force to use to pick up a glass, etc. Painters have studied how to depict the world and the materials therein for thousands of years. We believe that the material depictions within paintings can be leveraged into insights for the scientific understanding of material perception. In this thesis, we studied the perception of painterly depictions of materials and aimed to make the study thereof accessible to other researchers with the release of the Materials In Paintings dataset. We collected a large set of paintings from museums and galleries. Then, we used an online crowd-sourcing approach to annotate material identity (fabrics, stone, etc.,) and gather spatial material segmentations (i.e., “cutting out” piece of the painting that depict the material). In the first study, we measured the perception of material attributes (soft, rough, fragile, etc.,) across a range of materials and found that painterly materials trigger distinct distributions of perceived attributes and we furthermore compared these distributions to those for photographic materials. In the second study, we continued crowd-sourcing annotations on material identity and material segmentations and combined these into the Materials In Paintings dataset. In a number of cross-disciplinary demonstrations we presented novel findings across art history, human perception, and computer vision. While these demonstrations are useful in their own right, the main focus here was the release of the dataset. Next, we used the dataset as a source of stimuli for two studies into specific materials. First, for fabrics, we studied the perception of satin and velvet and the effect of presenting only local or, both local and global information, and found that the perceptual distinction between these two fabrics becomes more ambiguous when removing global information. Furthermore, we showed that local image cues can affect perceptual responses for shininess but not for softness. Lastly, we studied the perception and depiction of pearls by identifying three image features that might trigger the perception of pearliness. In a series of experiments, we confirm the role of these image features but find that the presence of only one of these image features, highlights, is already sufficient for naive participants to trigger the perception of pearliness. Conversely, expert participants (art historians or pearl experts) perceive depictions with all three features as more pearly, which implies the existence of visual expertise for pearl perception. All in all, in this thesis we show the benefits of studying material perception through painterly depictions of materials and enable further study with the release of the MIP dataset.@en

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

@en

Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In particular, we find that interactive segmentation tools can be used to cleanly annotate polygonal segments within paintings, a task which is time consuming to undertake by hand. We also find that FasterRCNN, a model which has been designed for object recognition in natural scenes, can be quickly repurposed for detection of materials in paintings. Second, we show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images. We find that training on paintings instead of natural images can improve the quality of learned features and we further find that a large number of paintings can be a valuable source of test data for evaluating domain adaptation algorithms. Our experiments are based on a novel large-scale annotated database of material depictions in paintings which we detail in a separate manuscript.

@en

Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction-the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.

@en

The human face is a popular motif in art and depictions of faces can be found throughout history in nearly every culture. Artists have mastered the depiction of faces after employing careful experimentation using the relatively limited means of paints and oils. Many of the results of these experimentations are now available to the scientific domain due to the digitization of large art collections. In this paper we study the depiction of the face throughout history. We used an automated facial detection network to detect a set of 11,659 faces in 15,534 predominately western artworks, from 6 international, digitized art galleries. We analyzed the pose and color of these faces and related those to changes over time and gender differences. We find a number of previously known conventions, such as the convention of depicting the left cheek for females and vice versa for males, as well as unknown conventions, such as the convention of females to be depicted looking slightly down. Our set of faces will be released to the scientific community for further study.@en

Light takes just a short moment to travel from one point to another and to reach our eyes. Always immersed in the experience of light, we tend to take this for granted. @en