The advancement in transparent screen technology has promoted adoption of full-screen design on mobile devices, reducing the area occupied by optical sensors to maximize the devices' screen-to-body ratio. In modern smartphones, front-facing optical sensors, such as ambient light sensor and camera, now must be placed under the transparent screen to capture ambient light and visual information. Motivated by this trend, we propose Through-Screen Computing in this dissertation. It is a new concept that refers to the processing of light signals for various computing purposes such as communication, sensing, and imaging, where the light comes from the physical world and passes through a special medium -- the transparent screen -- before reaching the under-screen optical sensors. This concept opens up new challenges and opportunities in connectivity, privacy, and security of future devices equipped with transparent screens. In this dissertation, we outline a vision for through-screen computing and address the challenges of transparent screens acting as both passive blockers and active interferers of input light signals.

This dissertation focuses on two subsystems in the context of through-screen computing: Through-Screen Visible Light Communication (VLC) and Screen Perturbation for Visual Privacy Protection. In the context of VLC, the full-screen trend challenges the deployment of this technology. We propose Through-Screen VLC with under-screen optical sensors as receivers. To address the attenuation of the light by the transparent screen, we develop SpiderWeb, a system exploiting the color domain to mitigate the color-pulling effect introduced by the transparent screen. We also leverage the Under-Screen Camera (USC) for VLC and design novel demodulation algorithms to reduce multi-pixel screen interference and improve performance. Experimental results show significant improvements in both data rate and transmission range, using  a prototype we build with two commercial smartphones. For privacy protection, we propose Screen Perturbations, modifying pixels displayed on the transparent screen to embed speckled color blocks and color shifts in the final image captured by the USC. Screen perturbations can be exploited to disrupt advanced deep neural networks used on image classification and face recognition tasks. We first design two image-specific methods to add screen perturbations to the images captured by USC. Next, we develop Unicorn, a universal screen perturbation method optimized for robustness in various conditions. All these designed perturbations work successfully against various deep neural network-based image classification services with high success rates.

Through these two subsystems, as well as the proposed theoretical and experimental approaches and results, we demonstrate the transformative potentials of through-screen computing, setting the stage for future research and development on various computing purposes in the era of transparent screen and full-screen devices.
@en

Smartphones are moving towards the fullscreen design for better user experience. This trend forces front cameras to be placed under screen, leading to Under-Screen Cameras (USC). Accordingly, a small area of the screen is made translucent to allow light to reach the USC. In this paper, we utilize the translucent screen's features to inconspicuously modify its pixels, imperceptible to human eyes but inducing perturbations on USC images. These screen perturbations affect deep learning models in image classification and face recognition. They can be employed to protect user privacy, or disrupt the front camera's functionality in the malicious case. We design two methods, one-pixel perturbation and multiple-pixel perturbation, that can add screen perturbations to images captured by USC and successfully fool various deep learning models. Our evaluations, with three commercial full-screen smartphones on testbed datasets and synthesized datasets, show that screen perturbations significantly decrease the average image classification accuracy, dropping from 85% to only 14% for one-pixel perturbation and 5.5% for multiple-pixel perturbation. For face recognition, the average accuracy drops from 91% to merely 1.8% and 0.25%, respectively.@en

While radio communication still dominates in 5G, light and radios are expected to complement each other in the coming 6G networks. Visible Light Communication (VLC) is therefore attracting a tremendous amount of attention from both academia and industry. Recent studies showed that the front camera of pervasive smartphones is an ideal candidate to serve as the VLC receiver. While promising, we observe a recent trend with smartphones that can greatly hinder the adoption of smartphones for VLC, i.e., smartphones are moving towards full-screen for the best user experience. This trend forces front cameras to be placed under the devices' screen - -leading to the so-called Under-Screen Camera (USC) - -but we observe a severe performance degradation in VLC with USC: the transmission range is reduced from a few meters to merely 0.04 m, and the throughput is decreased by more than 90%. To address this issue, we leverage the unique spatiotemporal characteristics of the rolling shutter effect on USC to design a pixel-sweeping algorithm to identify the sampling points with minimal interference from the translucent screen. We further propose a novel slope-boosting demodulation method to deal with color shift brought by the leakage interference. We build a proof-of-concept prototype using two commercial smart-phones. Experiment results show that our proposed design reduces the BER by two orders of magnitude on average and improves the data rate by 59×: from 914 b/s to 54.43 kb/s. The transmission range is extended by roughly 100×: from 0.04 m to 4.2 m.

@en

We are now witnessing a trend of realizing full-screen on electronic devices such as smartphones to maximize their screen-to-body ratio for a better user experience. Thus the bezel/narrow-bezel on today's devices to host various line-of-sight sensors would disappear. This trend not only is forcing sensors like the front cameras to be placed under the screen of devices, but also will challenge the deployment of the emerging Visible Light Communication (VLC) technology, a paradigm for the next-generation wireless communication. In this work, we propose the concept of through-screen VLC with photosensors placed under Organic Light-Emitting Diode (OLED) screen. Though being transparent, an OLED screen greatly attenuates the intensity of passing-through light, degrading the efficiency of intensity-based VLC systems. In this paper, we instead exploit the color domain to build SpiderWeb, a through-screen VLC system. For the first time, we observe that an OLED screen introduces a color-pulling effect at photosensors, affecting the decoding of color-based VLC signals. Motivated by this observation and by the structure of spider's web, we design the SWebCSK Color-Shift Keying modulation scheme and a slope-based demodulation method, which can eliminate the color-pulling effect. We prototype SpiderWeb with off-the-shelf hardware and evaluate its performance thoroughly under various scenarios. The results show that compared to existing solutions, our solutions can reduce the bit error rate by two orders of magnitude and can achieve a 3.4x data rate.

@en

Motivated by the trend of realizing full screens on devices such as smartphones, in this work we propose through-screen sensing with visible light for the application of fingertip air-writing. The system can recognize handwritten digits with under-screen photodiodes as the receiver. The key idea is to recognize the weak light reflected by the finger when the finger writes the digits on top of a screen. The proposed air-writing system has immunity to scene changes because it has a fixed screen light source. However, the screen is a double-edged sword as both a signal source and a noise source. We propose a data preprocessing method to reduce the interference of the screen as a noise source. We design an embedded deep learning model, a customized model ConvRNN, to model the spatial and temporal patterns in the dynamic and weak reflected signal for air-writing digits recognition. The evaluation results show that our through-screen fingertip air-writing system with visible light can achieve accuracy up to 91%. Results further show that the size of the customized ConvRNN model can be reduced by 94% with less than a 10% drop in performance.

@en