Optical flow is a representation of projected real-world motion of the object between two consecutive images. The optical flow measures the pixel displacement on the image coordinate plane. However, it does not reveal the motion in depth explicitly, which could be useful as input in some tasks such as vehicle tracking. To extend the original optical flow approach, we model the depth change of the object as the scale change of object in the image and present an approach to estimate the scale change and optical flow jointly. Considering the scenario that obvious scale change occurs between two images, the traditional convolution network fails because it lacks the scale invariance. According to the Scale-space theory and the idea of learning a combination of Gaussian derivative basis to approximate arbitrary filters, we build a Basis Convolution layer that allows the network to see the scale change between two images and make use of it to better capture the same feature with various scales on two images.

We test our models on our own optical flow datasets which involve obvious scale change. According to the experimental results, our method is capable of estimating the scale change between images, and it significantly improves the optical flow estimation by modelling the scale change explicitly.