An end-to-end framework is developed to discover physical laws directly from videos, which can help facilitate the study on robust prediction, system stability analysis and gain the physical insight of a dynamic process. In this work, a video information extraction module is proposed to detect and collect the pixel position of moving objects, which would be further transformed into physical states we care about. A physical law discovery module is developed to learn closed-form expressions based on the extracted physical information. The video information extraction module takes advantage of contour detection and Hough transformation to extract position information. The physical law discovery module includes a deep neural network-like hierarchical structure Mathematical Operation Network (MathONet) which is consisted of basic mathematical operations. We develop a sparse Bayesian learning algorithm to learn both the topology and parameters of dynamic systems. Several simulated videos were generated to illustrate Newton’s law of motion, the state space of a Duffing oscillator, and the pendulum motion equation. By demonstrating on these examples, our method can discover the corresponding governing function without requiring much prior information.