Parkinson's disease (PD) diagnosis is based on clinical criteria, i.e. bradykinesia, rest tremor, rigidity, etc. Assessment of the severity of PD symptoms, however, is subject to inter-rater variability. In this paper, we propose a deep learning based automatic PD diagnosis method using videos recorded during the assessment with the Movement Disorders Society - Unified PD rating scale (MDS-UPDRS) part III. Seven tasks from the MDS-UPDRS III are investigated, which show the symptoms of bradykinesia and postural tremors. We demonstrate the effectiveness of automatic classification of PD severity using 3D Convolutional Neural Network (CNN) and the PD severity classification can benefit from non-medical datasets for transfer learning. We further design a temporal self-attention (TSA) model to focus on the subtle temporal vision changes in our PD video dataset. The temporal relative self-attention-based 3D CNN classifier gives promising classification results on task-level videos. We also propose a task-assembling method to predict the patient-level severity through stacking classifiers. We show the effectiveness of TSA and task-assembling method on our PD video dataset empirically.