Hand pose tracking in 3D configuration is one of the popular components of human-computer interaction research, which can be seen applying in virtual reality (VR) and augmented reality (AR). Although the research field has been thoroughly studied in the past few decades, most methods emphasized recognizing hand gestures or joint locations. However, computer interaction these days requires numerous manual object interactions. The paper first reviews state-of-the-art hand pose estimation and tracking methods to achieve accurate hand tracking and control. Then we describe the general procedure for recovery 3D hand model. In the second place, the results of tracking whole hand kinematics using the extended Kalman filter (EKF) based on data recorded from the surface markers are presented. Finally, in the part of teleoperation, the paper presents a novel method for applying synergy-based mapping strategy for SoftHand 2 control. And we demonstrate the simulation results of telemanipulation with hand racking through Simulink.