Robot dexterous manipulation research has drawn more attention in recent years since the development of various learning methods makes it possible for robots to achieve dexterity at the human level. Many attempts have been made to integrate human knowledge into Reinforcement Learning (RL) processes for faster learning speed and better performance. Despite their successes in many aspects, there are two open problems that still need to be carefully considered: 1. The effect of demonstrations gradually vanishes during RL. 2. In most cases, only imperfect demonstrations are available to robots. In this work, we proposed a new learning framework - Interactive Behavioural Cloning for faster Reinforcement Learning (IBC-RL), which could alleviate problems in complex manipulation tasks with long horizons. Different demonstrations are shown to robots at different learning stages. Robots learn complex tasks step by step with interactive demonstrations from human teachers. The framework is evaluated with four dexterous manipulation tasks simulated with the Isaac Gym engine. Human teachers perform demonstrations by controlling the simulated robot hands through a hand-tracking system. The results of the experiments could demonstrate the efficiency of IBC-RL in guiding and accelerating the learning processes with imperfect demonstrations.