Rutting is one of the most common distresses in early damage to asphalt pavements. It can raise the risk of ride safety issues, accelerate pavement deterioration, and increase maintenance costs. To investigate the factors that affect the rutting resistance of asphalt mixtures, internal factors (such as aggregate gradation, asphalt content, and layer thickness), external factors (including temperature and traffic loading), and human factors (such as compaction degree) were tested using wheel tracking tests. The test results showed that the rutting resistance of asphalt mixtures can be improved by designing a tightly interlocked aggregate skeleton using the Bailey method's primary control sieve, using an optimal asphalt content, achieving sufficient compaction, maintaining a layer thickness of 2.5–3 times its nominal maximum aggregate size, using an asphalt softening point higher than the pavement temperature, and avoiding overloaded vehicles. In highly rutted areas, it is recommended to use a stone mastic asphalt with a stable aggregate skeleton matrix and styrene-butadiene-styrene modified asphalt with a softening point higher than the highest pavement temperature.