The concept of Internet of Vehicles (IoV) played an important role for the improvements that happened in the smart transportation field. Nowadays, it is one of the hot topics in academia, automotive sector, industry, and research. The message transfer between the vehicles or infrastructures are necessary for communication in a vehicular network. A public channel is used for all these types of communication, which offers adversary to get a chance to delete, modify, insert, and extract data. Similar to security, lightweight property is yet another important factor needed to be considered thoroughly because of the highly dynamic nature of IoVs. In the state-of-the-art, several schemes are designed separately for security and lightweight property; however, we have only a small number of works that discussed the importance of maintaining a balance between these two criteria. Among them all schemes use a secure medium for the registration process. Different from them, we used a public channel, which makes our scheme more vulnerable to attacks than the state-of-the-art. In this paper, we designed a lightweight, secure authentication and key agreement scheme (LightKey), which is strong enough to resist different types of attacks and at the same time lightweight as well. In particular, LightKey is resistant to replay, man-in-the-middle, impersonation, physical capture, session key disclosure, perfect forward secrecy, backward secrecy, and ephemeral secret leakage attacks. We have simulated LightKey in Scyther (automated security protocol verification tool) and proved that it is secure against the above mentioned attacks. The performance analysis results show that LightKey takes 10% less computation cost than most related competitive scheme, which makes it to fit in the On-Board Unit (OBU) of any vehicles easily.