COVID-19 significantly influenced travel behaviours and public attitudes towards public transport. Various studies have illustrated complicated factors related to long-term travel behaviour, indicating difficulty in understanding and predicting post-pandemic long-term travel behaviour via traditional methods. In these complex circumstances, it is valuable to take advantage of social media data to obtain real-time public opinions to understand dynamic travel behaviour changes from the passenger perspective. The present study provides a means - leveraging Twitter data and state-of-art Natural Language Processing (NLP) technologies - to interpret the underlying associations among public attitude, COVID-19 trends and public travel behaviour. Concretely, New York City has been selected due to its dependence on public transit for daily commuting. More than 500K tweets have been collected from January 2019 to June 2022. Automated text mining, topic modelling, and sentiment analysis have been implemented in these contexts to identify dynamic public reactions. A consistently negative attitude to public transit is detected and five main topics, including derivative topics from COVID-19, are discovered within the COVID-19 duration. Policy makers and transit managers can use these topics to take onboard the public's concerns. The paper thus exemplifies how social media data and NLP technologies can support policy-making progress and can benefit other tasks in the transportation domain.