Today market success and competitive advantage depends on information, and the proper flow of information using information technologies is critical to maintain competitive advantage. To improve their business processes and networks, organizations need to leverage new information systems/information technologies (IS/IT). These new IT systems, which see a constant flux of innovations in software, computational and automation capabilities, include technologies such as robotic process automation (RPA), chatbots, analytics using machine learning and artificial intelligence to name a few. These technologies are currently gaining a lot of traction and popularity as solutions to bolstering an organization’s competitiveness. Organizations are struggling with managing these technologies to maintain competitive advantage. A challenge for organizations is to judge the effectiveness and the impact the new technology would have on their respective business processes before implementing the IT system. Organizations lack a framework that encompasses both the short-term and the long-term view of implementing these technologies as well as the challenges they would face because of these technologies. Through this research, we hope to allay this issue by asking and answering: How can an organization effectively evaluate the impact of new IT systems to improve its business processes? There exist various frameworks and measures in management literature that help organizations choose between multiple IT investments and there exists frameworks such as the balanced scorecard that help organization as performance measuring tools. The IT scorecard, a subset of the balance scorecard, was developed to measure performance of existing IT products and services in use. However the use of existing frameworks to evaluate new IT systems has not been empirically recorded. Existing frameworks also do not take into consideration the uniqueness of individual IT systems and the challenges that are associated with these systems. Other challenges include a greater focus on short- term monetary gains rather than long-term benefits, a lack of standardised parameters for accurate measurements of performance and the ethical challenges that come with using these new controversial technologies. The proposed evaluation framework aims to accommodate the weaknesses of the earlier frameworks. This is why an integrated approach has been taken in the framework. The framework consists of three levels with the evaluation of the IT system conducted at the final level. The top most strategic level offers the organization a perspective on the needs and goals of the business. This allows them to see if the IT system truly aligns with the organization’s strategic objectives. The next level is the business process level. The reason IT systems are introduced into organizational business processes is to improve efficiency of the business process by automating certain activities or reducing certain steps in the process, improve performance of employees or enhance customer experience thereby boosting overall productivity. However, if the IT systems are introduced into the existing business process without correcting for inefficiencies, these inefficiencies are carried into the new business process (the existing business process with the IT system introduced) which is detrimental to the organization in the long run. Therefore analysing the business process helps us identify these gaps existing in the process, solve for them and redesign the business process with the implementation of the IT system. This is achieved at the business process level of the framework. Finally the level of the IT system where its impact on the business process is evaluated. The IT system is evaluated from different perspectives which include the customer, the business value the system provides, the internal processes that are affected by the IT system and the future readiness of the organization to the IT system. This integrated approach, similar to the Balanced and IT scorecards, is augmented and made more dynamic and robust enough to evaluate new IT systems by using KPIs derived by the system developers and engineers as metrics of measurement. The framework also brings to light the ethical values and challenges that come with using new IT systems. To the initial four perspectives, the two new perspectives will provide a more complete view of the impact the IT system has on the business process. The better the evaluation, the better the decision making of the organization. To demonstrate the utility of the evaluation framework, it is applied to the incident management process which is a common business process of the service based enterprises. This business process is analysed and the common challenges found in this process are discussed. In order to improve the business process, two new IT systems are introduced separately; the robotic process automation (RPA) and the chatbot. The proposed evaluation framework is applied to both the situations and the results are compared to see which IT system better improves the business process. This work has shown that including the distinctive nature of the new IT systems such as RPA and chatbots improves the results of the impact evaluation. In case of RPA, aside from the obvious benefits of the technology organizations might not be aware of the improvement in compliance the system provides or the increased quality of data being processed due to fewer errors or the heightened security risks that increased automation can cause. Organizations must also be wary of any underlying biases and stereotyping when implementing chatbots into their business processes. The idiosyncratic properties of each technology can influence the evaluation results and hence the decision on whether the IT system will be implemented. Therefore, by including the indicators specific to these systems, organizations can be aware of the potential impact these systems have on a particular business process. In this way undesired effects such as the inherent challenges of the IT system as well as the consequences of ethical issues are captured. Besides the impact evaluation of the IT system itself, the analysis at the strategic level ensures that there is a strategic alignment with the objectives of the business and the IT system and the modelling of the business process ensures that a detailed analysis can be done to find the inefficiencies and challenges in the business process. Through the proposed framework, this study focuses largely on the IT system and the impact it has on a business process of an organization. Once these IT systems are more prolific in their applications and use, higher levels of evaluation such as at the strategic level will be possible. This will also be very useful for organizations interested in choosing the right IT system for their organizational structure, culture and business processes and therefore be an interesting field for future research. A limitation of this research is that while this paper explains the working of the evaluation framework, a detailed analysis of the application of the framework has not been made. This can be captured by future research through in depth empirical studies with organizations implementing these new IT systems and using this framework to conduct the impact evaluation on the business process. Comparative studies can also be conducted such as comparing this framework with other integrative evaluation approaches such as the multi-attribute utility theory, the voting analytic hierarchy process and information economics.