Rotterdam the Hague Airport (RTHA) is planning to implement a new self-service luggage check-in system which has a different concept of operations compared to the current self-service system for luggage check-in. To efficiently develop, evaluate and implement this new system, a simulation optimization model is required to test different system configurations. The simulation optimization model is required to have a high level of detail to accurately represent the airport terminal process. An agent-based model will be used as the basis as that approach provides the necessary level of detail. This model is then combined with the metaheuristic tabu search algorithm to optimize the new self-service luggage check-in system. To address challenges related to the large required computational resources to perform simulation optimization with a high level of detail, the Simheuristic framework will be used. Combining an agent-based model with the Simheuristic framework is a novel approach to simulation optimization. This approach requires the development of a deterministic version of the airport terminal model. It was found that implementing an optimally configured self-service luggage check-in system can improve the performance of the check-in process while not compromising the performance of the security checkpoint. Additionally, the Simheuristic framework is capable of reducing the computational resources required.