This paper presents the selection, development and testing of an Extended Kalman filter with an intermediate smoothing step (EKFS) and an unscented Kalman filter (UKF) as alternatives for a conventional Extended Kalman filter (EKF). The research aims to identify the viability of these filter modes for application in close-proximity satellite operation. A simulation tool for simple performance testing of the filters was developed and used before the filters were implemented and tested in the hardware-in-the-loop facility EPOS 2.0 at DLR Oberpfaffenhofen, Germany. To test the performance against identified key criteria for close-proximity satellite operation, primarily accuracy, run time and convergence time, numerous test cases were designed, including intentional outliers, delayed and withheld measurement input and differences in filter- and measurement update times. The simulation performance test showed up to 24 % accuracy improvement of the EKFS over the original EKF in the estimation of position of the target satellite. However, the performance in attitude and spin rate estimation decreased dramatically, and the UKF's performance was highly dependent on the selected test cases. These prior results could not be confirmed in the hardware-in-the-loop test since the new filters showed pose estimate divergence at close distances to the target (<5 meters). It is expected that further testing and improvement of the simulation performance test and the implementation of delayed measurement and stability functionalities could improve the performance of the new filter modes for the use case at hand. However, for the time being, the new filter modes need further improvement to become dependable alternatives for real-life, real-time close-proximity satellite operation for on orbit servicing missions.