The deployment of voltage source converters (VSC) to facilitate flexible interconnections between the AC grid, renewable energy system (RES) and Multi-terminal DC (MTDC) grid is on the rise. However, significant challenges exist in exploiting coordinated operations for such AC/VSC-MTDC hybrid power systems. One of the most critical issues is how to achieve the optimal operation of such wide-area systems involving several power entities with as minimal communication burden as possible. To address this issue, an enhanced AC/DC optimal power flow (OPF) is specifically proposed. Firstly, a mixed-integer convex AC/DC OPF model is explicitly formulated to describe the optimal operation of such hybrid power systems. Subsequently, a nested distributed optimization method with double iteration loops is developed to offer optimal system-wide decision-making through a more “thorough” distributed communication architecture. In the outer iteration, the original AC/DC OPF problem is decomposed into several slave problems (SPs) associated with systems (including the AC grid and RESs) and one master problem (MP) associated with the integrated VSC-MTDC grid. Generalized Benders decomposition (GBD) serves to solve the master and slave problems iteratively. Techniques such as multi-cut generation and asynchronous updating are utilized to upgrade the GBD performance of computation efficiency and address communication delays. In the inner iteration, the master problem is continuously decomposed into multiple sub-MPs associated with individual VSCs. The alternating direction method of multipliers (ADMM) is employed to solve these sub-MPs iteratively. Proximal terms and heuristic approaches are embedded to enable parallel computation and handling of integer variables. Numerical experiment results finally validate the effectiveness of the proposed enhanced AC/DC OPF. The constructed AC/DC OPF model exhibits acceptable accuracy in terms of power flow calculation, and the developed nested distributed optimization method showcases decent convergence rate and solution optimality performances.