In this paper we propose a novel interaction model for augmented environments based on the concept of mixed initiative interaction, and describe the design of the Pendle, a gesture-based wearable device. By splitting control between user and environment, the interaction model combines the advantages of explicit, direct manipulation with the power of sensor-based proactive environments while avoiding the lack of user control and personalization usually associated with the later. The Pendle is a personalizable wearable device with the capability to recognize hand gestures. It acts as mediator between user and environment, and provides a simple, natural interface that lends itself to casual interaction. Experiences with two concrete examples, the MusicPendle and NewsPendle, demonstrate the advantages of the personalized user experience and the flexibility of the device architecture.