Cyber attackers leverage malware to infiltrate systems, steal sensitive data, and extort victims, posing a significant cybersecurity threat. Security experts address this challenge by employing machine learning and deep learning approaches to detect malware precisely, using static, dynamic, or hybrid methodologies. They visualize malware to identify patterns, behaviors, and common features across different malware families. Various methods and tools are used for malware visualization to represent different aspects of malware behavior, characteristics, and relationships. This article evaluates the effectiveness of visualization techniques in detecting and classifying malware. We methodically categorize studies based on their approach to information retrieval, visualization, feature extraction, classification, and evaluation, allowing for an in-depth review of cutting-edge methods. This analysis identifies key challenges in visualization-based techniques and sheds light on the field's progress and future possibilities. Our thorough analysis can provide valuable insights to researchers, helping them establish optimal practices for selecting suitable visualizations based on the specific characteristics of the analyzed malware.