The rapidly evolving field of computer vision has witnessed a paradigm shift with the introduction of Transformerbased architectures, particularly Vision Transformers (ViTs). As these models expand in complexity, ensuring their efficient deployment on resource-limited devices becomes crucial. This paper proposes a solution for the model compression problem, emphasizing quantization, and highlights a notable gap in current methodologies: their need to consider outliers in the quantization process. We propose a distillation-guided quantization approach for ViTs, leveraging the Centered Kernel Alignment (CKA) similarity score. Empirical experiments are carried out on the DeiT architecture using the ImageNet dataset, with our CKA approach demonstrating promising results in retaining model intricacies during compression.