Caligula's Palace and the Temple of Vesta

From 37-41 AD, Caligula ruled Rome as Emperor. Although his track record has alot to be desired, his palace ruins are stunning. Temple of Vesta is where the flame of Rome always burned and was protected by the Vestal Virgins. I found it odd that the Vestal Virgins would be found neighboring, of all people, Caligula.