The design of effective cooling strategies is a crucial component in simulated annealing algorithms based on the Metropolis method. Traditionally, this is achieved through inverse logarithmic decays of the temperature to ensure convergence to global minima. In this work, we propose Collective Annealing by Switching Temperatures (CAST), a novel collective simulated annealing dynamic in which agents interact to learn an adaptive cooling schedule. Inspired by the particle-swapping mechanism of parallel tempering, we introduce a Boltzmann-type framework in which particles exchange temperatures through stochastic binary interactions. This process leads to a gradual decrease of the average temperature in the system. Numerical results demonstrate that the proposed approach consistently outperforms classical simulated annealing with both logarithmic and geometric cooling schedules, particularly in terms of convergence speed.