Rodent models of both aging and obesity are characterized by inflammation in specific brain regions, notably the corpus callosum, fornix, and hypothalamus. Microglia, the resident macrophages of the central nervous system, are important for brain development, neural support, and homeostasis. However, the effects of diet and lifestyle on microglia during aging are only partly understood. Here, we report alterations in microglia phenotype and functions in different brain regions of mice on a high-fat diet (HFD) or low-fat diet (LFD) during aging and in response to voluntary running wheel exercise. We compared the expression levels of genes involved in immune response, phagocytosis, and metabolism in the hypothalamus of 6-month-old HFD and LFD mice. We also compared the immune response of microglia from HFD or LFD mice to peripheral inflammation induced by intraperitoneal injection of lipopolysaccharide (LPS). Finally, we investigated the effect of diet, physical exercise, and caloric restriction (40% reduction compared to ad libitum intake) on microglia in 24-month-old HFD and LFD mice. Changes in diet caused morphological changes in microglia, but did not change the microglia response to LPS-induced systemic inflammation. Expression of phagocytic markers (i.e., Mac-2/Lgals3, Dectin-1/Clec7a, and CD16/CD32) in the white matter microglia of 24-month-old brain was markedly decreased in calorically restricted LFD mice. In conclusion, LFD resulted in reduced activation of microglia, which might be an underlying mechanism for the protective role of caloric restriction during aging-associated decline.