Introduction Exercise is considered a cornerstone in achieving an optimized blood glucose level and reducing body weight, body mass index (BMI), and waist circumference. This study aimed to investigate and quantitatively summarize the literature regarding the therapeutic effects of exercise and general physical activity on glycemic control. Methods A systematic review and meta-analysis of the literature on the therapeutic effects of exercise on glycemic control in patients with type 2 diabetes mellitus (type 2 DM) were conducted using electronic databases. Studies with an exercise intervention lasting more than 8 weeks were included. Both qualitative and quantitative analyses were performed. Qualitative data were presented narratively in table form. Quantitative analysis was performed using a random-effects model with a 95% confidence interval and a significance level of 0.05. The Physiotherapy Evidence Database (PEDro) scale and the Cochrane Risk of Bias 2 (RoB2) tool were used to assess the quality of evidence and the risk of bias. Results A total of 21,559 articles were identified through different databases. Out of 21559 studies, only 32 randomized controlled trials were deemed eligible for inclusion in this study. The average exercise session was 45.15 min, while the average follow-up duration was 21.94 weeks. The mean exercise frequency according to our findings was 3.25 days/week. Almost all the studies reported decreases in glycated hemoglobin (HbA1c; P < 0.0001), fasting blood glucose (P = 0.03), BMI (P = 0.04), and waist circumference (P = 0.007) after the exercise intervention. Conclusion Exercise plays an important role in optimizing glycemic control and improving quality of life (QoL), BMI, and waist circumference in type 2 DM patients. Exercise could be a safe adjunct therapy to medical treatments in these patients. Registration PROSPERO: CRD42020210816