Endometriosis (EMS) is a gynecologic disorder associated with infertility and characterized by the endometrial-type mucosa outside the uterine cavity. Currently available treatment modalities are limited to undesirable effects. Thus, in the present study, we sought to study the pathogenesis mechanism of EMS. For this purpose, the ectopic and eutopic endometrial tissues were resected from 86 patients with EMS and 54 infertile patients without EMS, respectively. The regulatory mechanism among HES family bHLH transcription factor 5 (HES5), transforming growth factor-beta (TGF-beta)-induced factor 1 (TGIF1), F-box, and WD repeat domain containing 7 (FBXW7) was studied by performing co-immunoprecipitation, dual-luciferase reporter gene assay, and chromatin immunoprecipitation, respectively. A mouse model of EMS was established to verify the aforementioned regulatory mechanism in vivo. Upregulation of HES5 and TGIF1, as well as downregulation of FBXW7, was observed in EMS endometrial tissues and human endometrial stromal cells (hESCs), respectively. The overexpression of HES5 was found to suppress the FBXW7 transcription and TGIF1 degradation, resulting in the inactivation of the TGF-beta signaling pathway, as well as inhibition of hESC proliferation and invasion, thereby enhancing apoptosis. Results from a mouse model of EMS showed that the presence of HES5 contributed to the alleviation of EMS. Collectively, we attempted to provide a mechanistic insight into the unrecognized roles of the HES5/FBXW7 in EMS progression.