Breast cancer (BC) ranks first in the incidence of tumors in women and remains the most prevalent malignancy in women worldwide. Cancer-associated fibroblasts (CAFs) in the tumor microenvironment (TME) profoundly influence the progression, recurrence, and therapeutic resistance in BC. Here, we intended to establish a risk signature based on screened CAF-associated genes in BC (BCCGs) for patient stratification. Initially, BCCGs were screened by a combination of several CAF gene sets. The identified BCGGs were found to differ significantly in the overall survival (OS) of BC patients. Accordingly, we constructed a prognostic prediction signature of 5 BCCGs, which were independent prognostic factors associated with BC based on univariate and multivariate Cox regression. The risk model divided patients into low- and high-risk groups, accompanied by different OS, clinical features, and immune infiltration characteristics. Receiver operating characteristic (ROC) curves and a nomogram further validated the predictive performance of the prognostic model. Notably, 21 anticancer agents targeting these BCCGs possessed better sensitivity in BC patients. Meanwhile, the elevated expression of the majority of immune checkpoint genes suggested that the high-risk group may benefit more from immune checkpoint inhibitors (ICIs) therapy. Taken together, our well-established model is a robust instrument to precisely and comprehensively predict the prognosis, immune features, and drug sensitivity in BC patients, for combating BC.