Background Head and neck squamous cell carcinoma (HNSCC) is one of the few malignant tumors that respond well to immunotherapy. We aimed to investigate the immune-related genes and immune cell infiltration of HNSCC and construct a predictive model for its prognosis. Methods We calculated the stromal/immune scores of patients with HNSCC from The Cancer Genome Atlas using the Estimation of STromal and Immune cells in MAlignant Tumor tissues using Expression data algorithm and investigated the relationship between the scores and patients' prognosis. Three machine learning algorithms (LASSO, Random Forest, and Rbsurv) were performed to screen key immune-related genes and constructed a predictive model. The immune cell infiltrating was calculated by the Tumor Immune Estimation Resource algorithm. Results The stromal and immune scores significantly correlated with prognosis. A 6-gene signature was selected and displayed a robust predictive effect. The expressions of key genes were associated with immune infiltrating. GSE65858 validated the results. Conclusion Our study comprehensively analyzed the tumor microenvironment of HNSCC and constructed a robust predictive model, providing a basis for further investigation of therapy.