Purpose: Tumor genomic features have been of particular interest because of their potential impact on the tumor immune micro-environment and response to immunotherapy. Due to the substantial heterogeneity, an integrative approach incorporating diverse molecular features is needed to characterize immunologic features underlying primary resistance to immunotherapy and for the establishment of novel predictive biomarkers. Experimental Design: We developed a pan-cancer deep machine learning model integrating tumor mutation burden, microsatellite instability, and somatic copy-number alterations to classify tumors of different types into different genomic clusters, and assessed the immune microenvironment in each genomic cluster and the association of each genomic cluster with response to immunotherapy. Results: Our model grouped 8,646 tumors of 29 cancer types from The Cancer Genome Atlas into four genomic clusters. Analysis of RNA-sequencing data revealed distinct immune microenvironment in tumors of each genomic class. Furthermore, applying this model to tumors from two melanoma immunotherapy clinical cohorts demonstrated that patients with melanoma of different genomic classes achieved different benefit from immunotherapy. Interestingly, tumors in cluster 4 demonstrated a cold immune microenvironment and lack of benefit from immunotherapy despite high microsatellite instability burden. Conclusions: Our study provides a proof for principle that deep learning modeling may have the potential to discover intrinsic statistical cross-modality correlations of multifactorial input data to dissect the molecular mechanisms underlying primary resistance to immunotherapy, which likely involves multiple factors from both the tumor and host at different molecular levels.