Myocardial perfusion imaging (MPI) is a clinical tool which can assess the heart’s perfusion status, thereby revealing impairments in patients’ cardiac function. Within the MPI modality, the acquired three-dimensional signals are typically represented as a sequence of two-dimensional grayscale tomographic images. Here, we proposed an end-to-end survival training approach for processing gray-scale MPI tomograms to generate a risk score which reflects subsequent time to cardiovascular incidents, including cardiovascular death, non-fatal myocardial infarction, and non-fatal ischemic stroke (collectively known as Major Adverse Cardiovascular Events; MACE) as well as Congestive Heart Failure (CHF). We recruited a total of 1928 patients who had undergone MPI followed by coronary interventions. Among them, 80% (n = 1540) were randomly reserved for the training and 5- fold cross-validation stage, while 20% (n = 388) were set aside for the testing stage. The end-to-end survival training can converge well in generating effective AI models via the fivefold cross-validation approach with 1540 patients. When a candidate model is evaluated using independent images, the model can stratify patients into below-median-risk (n = 194) and above-median-risk (n = 194) groups, the corresponding survival curves of the two groups have significant difference (P < 0.0001). We further stratify the above-median-risk group to the quartile 3 and 4 group (n = 97 each), and the three patient strata, referred to as the high, intermediate and low risk groups respectively, manifest statistically significant difference. Notably, the 5-year cardiovascular incident rate is less than 5% in the low-risk group (accounting for 50% of all patients), while the rate is nearly 40% in the high-risk group (accounting for 25% of all patients). Evaluation of patient subgroups revealed stronger effect size in patients with three blocked arteries (Hazard ratio [HR]: 18.377, 95% CI 3.719–90.801, p < 0.001), followed by those with two blocked vessels at HR 7.484 (95% CI 1.858–30.150; p = 0.005). Regarding stent placement, patients with a single stent displayed a HR of 4.410 (95% CI 1.399–13.904; p = 0.011). Patients with two stents show a HR of 10.699 (95% CI 2.262–50.601; p = 0.003), escalating notably to a HR of 57.446 (95% CI 1.922–1717.207; p = 0.019) for patients with three or more stents, indicating a substantial relationship between the disease severity and the predictive capability of the AI for subsequent cardiovascular inciidents. The success of the MPI AI model in stratifying patients into subgroups with distinct time-to-cardiovascular incidents demonstrated the feasibility of proposed end-to-end survival training approach.