The level of evidence of expert recommendations for starting extracorporeal cardiopulmonary resuscitation (ECPR) in refractory out-of-hospital cardiac arrest (OHCA) and in-hospital cardiac arrest (IHCA) is low. Therefore, we reported our experience in the field to identify factors associated with hospital mortality. We conducted a retrospective cohort study of all consecutive patients treated with ECPR for refractory cardiac arrest without return to spontaneous circulation, regardless of cause, at the Caen University Hospital. Factors associated with hospital mortality were analyzed. Eighty-six patients (i.e., 35 OHCA and 51 IHCA) were included. The overall hospital mortality rate was 81% (i.e., 91% and 75% in the OHCA and IHCA groups, respectively). Factors independently associated with mortality were: sex, age > 44 years, and time from collapse until extracorporeal life support (ECLS) initiation. Interestingly, no-shockable rhythm was not associated with mortality. The receiver operating characteristic-area under the curve values of pH value (0.75 [0.60–0.90]) and time from collapse until ECLS initiation over 61 minutes (0.87 [0.76–0.98]) or 74 minutes (0.90 [0.80–1.00]) for predicting hospital mortality showed good discrimination performance. No-shockable rhythm should not be considered a formal exclusion criterion for ECPR. Time from collapse until ECPR initiation is the cornerstone of success of an ECPR strategy in refractory cardiac arrest.