Primary resistance to androgen receptor (AR) directed therapies in metastatic castration-resistant prostate cancer (mCRPC) is poorly understood. We randomized 202 treatment-naive mCRPC patients to abiraterone or enzalutamide, and performed whole exome and deep targeted 72-gene sequencing of plasma cell-free DNA prior to therapy. For these agents, which have never been directly compared, time to progression was similar. Defects in BRCA2 and ATM were strongly associated with poor clinical outcomes independently of clinical prognostic factors and circulating tumor DNA abundance. Somatic alterations in TP53, previously linked to reduced tumor dependency on AR signaling, were also independently associated with rapid resistance. Although detection of AR amplifications did not outperform standard prognostic biomarkers, AR gene structural rearrangements truncating the ligand binding domain were identified in several patients with primary resistance. These findings establish genomic drivers of resistance to first-line AR directed therapy in mCRPC and identify potential minimally-invasive biomarkers.