We aimed to discover barriers and facilitators of HIV pre-exposure prophylaxis (PrEP) adherence in young men and transgender women of color who have sex with men (YMSM/TW). Short-term and sustained adherence were measured by urine tenofovir concentration and pharmacy refills, respectively. Optimal adherence was defined as having both urine tenofovir concentration consistent with dose ingestion within 48 h and pharmacy refills consistent with ≥ 4 doses per week use. Participants completed semi-structured interviews exploring adherence barriers and facilitators. Participants (n = 31) were primarily African-American (68%), mean age 22 years (SD: 1.8), and 48% had optimal adherence. Adherence barriers included stigma, health systems inaccessibility, side effects, competing stressors, and low HIV risk perception. Facilitators included social support, health system accessibility, reminders/routines, high HIV risk perception, and personal agency. Our findings identify targets for intervention to improve PrEP adherence in these populations, including augmenting health activation and improving accuracy of HIV risk perception.