Vitamin D deficiency has been associated with depressive symptoms and reduced physical functioning. The aim of the study was to characterize the relationship between polymorphisms of the vitamin D receptor (VDR) gene and the quality of life in patients with autoimmune hepatitis (AIH) and primary biliary cholangitis (PBC). Three polymorphisms of the VDR gene (TaqI-rs731236, BsmI-rs1544410, and ApaI-rs7975232) were analyzed in patients with AIH (n = 142) and PBC (n = 230) and in healthy individuals (n = 376). Patient quality of life was assessed by validated questionnaires such as Medical Outcomes Study Short-Form 36 (SF-36), State Trait Anxiety Inventory (STAI), Modified Fatigue-Impact Scale (MFIS), Patient-Health Questionnaire 9 (PHQ-9), and PBC-40. The TaqI C and ApaI A alleles are risk alleles in both AIH and PBC, and a significant dominance of the A allele in BsmI was observed in AIH patients. In terms of quality of life, the presence of the CC or CT TaqI genotype was associated with emotional reactions, including the fatigue and the cognitive skills of patients with PBC, whereas in the group of AIH patients, homozygotes CC of TaqI, AA of BsmI, and AA of ApaI had worse physical, social, emotional, and mental function. The genetic variations of VDR gene can influence individual susceptibility to develop chronic autoimmune liver diseases such as AIH and PBC and affect quality of life.