The study probed the association between bilirubin and hepatitis B virus (HBV) infection and progression. A cross-sectional analysis of 28,500 middle aged and elderly Chinese participants was performed to analyze the differences of bilirubin in terms of hepatitis B surface antigen (HBsAg) positive or negative and the correlation between bilirubin and severity of hepatic fibrosis estimated by non-invasive indices. Bilirubin was significantly higher in the HBsAg (+) group than the HBsAg (-) group. Higher bilirubin levels were consistently associated with elevated liver fibrosis indices among HBsAg carriers. Compared with quartile 1 of total bilirubin (TBil), the multivariable-adjusted ORs (95% CIs) for elevated fibrosis indices of quartile 4 were 2.24 (95% CIs, 1.57-3.21) estimated by fibrosis 4 score (FIB-4) and 2.22 (95% CIs, 1.60-3.08) estimated by aspartate transaminase to platelet ratio index (APRI). In addition, direct bilirubin (DBil) had a stronger association with elevated liver fibrosis indices than did indirect bilirubin (IBil). Furthermore, the relationship between DBil and elevated fibrosis indices was more robust among participants who were female, overweight or had central fat distribution. These findings suggested that bilirubin levels, especially DBil, were independently associated with an increased risk of increased fibrosis indices.