Pediatric skeletal bone age assessment (BAA) is a common clinical practice to diagnose endocrine and metabolic disorders in child development. Recently, several automated methods have been developed to assist the diagnosis. For most children, the chronological age will be close to the bone age. Besides, features of the left hand between males and females are different by using either Greulich and Pyle (G&P) method or Tanner Whitehouse (TW) method. However, it is truly challenging to learn a unified representation based on the male and female image samples that have completely different characteristics. We argue that chronological age and gender are necessary pieces of information for automated BAA, and delve into the auxiliary of chronological age and gender for BAA. In this paper, a new multi-domain neural network (MD-BAA) is proposed to assess bone age of males and females in a separative and end-to-end manner. Furthermore, we introduce two regularization approaches to improve the network training: 1) an explicit dropout approach to select either the male domain or the female domain; 2) a chronological age preserving loss function to prevent the predicted bone age discrepant too much from the chronological age. Experimental results show the proposed method outperforms the state-of-the-art models on two datasets.