Youth social withdrawal has raised clinical concerns, and prevention of withdrawal behavior is important yet difficult. While human evaluation of withdrawal behavior can be subjective, technology provides objective measurement for withdrawal behavior. This study aims to examine the association between withdrawal behaviors (home-stay and non-communication) and mental health status (stress, depression and loneliness). The open-access StudentLife dataset, including the location and conversation information derived from the sensor data, stress levels, and pre- and post-questionnaires of depression (PHQ-9) and loneliness (RULS) of 47 college students over 10 weeks was used. Multilevel modeling and functional regression were employed for data analysis. Daily duration of home-stay was negatively associated with daily stress levels, and the interaction effect of daily duration of home-stay and non-communication were positively associated with daily stress levels and changes in PHQ-9 and RULS scores. Smartphone data is useful to provide adjunct information to the professional clinical judgement and early detection on withdrawal behavior.