Refugees and asylum seekers are at a higher risk than the host population to poor health and financial stress. This study uses a unique longitudinal panel from Australia, the Building a New life in Australia (BNLA cohort) to understand the relationship over time between the social determinants of health, health, and financial hardship in refugees and asylum seekers. We employ a longitudinal; dynamic multivariate logistic regression to firstly estimate the relationship between the social determinants of health and poor physical and mental health. Next, we include variables related to financial hardship in our model to determine if there is an association independent of the social determinants of health. Finally, we estimate if there is a relationship between the number of financial hardships and poor physical and mental health. The results show that migrants from North Africa, the Middle East, and Sub-Sahara Africa and women are more likely to suffer from poor health. Financial hardship has an independent association with poor health. We find that going without meals had the highest odds of suffering from poor health. There was evidence of a dose response of financial hardship for those suffering from a limiting long-term health condition and post-traumatic stress syndrome. These findings suggest that refugees in Australia may need additional support past their first year to help them assimilate and contribute to economic productivity.