There were more than 6,400 arrests made in relation to people working illegally, often in poor conditions, the Home Office says.