Ideally, they should live a bit apart, like across the room if that is possible; at worst they can live above the girls (to prevent the pheromones getting to them).
The big no no is living within sight of sows! That can lead to problems, especially during the hormonal months.