It is common knowledge that there are five of the four colleges at Hogwarts.
So it’s not that weird to be in Azkaban at the start, right?