If the island were 100 times larger, the houses would take 1% of the land area, leaving 99%. The apartment complex would take up .04%, leaving 99.96%, which isn’t much of an improvement. The proportions of our planet are much closer to my scenario than this made up island. That’s a reason why we might not “prefer apartments in our own town.”
There are good reasons you might want density, this just isn’t one of them.
Easier travel is definitely a great reason to increase density. Walking & biking > cars