This was because "Allow Move AI" was turned off in Realtime Combat settings.
I haven't had followers in my game for a while, so I'm not sure what might have gone wrong.
But if I activate my Real Time Battle Arena they stop following the leader. They follow if it's turned off though. They just stand perfectly still and seem to ignore their MoveAI They still attack if enemies get close enough though