You're right that the interpretation of this rule is something that varies too much, far more than it should from official to official, crew to crew, and chapter to chapter. This is purely a safety rule -- a horse-collar tackle is an action very likely to cause serious injury to the ball carrier and thus has been removed from the game (former Oklahoma Sooner and Dallas Cowboy safety Roy Williams is the poster child for the creation of this rule).
As far as the "immediately pulling the ball carrier down," the interpretation by philosophy is that the action of pulling the ball carrier down has to be against the momentum of the runner (which is where the injuries come from). If a defender were to grab the runner inside the collar and run with him for a few yards prior to an immediate and powerful jerking takedown against the momentum of the runner, the danger of injury would be no less and this, by philosophy, is a horse-collar tackle and a foul.
The "immediately" is interpreted to apply to the "pulling the ball carrier down" action, not the time it occurs in relation the grab of the inside of the collar. This is compared to a slow "riding the back" of the ball carrier kind of tackle.
That said, I know this isn't applied universally by the officials y'all are seeing in your games, but this is the philosophy and interpretation and is being taught and hopefully applied for the safety of the players.