Muhabharat - The Bharat family were caretakers of Histanipure for a king.?

However, after Histanipure got gambled away in a game of dice, it follows that the Bharat family were unfit to rule in this kings stead. Why didn't he come storming in and say stuff it, you had your chance? I can understand if the said royal was only a few years old, but the Bharat mob were exiled into the jungle for more than enough time to allow the true king to come back and claim the kingdom, especially after the Bharat family involved the kingdom in a power struggle for not a kingdom, but stewardship of the throne.