Why are there so many lives to live?

You do not reincarnate unless it is to learn a lesson—or it is to enter a personality fitted for, and in harmony with your soul. 

That is why so many great souls do not accomplish. 

They want to return to the earth before their time. 

Why is one allowed to go when it is not for his own good?

Because there are so many lives to live, and if the soul is so wrapped up in the physical, you will become earth-bound in any event. 

Many who have passed their previous tests are allowed to go back into an existence, which is not worthy of them just because they prefer not to wait until the chosen time comes for them to take a fitting role and become a helper and not a hindrance to the earth. 

Of course, the inferior personalities and lives upon the earth-plane are not lived by advanced souls.

But it takes a student of philosophy to know who is who. 

Often the souls who have taken lower physical lives without waiting for their real mission are put back so far when they return to the astral that in understanding, they are like mere infants, and it takes a long, long time for them to advance. 

That you have become advanced is no sign that you can throw aside rules and still realise that mind, which is of the highest.

Do these beings who return into the physical in such a state often redeem themselves before they return to the astral?

Indeed they do, and sometimes they will develop an understanding, physically and spiritually, which will help others—sometimes, in their hard lives and physical trials of the flesh, they will become greater—sometimes not.