Unfit in middle age: Are you doomed?

How bad is a lack of fitness in middle age and is it condemning you to bad health in the future?