Upper Bounds and Cost Evaluation in Dynamic Two-player Zero-Sum Games