You, as a manager, are judged on results and not on the work you do and the performance of the team. Imagine paying a guy a huge amount of money and then judging him not on the things he can control, but on those he can’t.
Never let them judge you, they don’t know what you’ve been through.