There are countless reports of the genetics of athletic excellence. Many athletes respond more effectively to training. For some, the shape of their bodies is optimized for certain kinds of athletic activities. Many of them have an unfair advantage as they carry genes that put them far ahead of ordinary athletes. Hence many sports inherently often favor people with certain genetic gifts and higher testosterone levels should not be treated differently.
Simply because other biological variations among athletes are not diminished is not a case for why testosterone, in particular, should not be.