HEc11 to TEM mode? I know for high frequencies, it’s a loss issue. It order to stay TEM for coax, you have too keep reducing the diameter, then the loss increases substantially. Using coax past 26 GHz for more than a few meters length is problematic.
Waveguide has much lower loss per unit length than the equivalent TEM coax, but now dispersion becomes the issue.
So back in school, my Microwave professor said Bell had manufactured miles of circular waveguide for use in telecom, which was obsolete overnight by fiber. There was a certain circular mode that has very low dispersion, compared to rectangular.
This looks like a good discussion of the elliptical waveguide.
http://www2.rfsworld.com/RFS_Edition4/pdfs/FLEXWELL_Elliptical_Waveguide_347-369.pdf