The reason you don’t need to freak out is that this linking is disabled by default.

If you select the “Let others find my channel on YouTube if they have my email address” box, then they will link, otherwise, keep your stuff separate and don’t worry.