After doing my best to try and find the answer, I went with a Google sitemap plugin to correct the sitemap issues.
Installed: http://wordpress.org/extend/plugins/google-xml-sitemaps-v3-for-qtranslate/
I think Yoast SEO is more geared toward website content and not sitemap generation. Both are very difficult and huge projects to undertake. So I deactivated the XML Sitemap option in Yoast SEO and added the plugin above. It's working great now.