You can't do this from a conventional shortcode handler. The nextpage tag is processed very early in the loop, so at that time, all that exists is your shortcode tag. By the time shortcodes are processed, it is way past time for the nextpage tag to have any effect. Thus it just ends up as a HTML content.
For this to work, you'd have to hook something like the 'loop_start' action, search the content for your shortcode tag, and replace it with the nextpage tag.
I'm curious, how is entering a shortcode tag any better than entering the nextpage tag? I don't really see the point. (not that it matters, just curious)