Are you using Pulseaudio output (or ALSA redirected through Pulseaudio)? My first guess is that you're hitting one of Pulseaudio's numerous bugs, where it keeps falsely reporting that there's still a significant amount of unplayed audio left; the player keeps waiting for the audio to finish playing.