Seems this is not an uncommon problem, but the solution was not easy to find - there are lots of proposed solutions on different forums and I've tried several until I finally found the one that worked. It turned out that the culprit in my setup was pulse audio. The solution that worked for me was simply to remove that:

killall pulseaudio
sudo apt-get remove pulseaudio
sudo apt-get install esound esound-clients libao2
sudo rm /etc/X11/Xsession.d/70pulseaudio

After the above I just changed back to ALSA in sound settings, restarted Firefox and everything worked like a charm. Note that executing the apt-get remove statement above will display a prompt to stating that it will remove Ubuntu desktop. This is just a meta-package so it will NOT remove the ubuntu desktop for real.