It can even be something within the video itself, like a mic getting hit/plugged in, or sudden loud noises like someone screaming.

The worst offender is when an ad start playing (particularly youtube ads, but sometimes even someone cutting to a sponsor spot recorded in studio at different volume, with no editing to compensate), but aside from ads it can also be extremely annoying when using a music-playlist and one song is markedly louder or lower than the others (or ads while listening to one of those.). This obviously leads to differences such as the volume, and I have long been somewhat annoyed by this (even youtube themselves has noticed this being an issue, as I have read that they do automatically modify the extreme cases of this). Youtube videos are made by many different people, using many different softwares and tools.