The slider alters hp it seems at least, and you won't have as many knockbacks, critical strikes etc against you at lower difficulty.
I've tried the extremes and the median and this is what i discovered trying to kill a "boss" in an oblivion plane:
(was lvl 20 at the moment and a warrior type)
Lowest difficulty: 1 shot, 1 kill with the firespell you get from the start

Medium difficulty: Normal battle lasting a few minutes back and forward
Highest difficulty: 1 shot, 1 kill... just the other way around

However it should be noted that if you start playing the game at the lowest lvl your skills won't increase as quickly as on the higher difficulty due to the fact that you won't have to hit enemies that much to kill them

so the fastest way to progress in levels would be to play at the maximum difficulty, providing you can acctually survive that is