I voted It Varies. Usually I start on Adept, the exception being Illusionist/Conjurers like Achille who start on Expert. Achille wore robes and did no direct damage at all, so the difficulty only increased the damage done to him. It doesn’t make the fights any slower because the offensive damage by follower/summons is still being done on the default level, but one or two hits
should kill the character. Achille had to seriously run for his life while listening to hear when his summons got killed so that he could turn and re-summon. I noticed things with him like all of the places in dungeons where you can retreat and close a gate behind you, then summon on the other side of the gate. He wouldn’t have needed that sort of thing if he could get hit and then heal.
I also sometimes move to Expert with characters whose skills are tightly focused, allowing a high skill+perk level for them compared to the world. Sometimes I go too far with the damage/defense increase perks. Jensa got to the point where she was wading through everything with ease. In her case I just put her back in unimproved Hide with unimproved weapons and quit adding points to Health until she was really getting hit again (after some more levels). That’s also when I learned how much more fun it is to fight with a character who has a lot of Stamina.
I have a vague idea for an Altmer Conjurer/Destruction mage that I will probably start on Apprentice. Darnand has been doing well on Adept, but he still runs out of magicka before he can burn everyone up in some fights. He really needs a follower since he relies on Destruction for all of his damage. (Also I miss a lot. That’s totally MY fault.) So if I want a mage to come bursting out of Helgen in a blaze of wrath and no armor at Level 1 (per my idea), I think a lower difficulty is the way to go. UNLESS I can get him a staff early on. Hmm…
(Edited for clarity.

)