Now this is not lore just speculation however there is in game named bandits and some you can even help "Mist-watch" after helping them they will no longer see you as an enemy. I believe the bandits are ex-soldiers, mages, refugees, farmers ect...

Their classes are diverse and hold to their specialty be it heavy armor and weapons to archers and light armor. I changed the bandits in game using this mod  why?... simple it changes hold guards, Foresworn,  bandits merc. ect.. to a uncapped level with use of weapons, magic spells, and armor allowed by your level whatever level you are so are they. Many stated this would not be lore friendly really?... only you can get stronger? .. You loot a boss chest at a bandit camp and it has a Dadric weapon so they have Dadric weapons but cannot use them they just use and iron sword ... sounds ridicules to me.... if found in game it should be made to be used by all... that would be lore friendly and realistic.

What follows is pictures of various bandits at some point I may include stats, specs and yes they have perks. 

 more to come....