From the very beginning, we're lead to believe that the Dursley family is the absolute worst. They forbid Harry from going to Hogwarts, locked him away for months at a time and probably also recline their seats on airplanes. They're real pieces of shit.
The Dursleys are so bad it's almost comical. Yeah, Dudley sort of makes peace with his cousin and Harry might've not grown up so humble if he didn't have such a godawful upbringing, but that doesn't excuse the Dursleys from being a barrel of fermented buttmunches.
Well, besides the fact that it's all Harry's fault.
As repugnant as they are, the Dursleys were nice enough to at least take in the orphaned Harry into their home. It's an act of charity that doesn't exactly gel with the jaded monsters that we see later on, so something must have happened in those ten years to make them hard, disgruntled and callous human beings. It's like they had a Horcrux living under the stairs of their home.
Remember, a Horcrux is any vessel that houses a piece of Voldemort's twisted soul -- and that includes Harry Potter himself. Horcruxes radiate poison; just being close to them will drive you mad.
For instance, near the end of the series, the gang is traveling the globe in an attempt to destroy all of the Horcruxes, and slowly each of them starts to unwind. Ron started feeling it after just a few weeks of Horcrux Hunting. And poor Ginny, she was stuck for a whole year with Tom Riddle's diary (also a Horcrux) back during The Chamber of Secrets, and that ordeal nearly killed her. Horcruxes are like if someone dipped the One Ring in a vat of ketchup water and then made plans with you and forgot to text you that they were bailing on you so you show up at the Chili's and eat Chili's alone and you are the saddest person at the Chili's full of people sad enough to eat at Chili's.
Horcruxes are a bad scene.
What does this have to do with the Dursleys? Well, if Ron was in dire straights after under a month with a Horcrux, imagine spending an entire decade living with something eating away at you year after year. To say you'd turn out to be a prolapsed asshole would be an understatement.