Much has been made of the rules for Thor's hammer, Mjolnir, and how it defines "worthiness." It's even a joke-y scene (and surprisingly major plot point) in Age of Ultron! But...why is Thor the only one "worthy"? Because he stood up to a robot that was gonna wreck a sleepy town in New Mexico? Captain America took on the LASER NAZIS and kamikaze'd a plane to save the world. Iron Man flew into a wormhole while holding a nuke, knowing he was low on power and likely would die. Bruce Banner confronted his greatest fears about himself to help the Avengers punch robo-space snakes.
The point is, pretty much ALL of the Avengers are either as worth or MORE worthy than Thor.