Look, he annoys the hell out of me, too.
But if we were to be completely honest, those of us who grew up watching Williams on his HBO specials after he made his hairy-armed mark on the "Happy Days" spin-off "Mork & Mindy" in the late 70s appreciated him in the same way music fans gobbled up punk and New Wave-influenced music.
What's the connection?
Williams was a comedy fan's comedian — the way that Elvis Costello or Bruce Springsteen were music fan's music. A frenetic blend of influences based in obscure pop culture references and a strangely creepy need for attention, Williams was briefly considered kind of punk. Costello combined Buddy Holly glasses, "Highway 61"-era Dylan hipness, and the angry nerd persona of Woody Allen. Hey, wait, Allen is much more known as a comedian than a musician. Whatever.
OK, Steve Martin was and is infinitely cooler. But I bet more than half the guys and gals you see chopping up weird-ass pop culture stuff in their sets at any alt comedy club or rock venue might admit that Williams' strange blend of weirdness, friendliness, proto-ADD angst, and hyper-parodic style had an influence. And if they won't, Patton Oswalt at least should.