Here's the thing... unfortunately this is one word against another, and due to that Kavanaugh shouldn't be denied his role on these accusations alone, because if we set a precedent where people can come out of the woodwork, say whatever - no matter if completely unsubstantiated - and stop something they don't like from happening.
I say all this despite being in a position where if I was put on the spot, I'd have to say I'm fairly sure Kavanaugh is guilty of the accusations coming his way... but there's no proof in terms of substantial evidence.
And to be perfectly honest, all more people coming forward does at this point is make it look like they have an agenda rather than a real complaint. That may or may not be true, but that's the reality of the situation - the complainants are too easy to dismiss as politically motivated.