In the 1970s the usual suspects were rough: West Ham, Chelsea, Leeds, Manchester, but, actually every ground could be dangerous.
As others have said, Boro and Cardiff in the cup, were "memorable". I don't remember the date (any ideas oldies?) but after a night match at Wigan, at their old ground, all hell broke loose... I think every MU and Citeh fan around there turned up that night, too.
Anyone recall the Chelsea "Kensington High Street" affair?
One thing that did shock me was when my Dad (RIP) took me to Birmingham. I was probably about 10-12. On the walk back to the coach they attacked us and my Dad got punched. He was not being agressive or anything, he was just not a Brummie I suppose.
Years later, not saying we were particular hard, my mates and I were not really afraid. If we had been, I suppose we would not have gone to certain places. We were street-wise and stuck together. But also young, so reckless. We did not look for trouble (well, one mate did) but it was often unavoidable in those days and sometimes we did get our "revenge in first".
Rail and motorway service stations were often "lively", too.
PS. The thing is, Goodison and Anfield were terrible for away fans, too, and so at aways the locals were often out to get "revenge" on us.