It's fine if you don't wind people up by cheering the goals etc. Helps you you just pretend you're a neutral on a day out and don't know anything about either team.
I've done it at Villa about 4 times, Sheff Wed, Blackburn, City etc. Lots really. I don't have the accent so it's easier for me, and I just say I'm a Macc Town fan if anyone asks. It usually ends there cos nobody knows anything about them!
The best one was at Maine Road once when I was a kid, we'd won 2-0, kept quiet all game. Walking out through Moss Side at the front of the stream of City fans, heads down keeping quiet. Then two scrotty lads came up on bikes and asked the score, my dad said 2-0 to Everton. They then unzipped their jackets and had United shirts on underneath and started doing circles round us singing some Manc nonsense.
My dad waited a couple of minutes til they'd really got going, and then just said, "But we're Everton fans lads". About 50 City fans around us wet themselves and we chatted all the way back to the train station.