No never...
Although when I was young, I did go to The Valium from time to time. All my dads side were 'Wall, but my mum's side were mostly Spotters. If Wall were away, my dad would go to an occasional Charlton game with my uncle and I would sometimes be taken along.
That was back in the 60's and as far as I can remember, rivalry was light hearted back then because of the close proximity of the clubs and the same socialising areas. Also, a lot of people knew each through working locally together. I went a couple of times in the early 70's as well (with Spotter friends).
Funny thing though, two of my cousins are Spotters and they still have a soft spot for Millwall after all these years. Their dad would go to 'Wall with my dad and they would tag along.