Why when I have presented my work at international conferences with experts in my field am I really stressing about giving a research seminar to my colleagues (most of whom have no knowledge of the subject)?

Heh, I know that feeling! Something about familiarity with people making it more difficult to risk looking foolish in front of them, maybe? I also get this nagging doubt that people outside my area will raise an issue that's so blindingly obvious that I won't be able to believe I didn't think of it, and all my research will fall apart as a result. Not very likely, but The Fear is still there! In reality, it will all go well and nobody will think of anything to ask that you haven't already thought about, and it'll all be a big success and a relief to get out of the way!

Good luck with the seminar - I'm sure it'll go way better than it feels at the moment...