I don't think you need as many as you think to create a panic. I'm sure as little as five intelegentpeople working together would create lots of panic. Their main disadvantage seems that the smart ones stay at home in their countries leaving the work here up to the idiots like the ones that wanted to attack Fort Dix.
Then again I really don't feel this should affect my ability to my hobbies in the sciences at home because I really don't think they will be buying their supplies at companies like Untied Nuclear which is still open by the way.