^
You're right by stating that Papaver somniferum is an annual plant.. and mine, returning every year again, is obviously not. Well, it is a variant that you could use for making opium.. and I did collect the sap from it's seed pods and smoked it after drying. It's so fucking nice I consider the stuff dangerous myself. Too addictive.. but, must say, it makes one horny as hell. Other great downside of it.. is, that it makes my joins ache really terrible the next day.
Plus think your garden is probably quite clay-ish (polder and all) and poppy-plants tend to like sandy grounds mostly (region BoZ has sand grounds). Um, tiz also a drainage thing. Poppy seed and its young plants are quite vulnerable for moist/wet soil and tend to rot fast. Well, maybe that is why it never did well in your garden. (not sure though)
If you wish to try again.. maybe use a pot or something with an hole in its bottom (no pun).. with a 1:1 mixture of potting soil and regular sand for okay drainage.
(blahiah.. surry)