I can eat almost anything and still be skinny. If I eat very balanced and eat excessive amounts of natural fat, like almonds, cream etc and still stick to my strict diet of rice, fish and raw vegetables, I can gain a tiny amount of extra body fat.
However, if I have coke, fast carbs, pasta, cake or just "normal" food, what most western people eat, I get underweight. I can eat endless amounts of candy, snacks etc (but I don't because it still pollutes the body) without gaining much. I actually lose weight from a diet like that. Only way to get fat for me is to have a healthy diet as a foundation and add on top of that natural fats and extra servings.
So to some degree I believe it's genetical.