I wonder if our opinions of peoples' attractiveness are based in part simply on comparisons to people we've thought were attractive in the past and not simply an evaluation of their characteristics and how they look together... if, perhaps, we make a chain of associations between someone new we see and that other person from the past, and then the characteristics they share, concluding that they must be attractive too. Maybe that even explains why, as they say, people keep making the same mistakes over and over with who they date. (The stereotype woman who keeps dating the kind of men who will cheat on her, for example.) This kinda seems to fit the "learning by assocation" model.
At the most simplistic level, this seems to break down for me, though. Karen, a girl from elementary school who was the first girl I ever remembered thinking was attractive, was a lot of things that I don't find attractive right now: she had pretty standard European skin, but I like dark skin and very light skin; she had curly hair, but I don't like curly hair on young women*; she was short, but I like women of average height. Perhaps my tastes have just changed a lot since then.
(* It looks fine on older women, and certain types of curly hair look fine on guys. If you're a young woman, curly hair may count for "cute" points, but not for "attractive" points.)