I don't claim to be an expert on the subject, but here are my thoughts:
A lack of genetic diversity in the food crops could prove dangerous; simple organisms adapt and evolve much more quickly than complex ones do. I seem to remember the Great Potato Famine having genetic homogeneity as a strong factor in the scope of the tragedy. Even if there are no other motivations in gentically modifying foods other than to simply provide hardier, disease-resistant food crops, won't a pest or disease eventually evolve that will prefer the genetic profile of the engineered species? I'm reminded of antibiotic-resistant staph infections, as well as Deinococcus Radiodurans, and the fungi found thriving inside the shroud at Chernobyl.
All that being said, aside from some sudden dramatic drop in birthrate, average lifespan, and present world population, just how are we supposed to feed everyone? Were the world's agriculture to become completely organic in nature starting tomorrow, only about 2/3 of the world's population could be supported- and that's factoring in conversion of grain for animal feed purposes to human food, conversion of grazing land to cropland, etc.
I guess my final take on it is "Damned if we do, damned if we don't"...
Give sorrow words: the grief that does not speak
Whispers the o'er-fraught heart and bids it break.
-William Shakespeare, Macbeth, Act IV, Sc. III