In Chapters and Indigo, Canada’s big-box chain bookstores, they separate out SF and fantasy into two separate sections. This always annoys me, and not just because it means that my own books aren’t together, but because they’re not very good at making distinctions. Right now, the new Bujold Sharing Knife book is in SF, and the first two in the series are in Fantasy. Right. Way to go.
There are a number of obvious edge cases we could use to taunt the bookstore clerk, who’s overworked, knows nothing about SF and probably likes to read Kant in her spare time. Marion Zimmer Bradley’s Darkover books, for instance, where people from Earth encounter people from Darkover who have magic, except it’s really genetically bred psi-talent, except it’s really magic, especially in the books set before the Terrans arrive, which read just like fantasy. Anne McCaffrey’s Pern, where the first part of the first book was published in that bastion of hard SF Analog when there wasn’t really a fantasy market, but which really are books about a feudal society of dragonriders, except for Dragonsdawn which is about the settlement of the planet and genetic engineering. It would be perfectly possible to make a good case for parts of each of those series to be shelved in SF and other parts in fantasy, which is clearly nonsense. Then there are Norton’s Witchworld books, and I'm sure you can think of other examples, because to a genre reader they are obvious examples.
In these books, SF is using some of the furniture of fantasy--magic and dragons and castles--or perhaps fantasy is using some of the furniture of SF, spaceships and laser guns, to play with culture clashes. (Exploring culture clashes across very different cultures seems to me one of the interesting things SF persistently does.)
But there’s another kind of book that can’t be neatly filed on one side or the other, where it isn’t the furniture but the fundamental axioms of the world that can’t be categorised. There’s Richard Garfinkle's Celestial Matters, a book with literal crystal spheres you can crash your actual spaceship through. There’s Michael Swanwick's The Iron Dragon’s Daughter (which has a new sequel The Dragons of Babel, which I haven’t got to yet) which is a kind of steampunk fantasy version of Tam Lin, with rusty industrial factories and teinds to Hell. There's Lucius Shepherd’s The Scalehunter’s Beautiful Daughter in which people are living in villages on the back of a dragon, but it’s all rigorously worked out and makes sense. In Lisa Goldstein’s Summer King, Winter Fool there’s a solstice ceremony to make the sun come back--fairly standard. Except they do it wrong, and the days keep getting shorter. There’s Ted Chiang's “Seventy-two Letters” where instead of Darwin some very Victorian scientists discover that the way the medieval world imagined genetics to work is how it works, and we’re running out of homunculi. There’s Harry Turtledove's “Secret Names” where a post-collapse of civilization witch doctor finds a book with the Latin species names of animals, which he uses to summon them to the nets, and it works.
Stories like this reach the fabled “sense of wonder” which science fiction wants to evoke, but from the fantasy side.
And you can't shelve them neatly anywhere.