In the Handicapping the Hugos thread, there's a discussion of what "mainstream" means.
In the simplest sense, "Mainstream" is everything that is not genre. It's a marketing category like "mystery" or "SF" or "chicklit" or "literary fiction". It's everything that's mimetic. That's a fairly useless category, though, because it's too huge. We joke about simplistic equations like "If you loved Dragonflight you'll adore Mission of Gravity" but categories exist to help people find books they'll like, and "If you loved Middlemarch you'll adore Rainbow Six" isn't going to do much for anyone. Anyway, marketing categories may be useful for finding books, but they're not interesting to think about as edges of genre.
Mainstream is a term from within SF culture. Mainstream writers don't know they're mainstream, and I believe Tor (which started off publishing mostly fantasy and SF) is the only publisher to label a portion of its list "mainstream." Mainstream is defined in opposition to SF. Damon Knight famously said that SF is what we point at when we say SF, and mainstream is the same, it's what we (SF readers) point at when we say mainstream.
What I find interesting is when there are books that are "obviously" SF but that some people think are mainstream.
I think what people mean when they say The Yiddish Policeman's Union (an alternate history about a Jewish state in Alaska) is "mainstream" is that it has mainstream sensibilities, mainstream expectation, and, most of all, mainstream pacing. They may also mean that it had mainstream publication and that Michael Chabon is a writer who made his name selling mimetic fiction -- which is still true even though his last three books have been genre and he's spoken well of SF and even joined SFWA. I just made this kind of argument myself in that thread when I said that Ian McDonald was a long-standing SF writer who went to cons. The status of the author shouldn't make any difference... except that it kind of does. If some people are detecting mainstream sensibilities in Brasyl (a novel about quantum alternities in a historic, present and future Brazil) then I suppose they are. I don't know how, and I'd be interested to know how, because I just don't see it.
Samuel R. Delany has talked about the importance of reading protocols, and reading SF as SF. I tend to read everything as SF.
When mainstream writers come to write SF, it's normally the case that they don't understand the idioms of SF, the things we do when we (SF readers) read SF. This is very noticeable in things like Marge Piercy's Body of Glass (published as He, She and It in the US) where Piercy had clearly read Gibson but nothing much else, or Doris Lessing's Shikasta and sequels. The mainstream writers know how to do all the basic writing stuff, stories and characters and all of that, sometimes they know how to do that really well. They really want to write SF -- in Lessing's case she clearly admires SF -- but they don't know how SF works. They explain too much of the wrong things and not enough of the right things, they come up with embarrassing acronyms (SOWF, the "spirit of we feeling" from Shikasta, is burned onto my brain) and they don't understand how to put things over. They don't get the thing I call "incluing", where you pick up things about how the world works from scattered clues within the text. I don't feel that Chabon has this problem in the slightest, because he is an SF reader and knows how to inclue -- indeed I very much admire the brilliance of his worldbuilding -- but he's very unusual.
I had a great revelation about this some time ago when I was reading A.S. Byatt's The Djinn in the Nightingale's Eye. This is a mainstream story in which a female academic buys a bottle containing a djinn and gets it to give her wishes. It's a mainstream story because she finds the bottle on something like page 150 of 175. In a genre story she'd have found the bottle on the first page. It has mainstream pacing and expectations of what's important. The story is really about how simple answers are not fulfulling. The djinn is a metaphor in exactly the way Kelly Link's zombies aren't a metaphor. People talk about SF as a literature of ideas, as if you can't find any ideas in Middlemarch or Rainbow Six! I don't think it's so much the literature of ideas as the literature of worldbuilding.
In a science fiction novel, the world is a character, and often the most important character.
In a mainstream novel, the world is implicitly our world, and the characters are the world.
In a mainstream novel trying to be SF, this gets peculiar and can make the reading experience uneven.
In the old Zork text adventures, if you tried to pick up something that was described but not an object, you'd get the message "that's just scenery". The difference between a mainstream novel and an SF one is that different things are just scenery.