I don't want to write a bunch of markup next to my javascript.

It's cool, you don't have to use JSX, you can just use the React.DOM functions. Or build a backend for whatever your preferred template system is, as long as the backend ends up creating nodes of React.DOM objects. Or you go with Om[0] and use EDN[1], Enlive-style[2] or hiccup-style[3] tempting.

Now if you want to physically separate the view logic and the corresponding markup generation, that's more debatable: they're extremely strongly coupled (and in fairly small chunks ideally) so you often can't trivially change one without the other, and thus keeping them together makes logical sense. See Pete Hunt's presentation which lumpypua linked, it tries to make that point fairly nicely.

[0] https://github.com/swannodette/om

[1] https://github.com/edn-format/edn

[2] https://github.com/ckirkendall/kioo

[3] https://github.com/r0man/sablono