Personally, what has always worked for me is writing the first draft, and maybe allowing a few people to read it (so far I’ve only done this for sequels and only for a handful of people who beta read the first novel) but more for encouragement than critique purposes. Then I write a second draft, fixing everything that I can see myself and then I open it up to about ten beta readers to give me their thoughts on it.
However, I do think that you’re thinking about this the wrong way.
On the one hand, I don’t want to put hours of work into this if it actually isn’t very good
It’s your first draft, it’s probably not going to be the best thing in the entire world right now, but that doesn’t mean it doesn’t have the potential to be. I also don’t think that there’s such thing as wasted time when it comes to writing and particularly editing. It may never get published but every time you complete a draft of it, you’ve improved your writing and editing skills which is going to make your next book better anyway. It’s also not wasted time for beta readers, since it allows them to get better at their own craft, and get better at learning how to critique other people. Does that make sense?
What I would do in this situation (and what I’m currently doing with a novel I’ve just finished) is to read over your draft, rewrite it, fix the major plot problems and then get a few betas (not too many) to look it over for you. That’s what has always worked for me in the past, but it may or may not work for you. Its up to you to figure that out 😀
Mahalo keia huiʻana