Every game starts with an idea. Often, this idea represents the core mechanic in your game. Games often have a core game loop that represents the foundation of all the rest of the gameplay. Core game loops can be simple (e.g. match-3 in Candy Crush) or complex (e.g. drop > collect loot > kill others > survive in battle royale games such as PUBG or Fortnite). There is but one fundamental truth that is important though: if your core game loop isn't fun, your game won't be fun.
An example: Roche Fusion
Let's rewind a bit. Roche Fusion is an almost-casual space shoot'em mixed with roguelite elements such as full procedural generation and a unique upgrade path in each game. Below you can see screenshots both of the released version, and also a screenshot of one of the first prototypes.
The screenshots show that the final version is much more polished and visually appealing, but there is even more that the screenshots don't show: a full soundtrack, dozens of enemy types and behaviours, many different upgrades, eight playable ships with completely different playstyles, and more. Here comes the kicker though: I could fire up that early prototype, and still have fun. The graphics were simple, the enemies programmed to follow an exact predefined pattern of behaviours, and it was enough to prove to us that people would have fun playing this game.
Of course all the layers of juice, content, and other fanciness made Roche Fusion (or beardgame, as we called it back them) from something that was merely fun into a production-ready game. Yet, I cannot imagine that all the layers that we added would have made a successful game if our earliest prototype hadn't been so much fun to play already.
Game jams
It may be hard to believe this as a truth, but I think there is another phenomenon that makes this truth even more convincing: game jams. Game jams are contests where developers are giving a limited time (often 48 or 72 hours) to make a game from scratch. It comes as no surprise that in that time you don't have the time to add a whole lot of content or to build a graphics engine that will define games for the next decade. Game jams often have simple graphics, simple sounds (if you are lucky), and can usually not be considered as much more than a proof of concept. In other words, game jam games are often just the core game loop with a thin layer of fanciness to make the whole package presentable.
There is only one game jam game that I am really satisfied with. The game started out with a silly idea, and after a day of fun, it just didn't work when playing. I made the hard call to start over the second day, and ended up with something that was extremely simple but a lot of fun. Once the core game mechanic had been revised, everything just clicked in place.
The lesson
I think of all the lessons I have learned during my career as game developer, this truth has been one of the most important. It is also the most difficult to put in practice. Game design is a creative process after all, and inspiration comes and goes.
To make things even harder, you can't know for sure whether a game mechanic is fun or not without implementing it and trying it out. You could compare it to other creative processes, such as painting: in your mind a colour combination might work really well, but as soon as your paint it, it may not look as well as you intended. If you practice more and more, you'll build up an intuition for what works and what doesn't. There will still be times where you're wrong though, game design will never become more certain than a series of educated guesses.
Luckily there is a fairly easy fix for this: iteration. Build prototypes to see what works and what doesn't. Not only does this help you to find ideas that work, it's also good practice: it will help you build an intuition, but it will also make you more able to distinguish what separates a good idea from a bad one. Game design is as much about dropping ideas as it is about generating them. Participating in game jams is a great tool to force yourself to work on an idea that you can drop after just a couple of days.
My current project, simply called TD, started from just such a (self-imposed) game jam. We spent a weekend building a prototype for a tower defence in a hexagonal grid with some novel ideas. Afterwards we played the game, and despite there not being any graphics or more than a single type of tower, it was fun. We decided to keep the idea and we're now two years in. If the game had been dull, we'd have dropped it and tried something else.
So to summarise: games stand or fall based on their core mechanics. If the foundation of your game isn't fun, then no matter how much prettiness you add on top, you'll never turn the game into something great. Determining whether your core mechanics are fun can only be done by building them. One day you'll stumble upon a great idea you can actually invest time in making great, knowing that your game has got that Fun Factor.