Designers often think of theme as this one thing that gets layered on top of mechanics. I would argue that theme has number of ways it is expressed and that different types of games employ different expressions of theming.
I divide theme into three main segments: thematic illustration, thematic setting, and thematic mechanics. Thematic illustration involves using representational art to make abstract games more visually and emotionally appealing. Thematic setting refers to the paragraph in most rulebooks explaining the connection between the illustration and the game. Thematic mechanics are game mechanisms that reinforce the thematic setting and help players make sense of the rules by providing a justification for play.
How much theme is required for a game to be thematic is an area rife with debate. No mass consensus has been reached on the exact definition of an abstract game. However, many people have described loosely themed games as “essentially abstracts.” These games usually have thematic art and a paragraph in the rulebook explaining the setting/story of the game. Thus, in order to feel thematic, a theme must be more than a justification for why a game has certain art assets. Thematic games require the theme to be reinforced by and have bearing on the mechanics. Games feel themeless when they have abstract mechanics, even when they have thematic illustration and a thematic setting. Seikatsu is an example of a game that has thematic illustration and a justification for gameplay (to be the player with the most beautiful garden view), but the mechanics of placing birds and flowers for best point combos has no bearing on the theme, and the bird scoring method even undermines the stated thematic goal (i.e. point-of-view no has bearing on bird scoring). The thematic goal (have the best view) somewhat aids players in remembering how end game scoring works, but does not provide a justification for the draw-one-place-one mechanic.
Theme only exists experientially when setting is used to make sense of mechanics, no matter how loose the connection. Great Western Trail feels thematic (as far as heavy euros go) because the movement of pieces across the board echos the movement of historical cowboys and cattle across the US, amongst other thematic ties.
Oftentimes, a game will have thematic setting and illustration that are seemingly well-executed, but combine with the mechanics in a jarring way. Many games that purport to be about exploration, adventure, or terraforming end up actually being about managing corporations, stocks, and profits. Heavy euros fall into this category so often that I won't even bother listing specific examples. I don't mind a good economic game; it's the bait and switch that bugs me. This is referred to as ludonarrative dissonance.
Similarly, other games mismanage expectations by overselling the theme. Nomads is a creative little set collection game that tried to oversell its storytelling theme (and its world-building) to a degree that some were off-put by the abstraction of the mechanics. I quite enjoy it as a game, but the point stands about not overselling an experience that your game does not provide. Again, this occurs because the setting and illustration (title and cover art are big offenders) don't fully match the experience of playing the game.
Thematic mechanics cannot oversell or jar players with cognitive dissonance; by their nature they are both thematic and are how the game is executed. And as we have seen, thematic mechanics are where most players experience the feeling of theme. However, in order to build an even greater feeling of what I call "thematic transportation" some games will add evocative theming, scripted narrative, and simulative actions.
Evocative theming encourages engagement by providing relatable or exciting themes that stimulate the imagination. Thematic settings that tell a brief compelling story are more evocative than those that simply invoke a certain genre of theme. For example, "Viking women on mission for revenge" is more evocative than "Vikings raiding a coastal region."
Scripted narratives leverage the elements of narrative transportation, providing world details that encourage players to ‘enter’ the world of the game. Scripted narrative may appear as thematic setting, chapter breaks (similar to cut scenes in video games), or flavor text sprinkled throughout the game. Games with scripted narrative tend to feel the most cinematic, but are difficult to design because of the amount of detail included to avoid feeling abstract while still feeling like a game.
Simulative actions are thematic mechanics that are so closely tied that performing the action creates the sense that you are actually carrying out the simulated activity. A great example is the hybrid game Keep Talking and Nobody Explodes. However, even more traditional games contain simulative actions. Sheriff of Nottingham has characters lie about the contents of their carts by having players lie about the contents of their bags. The primary difference for the players is that the lying is socially sanctioned within the bounds of the game.
These last three are not found in every game with a theme, so I think of them as auxiliary to the first three aspects. None of the above aspects is an attempt to describe successful execution of theme, merely the way theme is presented in board games.
I hope this exploration of the aspects of theme encourages you to look at theme as more than just a veneer of story stuck on top of some mechanics. Remember: theme is fun! Well-executed theme is an experience. Experiences create memories.