I should say, I've never really been a fan of the idea of 'bosses' as they tend to work out in either dungeon crawls or typical games, a big fellow lurking at the end of the dungeon who is arbitrarily a stronger, tougher foe. That seems uninteresting and often brings up questions for why they aren't guarding the front entrance personally. A lot of games have mechanics explicitly for this in enemies, and I try to avoid those systems.
What I do love is unique people that actually deserve either a position of having the power/resources to be a challenge. Singular monsters that come alone to mess things up, powerful pets, named, plotty important people. Its more appealing and at least a little more original. But I'm not a fan of running dungeon crawls, so that affects it too.
That's what I try to go for but these tricks will work with both schools, but the latter is my primary focus.
- Build them like a tank. A lot of games give multiple ways for people to defend themselves, these are easily broken down to they're either so fast they're hard to hit, or so tough a hit doesn't meaningfully hurt. These are both valid options for players, but a major enemy will often find itself outnumbered and surrounded. It needs to be able to endure multiple successful attacks, even if the attack needs to be very accurate, its a different game for a dodgey player avoiding one enemies attack and a dodgey boss avoiding four or five simultaneous attacks. Sheer luck will let enemies hit regularly, but without endurance, they'll be weakened far quicker. Conversely, if they're more resistant, they'll be hit more, but either lose so little health it won't matter for the time needed for them to provide the challenge, or they'll resist puny attacks, causing the players to need to expend resources to help, a bit of a challenge in itself.
- Give them a safety net. Some sort of way that lets them re-attempt a failed attack, avoid or reduce the impact of a particularly bad one, or heal up some damage. Its best if that's limited. The fight should end, but its nice to make a backup so that it doesn't just too quickly. Players, being the cruel and petty things they are, also enjoy removing the precious resources of their enemies.
- Mobility, this is more subjective to systems, but you want to ensure that every character can be reached, even if only difficultly. The ability to fly or shoot at flying ranged attackers. The ability to continue moving to chase down people that run. Players that learn they can exploit something because it can't get up to or catch up with them will take every opportunity to.
- When building or testing, if you can, aim for being just a tad over the line of too powerful. This is very hard to get right, and goes against initial common sense, but this ensures you'll have the closest you get to a desirable challenge. Not only that, players are like cornered animals. They'll scrape up every single little advantage they can get. This is fairly appropriate for important for challenging fights.
- Never be afraid to have the environment play a part in fights. Its often looked down on and considered a gimmick, but if its something thats telegraphed, everywhere or rare can be accepted, and if it's very manipulable environment during a fight, especially if with a little effort players can turn it against their enemies or use it for their own advantage they will often absolutely adore it. Players using something that isn't just their ability to devastating effect is probably one of the single most memorable things they'll come away from games with.