Q: What are the generic tactics of any flying character?
A: Any mode of movement other than moving normally over land offers the advantage of being able to go where one’s opponent(s) can’t. A creature with climbing movement, for example, can scale a vertical surface without being subject to any speed penalty or having to succeed on an ability check. In the case of flying, a creature has access to the air. It can hover out of reach; it can also launch itself airborne in order to flee.
Since the reach of most humanoids, armed or unarmed, is only 5 feet, a creature with 30 feet of flying movement can station itself 15 feet above its opponents’ heads, fly down, attack and fly back up using just its normal movement and action. A creature like the peryton, which has the complementary Dive Attack and Flyby features, will always use a tactic like this, because the combination does extra damage, and the peryton isn’t subject to an opportunity attack when it does so.
Opportunity attacks are the hitch with this tactic. Whenever a creature leaves its opponent’s reach, that opponent may use its reaction (if available) to make an opportunity attack against it. If the peryton didn’t have Flyby, for example, then every time it dove, its victim might get a free swing at it.
I’d conclude, then, that a creature that (a) can attack from the air and (b) can avoid being subjected to opportunity attacks when doing so will always attack this way. But if the creature is subject to opportunity attacks, it becomes more complicated. High-Dexterity shock attackers are more likely to dive attack than high-Constitution brutes, which will remain engaged after closing in. A creature whose opponent attacks with disadvantage is more likely to dive attack than a creature whose opponent attacks normally or with advantage. (If your opponent attacks with advantage, you absolutely don’t want to give that opponent additional opportunities to attack—you want to either negate the advantage or stay out of reach altogether.)
A creature that has greater flying speed than normal movement speed will move by flying leaps rather than run along the ground, unless its ability to fly is crippled. A high-Dexterity flying creature that has a ranged attack won’t even come near the ground to use that attack—it will simply attack from the air.
If a flying creature is large and strong enough to pick a smaller opponent up off the ground and either attack it (say, with its beak) while flying, dash it against a hard surface or drop it from a great height, and it doesn’t have some other, more effective form of attack, it might do that. A dragon probably wouldn’t bother, because it has so many other powerful choices. But a roc definitely would, because its talon attack includes grappling on a hit.
Q: When is it worth it or not worth it to provoke an opportunity attack?
The better its ability to avoid or absorb a hit, the less a creature has to concern itself with opportunity attacks. This might be because it has a high armor class or a high Constitution modifier beefing up its hit points, or it might be because the opponent has disadvantage on the opportunity attack roll. Also, if the opponent has already been goaded into using his or her reaction for something else, it’s spent for the round; the opponent no longer has the opportunity to make the attack. This is more likely to happen when the creature and its allies outnumber its opponents, rather than the other way around.
Another thing to consider is action economy. When a creature uses its reaction to make an opportunity attack, it gets only one swing/swipe/chomp. But if that creature has the Multiattack feature, it will get more than one attack when its turn comes around. It’s always worth it to risk one attack in order to avoid a second, third or fourth. The same applies if that creature has a bonus action that may do extra damage or impose a debilitating condition, such as knocking an opponent prone.
A more intelligent creature (let’s say, Intelligence 14 or higher) may be able to use its knowledge of how its opponent may attack, or how it’s already been seen to attack, to evaluate whether the risk of an incoming opportunity attack is greater or less than the danger of an attack that requires it to make a saving throw. Weapon and spell attacks put the burden of success on the attacker; saving throws put the burden of avoidance on the defender. A smart defender with a high armor class but either low ability scores, proficiency in no more than one of the big three saving throw abilities (Dexterity, Constitution and Wisdom) or both may decide it’s imperative to get out of sight, even at the cost of an opportunity attack.
On the flip side, a creature absolutely does not want to risk incurring opportunity attacks when it’s surrounded by multiple opponents, because each of those opponents may get an opportunity attack as it flees. This is what makes a spell such as dissonant whispers so potentially nasty: it forces an opponent to move, regardless of how many opponents then get to take potshots at it. Only a big boss monster, such as a dragon or a high-level demon, or an extremely stupid and foolish brute, such as an ogre, would be indifferent to its exposure to opportunity attacks from several opponents all standing around it.
Q: Under what circumstances might it be worth provoking an attack of opportunity for a flameskull to fly 40 feet above its enemies and use fireball?
A: None. A flameskull is immune to fire damage. It would cast fireball at ground zero if that meant it could get all its opponents in the blast.