The ridiculous flail snail probably ought to be categorized as a beast, but Volo’s Guide to Monsters declares it to be an elemental, meaning someone casting conjure elemental in the hope of summoning an earth elemental, xorn or gargoyle may end up stuck with one of these instead. Large, tough and most of all slow, the flail snail is technically a brute, but let’s not kid ourselves: This thing isn’t a predator, it’s prey.
Despite that, the flail snail has no effective means of running away when attacked, so it has to rely on a suite of defense mechanisms, the rudest of which is its Antimagic Shell, which has a chance of bouncing spell attacks back at their casters or refracting them into a multidirectional fusillade of force damage. Antimagic Shell, however, is passive. The one and only decision the flail snail needs to make when attacked is whether to use its Flail Tentacles, its Scintillating Shell or its Shell Defense on any given turn.
Note that flail snails, while brilliant in the visual sense, are far from it in the cognitive sense: with Intelligence 3, they’re not going to be employing advanced combat heuristics. A flail snail’s choice of action is going to boil down to a couple of simple rules that it always follows.
Scintillating Shell is a once-per-combat ability. A flail snail is going to hold off on using it until it can gain the greatest benefit from it—an evaluation that the flail snail isn’t intellectually equipped to make. We need to boil it down to something very basic. Normally, per the Targets in Area of Effect table in chapter 8 of the Dungeon Master’s Guide, an ability with a 30-foot radius ought to target roughly six opponents. But if the flail snail waits to use Scintillating Shell until six opponents are within 30 feet of it, it may end up waiting forever—what if it’s being stalked by just two or three hunters? Let’s simply say that it uses Scintillating Shell when its tentacles and Shell Defense are turning out not to be enough to make its assailants go away—but that, at the very least, it knows not to use Scintillating Shell when there’s no enemy within 30 feet.
Shell Defense, as an action, is one-way: it activates a benefit. To deactivate the benefit requires only a bonus action—and why would it ever deactivate the benefit? Following the principle of always interpreting every fifth-edition rule absolutely literally tells us something counterintuitive in this case: Shell Defense doesn’t prevent the flail snail from attacking. “The flail snail withdraws into its shell” seems like it should prevent it from attacking, but contrast the benefit of Shell Defense, a +4 AC bonus, with the Flail Tentacles trait: “If all its tentacles die, the snail retracts into its shell, gaining total cover” (emphasis mine). A +4 bonus to AC isn’t total cover; it’s not even three-quarters cover. Even after the flail snail retracts, something is still sticking out of that shell—and by deduction, it’s the tentacles.
Flail Tentacle, meanwhile, is a plain vanilla melee weapon attack, no special conditions, no riders.
when a flail snail is attacked (No! As commenter Scepstherep notes, the flail snail has tremorsense—it will go to red alert whenever it’s attacked or a Medium or larger creature comes within 60 ft of it, and it won’t come back out until that area is clear of threats), its first action is Shell Defense; it uses its movement to move directly away from its attackers, heedless of opportunity attacks (it’s not clever enough to know that such a thing exists). On its second turn, it Multiattacks, using its Flail Tentacles, against a randomly chosen enemy within 10 feet who’s dealt damage to it. On its third turn, if its foes are still hurting it, it uses Scintillating Shell. On its fourth turn, it goes back to Multiattack, but now it attacks only a stunned opponent with its tentacles. On its fifth turn, it Multiattacks, without any further thought given to target selection, and resumes moving away. It never drops its Shell Defense as long as there’s an opponent either within 60 feet of it or dealing damage to it from farther away.