Canoloths are quadrupedal, weirdly doglike yugoloths whose function, like many other dogs, is to stand guard. They have expertise in Perception and Investigation, 120 feet of truesight, and immunity to surprise (unless they happen to be incapacitated), and their very presence suppresses teleportation out to a radius of 60 feet. Good luck sneaking up on these beasties.
They have exceptional Strength, Constitution and Wisdom, but their Wisdom influences only their senses, not their combat abilities; they’re not spellcasters. Really, therefore, they’re just brutes that happen to have exceptionally high Perception—and, perhaps, a particular knack for knowing how much threat an enemy or group of enemies poses. However, with their low Intelligence, they can’t do much with this information—it’s not going to have a meaningful effect on how they act.
Normally, the modus operandi of a brute is to charge and engage. But if it were so easy to entice a fairly stupid guard fiend away from whatever it was guarding, it wouldn’t be much of a guard. Thus, rather than leave its post to charge intruders, a canoloth lashes out at them with a spiky, prehensile tongue—with a 30-foot reach!—and yanks them into melee range.
The canoloth doesn’t distinguish between enemies. It simply attacks whichever one is the first to get too close to it. When a trespasser wanders into its 30-foot zone of control, it Multiattacks, choosing the Tongue/Claws combination. On a hit, the Tongue attack grapples and restrains its target and pulls the target up to the canoloth. Since the target is restrained, the canoloth then makes its Claws attack with advantage.
As long as it’s grappling a foe with its tongue (it can hold only one at a time this way), it continues to Multiattack that foe, switching to Bite/Claws once it’s holding them adjacent to it, and continues to have advantage on its attack rolls as it does so. What if allies of its target rush in and try to distract it? They can’t. Like a terrier worrying a rat, it keeps attacking whichever foe it has its tongue wrapped around and doesn’t let up for anything.
If the target manages to escape (the DC is 15, pretty doable for mid- to high-level PCs) and tries to get away, rather than make an immediate opportunity attack when they leave the 5-foot reach of its Bite and Claws,
the canoloth waits for them to leave its 30-foot Tongue reach and makes its opportunity attack with its Tongue; if it hits, it yanks the target all the way back and grapples them again, so their effort is wasted. Not as written, it doesn’t: As commenter David points out below, Tongue is a ranged weapon attack, so it has no reach, so it can’t be used to make opportunity attacks. In which case, the canoloth should use its Bite against a foe leaving its 5-foot reach—unless you want to decide that in your game, Tongue is a melee weapon attack after all!
But that’s if no one else has closed in to engage it in melee. Then it has to decide whether to seize its fleeing quarry or deal with the more immediate problem. The basis for this, as with the dhergoloth, is whether an opponent can hurt it badly enough with damage that gets past its immunities (acid, poison) and resistances (cold, fire, lightning, physical damage from nonmagical weapons). If a melee attacker successfully deals 12 damage or more to a canoloth, it switches its attention to them, using a Bite/Claws Multiattack; if more than one melee attacker deals 12 damage or more, it attacks whichever has dealt the most. If no melee attacker deals 12 damage or more but a ranged attacker does, and that ranged attacker is within 30 feet, it tries to snatch them with Tongue/Claws. If the ranged attacker is out of reach, or if no melee attacker manages to do enough damage, it picks a melee opponent at random. A canoloth taking damage from a ranged attacker more than 30 feet away which has no melee opponent within reach howls in frustration and telepathically tries to summon help—but it still doesn’t leave its post. No matter how much damage it takes, it never flees, not even in Gehenna.
On rare occasions, a canoloth may be ordered to seize and hold a trespasser rather than kill it outright. When so ordered, a canoloth still follows its normal combat heuristic—it doesn’t know any other, and this one works—but it makes its attacks nonlethal.