Vampire Muzzles were used back in 1864 by the people of the town of Mystic Falls when trapping vampires in the tomb. Katherine was the first vampire it was shown to be used on, soon followed by Pearl. Years later, Elena Gilbert was looking for Johnathan Gilbert's journal and she came across the muzzle, and Stefan had a flashback to when Katherine had been taken by the founding families, when she was wearing the muzzle.
As you can see with the images below, these muzzles are to tame vampires's to speak. But when a vampire is muzzled they have either been captured ready to be killed or captured ready to be tortured, but either way they will suffer from lack of blood and begin to dissipate into a coma.