I have wild violets throughout my garden beds. In Spring, the new ones produce blue or purple flowers. These act as good ground covers for suppressing weeds and provide beauty before the garden really kicks into action. Of course, I do need to remove some of them periodically, otherwise they will completely cover the garden. A few years ago I learned that wild violets are the only host plant for fritillaries. "Without violets there would be no Fritillaries."
So give yourself and the butterflies a break. Leave some violets around your garden.
Update 9 20 2018: Gulf Fritillaries use Passion Vine as a host plant, so perhaps the violets are not the only host plant, but they do support many types of Fritillaries.