I can make text boxes in matplotlib fine. But I dont see how to remove them from a rendered plot? There seems to be no figure.text.clear() or figure.text(visible=False) after you draw a text box? How is this done? and unlike legends, you seem to be unable to make them draggable?

Text boxes are artists. As such, you should be able to do lots of things with them if you keep a reference to them. Hence, in any plotting code, instead of

fig.text(0, 0, 'My text')

you can do

textvar = fig.text(0, 0, 'My text')

If you've lost the references, though, all the text objects can be found in the texts attribute:

fig.texts # is a list of Text objects

In version 1.3.1, doing textvar.remove() generates a NotImplementedError (apparently fixed in 1.4). However, you can get around that to some degree by setting the visibility to False.

for txt in fig.texts:
    txt.set_visible(False)

will make all your text boxes disappear.