http://www.learnenglish.de/grammar/usedtotext.htm

You should use 'use to' without a d in sentences when it follows
'did' or 'didn't' (don't worry too much about this because lots of people get it
wrong).

http://www.learnenglish.de/grammar/usedtotext2.htm