This plugin adds a button to the widget’s admin page that will delete all inactive widgets.
Screenshot of what the button looks like
Extract the zip file and just drop the contents in the wp-content/plugins/ directory of your WordPress installation and then activate the Plugin from Plugins page.
Verify working in 3.5.1
Verify working in 3.5
Added a screenshot
Fixed some styling with the button to keep it from wrapping making it hard to read