Remember What Matters and Forget What Does Not