Memorize What Matters