Which of the many memory related metrics exactly are you watching? I suppose it's memory active?
It's normal to see delays here, since "active memory" is based on statistical sampling of how long memory pages have not been touched by the guest OS. For pages to be considered "inactive" again, they must not have been touched by the guest for a certain amount of time, so you see a gradual decrease in the performance charts.
CPU usage metrics on the other hand are much more dynamic and can more reliably be measured, which is why you see immediate effects on these counters.