Recently I have heard a lot of people talking about how they change or reduce their dietary intake on 'non-workout' days. Don't be one of these people.
Recovery days are perhaps the most important nutritional intake days. These are the days when your body is rebuilding and repairing from the previous days (plural) workouts. And these 'non-workout' / recovery days are the days that you are actually making gains in your health and/or fitness. During a workout, we are breaking ourselves down and tapping our energy reserves, we don't gain during a workout, but rather gain in the hours to days following. Your neurotic mind may be telling you that because you didn't workout today that you "shouldn't" eat as much. But, this is just your mind messing with you and feeding (pun alert!) your neuroses about getting fat, or not getting in better shape.
So on your 'non-workout' / recovery days, make sure that you are keeping your calorie level and intake of healthy foods at the same or higher levels as your regular workout days.