Humor is contrast

My answer to the question "What makes something funny?" is contrast between what is expected and what is observed. Humor is observing the unexpected. 

I sense there is another element, but I do not know what it is. Perhaps that the observation is innocuous? For I can think of unexpected observations that would be terrifying rather than hilarious.

Yet I think that contrast is at the heart of humor. Let's have an example. Two lovers are having dinner at a fancy restaurant. They are both wearing nice clothes, smiling, staring into each other's eyes. They lean closer together, and the man opens his mouth to say something. 

If he says something like "You look beautiful" or "I love you" we will not think that is funny. That is expected. But if he says "I think I made a mistake on my tax return" or "Are those your real teeth?" we might think that is funny.