For a proper vertex coloring $c$ of a graph $G$, let $\varphi_c(G)$ denote the maximum, over all induced subgraphs $H$ of $G$, the difference between the chromatic number $\chi(H)$ and the number of colors used by $c$ to color $H$. We define the chromatic discrepancy of a graph $G$, denoted by $\varphi(G)$, to be the minimum $\varphi_c(G)$, over all proper colorings $c$ of $G$. If $H$ is restricted to only connected induced subgraphs, we denote the corresponding parameter by $\hat{\varphi}(G)$. These parameters are aimed at studying graph colorings that use as few colors as possible in a graph and all its induced subgraphs. We study the parameters $\varphi(G)$ and $\hat{\varphi}(G)$ and obtain bounds on them. We obtain general bounds, as well as bounds for certain special classes of graphs including random graphs. We provide structural characterizations of graphs with $\varphi(G)=0$ and graphs with $\hat{\varphi}(G)=0$. We also show that computing these parameters is NP-hard.