Don't Paint Just by Numbers