It is a porous stone so the more exposed it is to chemicals from your skin the more it will absorb. It is not dangerous but the stone will chip or crack easily.
Here is an exerpt from minerals.net:
Lapis Lazuli is a sensitive gemstone, and can be chipped or cracked easily when banged. It is also relatively soft so care should be exercised to prevent it from getting scratched. It is also slightly porous and should be protected from chemicals and cleaning solvents.