I am thinking that there is some way to block laser light, i'm thinking polarizing filters?
As for refracted laser light, it really depends on how coherent the refracted light remains. My guess with an ordinary laser pointer, this wouldn't be a problem. The light which exits the gem may be a tiny fraction of the energy output by the laser, and the light will be scattered.
The reason why lasers are dangerous is because they focus all their energy into one place and in similar phase. If that quality is broken it's safety depends on the levels of concentration and coherence. Regardless, very high output lasers, greater than what you'd find in laser pointers, should be handled with caution.
In any case, you should use eye protection when using a laser with an optical device.