No, you can do some things that are observable, e.g.:
ref = unsafePerformIO (newIORef True)
set x = unsafePerformIO (writeIORef ref x)
get = unsafePerformIO (readIORef ref)
On their own get and set are relatively harmless. And the ref is even a common practice. But having the ability to both set and get a mutable value means that you now have the ability to observe the effects.
Similarly, you could imagine having an SMT solver function and having a function that gets the list of running processes, both using unsafePerformIO. The latter can observe the effects of the former, so this is unsafe. Also, in this situation the list of running process probably changes all the time, so that is a problem all on its own.