Wednesday, January 23, 2008

MacOS X/DTrace tryst ends in tears.

See Adam's break-up note. This also got picked up on slashdot.

My thoughts, in no particular order:

  1. Shame on Apple. What is the point of building system-wide infrastructure that any random process can opt out of? Forget about DRM; what about legitimate uses of DTrace for system health monitoring? Won't all malware be sure to set the "don't dtrace me" bit now?
  2. This highlights the value of having system-wide instrumentation at the hardware level. Systems like VProbes do not allow a process, or indeed even a kernel, to "opt out." If the VM's user wants something traced, he gets it traced, by golly.
  3. On the other hand: if code really wants to evade VProbes, DTrace, debuggers, etc., the arms race is heavily stacked in favor of the sneaky code. E.g., suppose you've got some super-s3kr3t, double-plus DRM code that you absolutely don't want instrumented with DTrace's pid provider; on x86's, you could easily checksum the text to find smashed-in int3's. More elaborate evasion/detection schemes are possible too, similar in spirit to the VMM detection techniques summarized in our HOTOS paper about VMM detectors from last year. We concluded that creating a completely invisible VMM is an ill-posed problem, and trying too hard is a waste of time. I believe a similar dynamic is at work with dynamic tracing systems.