HotSpot source: Serviceability Agent
By sundararajan on Nov 01, 2006
The HotSpot Serviceability Agent (SA) is a set of APIs and tools for debugging HotSpot Virtual Machine. Unlike most previous debugging systems that assume a "cooperative" model in which the target process runs code to assist in the debugging process, the SA requires no code to be run in the target VM. The HotSpot SA uses primitives like symbol lookup and reading of process memory to implement its functionality. Because of that, HotSpot SA can work on core dumps as well. You may want to read this (somewhat dated, but still useful) paper - The HotSpot Serviceability Agent: An out-of-process high level debugger for a Java virtual machine.
Now, coming to the source code: In the source download from jdk6.dev.java.net, you may want to look at $JDK/hotspot/agent directory. There is some documentation of SA under $JDK/hotspot/agent/doc directory. The following tools are using SA.
Starting from JDK 6, for live processes, (few of) these tools (under few options) use attach-on-demand.