Caches Considered Harmful

For what seems like forever, designers have been adding more and more cache to systems to reduce latency to memory. This has been successful, but it hasn't been the only approach, but it has been the most typical.

But has it been Good?
  • Caches are very energy intensive (essentially large amount of SRAM close to the CPU). The larger they are, the more energy wasteful they are.
  • Caches, on average, produce a benefit on the order of sqrt(size), so the heat outpaces the benefit.
    • Of course, with heat you pay several times. You pay for the electricity to create the heat, you pay in the system design to cool the device, you pay in the data center to cool the entire system, and you typically pay a price in RAS because heat kills.
    • Notably, adding cores (providing enough memory bandwidth has been provided) provides nearly linear improvement in throughput.
    • And for cache experts, increasing the associativity increases their effectiveness.
  • Caches help us avoid dealing with the underlying issue of doing useful work while waiting for memory. Putting off the harder work of innovation, or at least limiting the innovation to the process level rather than the architecture level is a form of laziness.
  • When the data one wants isn't in the cache, it's often worse than it would have been had there been no cache (so fancy non-cache polluting loads and stores may be added to the ISA, and compiler, etc.
So what are the alternatives to caching?

As in the citations above, the key observation is that if one has additional "threads" ready to do useful work, that work can be done while awaiting the data to be returned (from memory, from cache, from disk, ... wherever) rather than keeping all that hot and possibly expensive iron (silicon) hot. And that's precisely what UltraSPARC T1 does  ).

So when you hear someone making a spurious claim about the UST1 being cache starved, ask them how big a cache they think it should have, and why? What level of associativity? What's the downside? And, of course, point out that the application performance is what counts, and it doesn't support the contention that the UltraSPARC T1 family systems are cache starved.


NB: of course, caches aren't all bad. If you are focused on minimum latency (fastest response time for a single thread) they can be very effective. But if your goal is the most aggregate work for the least power, they are certainly not your friend.

To learn more about caches


Comments:

I like Niagara... but I don't agree with you. > Caches are very energy intensive Caches are not energy intensive - in fact, caches use smaller die area and use less energy than logic.

Posted by Rayson (via proxy) on December 08, 2005 at 02:57 PM PST #

Conceding that I'm not a hw engineer; my notes from when I shared some labspace with folks using some nifty measuring devices: (1) at least some chips, for some workloads have a lot more heat from their caches than from at least some of their logic (I suspect it was an integer intensive workload with large miss rates, and the fpu was idle and used marginal energy, but I didn't note the details), (2) at least some chips (e.g. upcoming IA64 chips) have really massive caches (9M) as their "solution to the memory wall" which I am hard pressed to believe is as energy efficient (many, many transistors, and benefit ranging from 5%-sqrt(N) vs. order N for additional cores) and lastly (3) some chips logic is a lot more energy intensive than others. Compare Pentium-M vs. NetBurst (with it's double pumped logic in key circuits running at 2x the clock). But you have a point (and IBM Research's onchip DRAM cache I vaguely recall seeing a paper about probably makes it even stronger), cache is not the least power consuming thing to do with transistors. But when you include the effectiveness I think it's a relatively poorer choice than appropriate logic beyond a certain point. Just what that point is varies based on architecture, process, workload and is a far from easy choice for chip and system architects to make.

Posted by Keith Bierman on December 13, 2005 at 07:55 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

khb

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today