Average Memory Access Time (AMAT)

Learning Outcomes

• Define hit rate, hit time, miss rate, and miss penalty.
• Use the average memory access time (AMAT) formula to compare multi-level cache designs.

🎥 Lecture Video

Because performance is the major reason for a memory hierarchy, it is important to measure the time to service hits or misses. We therefore define the following terminology in #tab-cache-terminology:

Because the cache is smaller and built using faster memory parts, the hit time will be much smaller than the miss penalty, which includes the time to access the next level in the hierarchy.

Average Memory Access Time

The time to access data for both hits and misses affects performance. Designers sometimes use average memory access time (AMAT) as a way to compare cache designs. From P&H 5.4:

Average memory access time is the average time to access memory considering both hits and misses and the frequency of different accesses.

$$\text{AMAT} = \text{Hit Time} + \text{Miss Rate} \times \text{Miss Penalty}$$

We will use the following assumptions in this course:

• On a cache miss, the total time to retrieve data is the sum of hit time plus miss penalty.
• The miss rate of a lower-level cache (e.g., L2) is the fraction of misses from a higher-level cache (e.g., L1) that also miss in this lower-level cache.

Solution

Using Equation #eq-amat:

$$\begin{aligned} \text{AMAT} &= \text{L1 Hit Time} + \text{L1 Miss Rate} \times \text{L1 Miss Penalty} \\ &= 1 + 0.05 \cdot 200 \\ &= 11 \text{ cycles} \\ \end{aligned}$$

When the miss penalty is incurred, we still incur round-trip hit time. #fig-amat-l1-only-tree illustrates the two cases.

Single-layer cache performance analysis. 95% of the time, we incur 1 cycle delay to access the L1 cache. 5% of time, we incur 201 cycles of delay (to access the L1 cache and to access memory).

Solution

Based on AMAT assumptions, the miss rate of the L2 cache is the fraction of misses from the L1 cache that also miss in the L2 cache.

We can use Equation #eq-amat recursively:

$$\begin{aligned} \text{AMAT} &= \text{L1 Hit Time} + \text{L1 Miss Rate} \times \text{L1 Average Miss Penalty} \\ &= \text{L1 Hit Time} + \text{L1 Miss Rate} \times \bigl(\text{L2 Hit Time} + \text{L2 Miss Rate} \times \text{L2 Miss Penalty}\bigr) \\ &= \text{L1 Hit Time} + \text{L1 Miss Rate} \times \bigl(5 + 0.15 \cdot 200\bigr) \\ &= 1 + 0.05 \cdot 35 \\ &= 2.75 \text{ cycles} \\ \end{aligned}$$

Now, L1 miss penalty includes both the L2-hit case and the L2-miss case, as shown in #fig-amat-l1-l2-tree.

Two-layer cache performance analysis. 95% of the time, we incur 1 cycle delay to access the L1 cache. 5% of the time, we miss the L1 cache. Of this L1 miss scenario, 85% of the time we incur 6 cycles of delay (to access both the L1 and L2 cache). 15% of the time we incur 206 cycles of delay (to access the L1 cache, the L2 cache, and memory).

The L1 and L2 cache design is 4 times as fast as the L1-only cache design!

Preview: Cache Optimizations

We mentioned that AMAT is used to compare cache designs. The key performance hit to AMAT is miss rate. This can be measured over multiple program benchmarks, each with different memory access patterns.

In this section, we have seen one way to optimize cache performance by introducing multilevel caches to reduce miss penalty.

In this chapter, we will introduce the key principles of cache design. Then, with these design principles in mind, we revisit basic optimization techniques for improving cache performance.

Footnotes

1. Hashemi et al. “Learning Memory Access Patterns.” 2018 arXiV:1803.02329 ↩