The second major component in any computer is the memory. Ideally, a memory should be extremely fast (faster than executing an instruction so the CPU is not held up by the memory), abundantly large, and cheap. Current technology is not developed to satisfy these requirements. The memory system is constructed as a hierarchy of layers.

The top layer consists of the registers internal to the CPU. They are made up of the same material as the CPU and are as fast as the CPU. Consequently, there is no delay in accessing them. The storage capacity available in them is typically according to the number of bits present. Programs must manage the registers.

The cache memory is mostly controlled by the hardware. Main memory is divided up into cache lines, typically 64 bytes, with addresses 0 to 63 in cache line 0, addresses 64 to 127 in cache line 1, and so on. The most heavily used cache lines are kept in a high-speed cache located inside or very close to the CPU.

When the program needs to read a memory word, the cache hardware checks to see if the line needed is in the cache. If it is, called a cache hit, the requested is satisfied from the cache and no memory request is sent over the bus to the main memory.

Cache hits normally take about two clock cycles. Cache misses have to go to memory, with a substantial time lag. Cache memory is limited in size due to its high cost. Some machines have two or even three levels of cache, each one slower and bigger than the one before it.

Main memory comes next. This is the workhorse of the memory system. Main memory is often called RAM (Random Access Memory). Currently, memories are tens to hundreds of megabytes and growing rapidly. All CPU requests that cannot be satisfied out of the cache go to main memory.

In addition to the kinds of memory discussed above, many computers have a small amount of nonvolatile random access memory. Unlike RAM, nonvolatile memory does not lose its contents when the power is switched off. ROM is programmed at the factory and cannot be charged afterward. It is fast and inexpensive. On some computers, the bootstrap loader used to start the computer is contained in ROM. Also, some Input and output cards come with ROM for handling low-level device control.