See:
- Moore’s law, Dennard scaling for semiconductors
- Neural scaling law for machine learning
People will say that an engineering paradigm might have reached the limit of a physical law, i.e., it stops holding. Oftentimes this is a physical reality of our world — perhaps the physics or the mathematics doesn’t support the law continuing to hold.
For example, the clock speeds on ICs (related closely to Moore’s law) reached a plateau in the 2010s, and they generally kept stable around 2-5 GHz. This is mainly because of high power draws with higher clock speeds. In that time, engineers at chip companies continued to squeeze increasing performance out of new generations of chips by advancing fundamental transistor technology, chip packaging, increasingly parallel hardware, and better inter-chip communication. So Moore’s law for performance continued being maintained.
In essence, even though a physical law might have ended, it’s possible for progress to continue at the same pace because we don’t have to continue to invest in a single path to improve. We can invest in other new trends.
”Despite the slowing down of one trend, the industry collectively remains moving forward at a breakneck pace due to other new emerging paradigms that are ripe for scaling and expansion.”1
Footnotes
-
From Scaling Laws, by Dylan Patel, Daniel Nishball, and AJ Kourabi. ↩