Original Link: Commoditizing the Petaflop — with George Hotz of the tiny corp
Summary of Latent Space Podcast with Swyx, Alessio, and Guest Geohot (George Hotz)
Introduction:
Swyx is the writer and editor of Latent Space.
Alessio is Partner and CTO in residence at Decibel Partners.
Geohot (George Hotz) is the guest, known for his pioneering hacks such as unlocking the first iPhone, breaching the PS3 system, and founding Comma.ai. He has a controversial history with tech giants and regulatory authorities.
Discussion Points:
Geohot has concerns about the closed nature of some major tech players in the AI space, emphasizing the need for open-source and accessible tools.
The interview is peppered with technical insights, discussions on AI's future, and Geohot's experiences and beliefs.
Geohot's Achievements: Traded the unlocked iPhone for a car and iPhones, hacked PS3, faced a lawsuit from Sony, started Comma.ai but faced governmental restrictions. He clarifies that his products are for developers, emphasizing the difference between a dev kit and a standard product.
Hero's Journey: Discussion on Geohot's blog post relating to the concept of "The Hero's Journey" and its relation to TinyGrad, a project he's now heavily involved in.
Concerns on AI Regulation: Geohot expresses concern about potential government restrictions on AI, using Sam Altman's congressional hearing as a pivotal moment that made him realize the importance of his work.
TinyGrad & TinyCorp: Geohot emphasizes the need for simplicity in AI and machine learning models, comparing complex and reduced instruction sets. He advocates for a "RISC" approach, simplifying the ML process.
AI Chips Debate: The discussion revolves around the efficiency of AI chips and the infrastructure required for optimal performance. George suggests that if one can't develop an efficient ML framework for standard GPUs, they can't for a unique chip.
Turing Completeness: Both George and Swyx discuss the downsides of Turing Completeness in ML. Turing Completeness makes it easier to write codes but isn't always the most efficient. The conversation touches on TPUs, how they are a better option than CUDA, and the problem with closed-source systems like Google's TPU.
Explanation of Systolic Arrays: An attempt to demystify the concept of Systolic Arrays, which are efficient for power but may not be the best fit for all computations.