Tai-Danae Bradley | Category Theory and Language Models Video

Justin Clarke-Doane is a professor of philosophy at Columbia University, whose interests span metaethics, epistemology, and the philosophy of logic & mathematics. In this thought provoking-discussion, Justin and I go deep into topics that are typically neglected by most mathematicians and scientists, namely the philosophy of mathematics and morality. Justin has contributed to both these areas via his book Morality and Mathematics, which takes the view that the standard position of being both a mathematical realist and moral antirealist is incoherent. Perhaps the most novel aspect of Justin's work is the treatment of the philosophy of mathematics and morality side-by-side, showing how these two topics, which are usually thought of as being unrelated, in fact have strong analogies. Along the way, we discuss many other foundational topics in epistemology and ethics, with elements of set theory, metaphysics, and logic sprinkled in. Patreon (bonus materials + video chat): https://www.patreon.com/timothynguyen Part I. Introduction

Jay McClelland is a pioneer in the field of artificial intelligence and is a cognitive psychologist and professor at Stanford University in the psychology, linguistics, and computer science departments. Together with David Rumelhart, Jay published the two volume work Parallel Distributed Processing, which has led to the flourishing of the connectionist approach to understanding cognition. In this conversation, Jay gives us a crash course in how neurons and biological brains work. This sets the stage for how psychologists such as Jay, David Rumelhart, and Geoffrey Hinton historically approached the development of models of cognition and ultimately artificial intelligence. We also discuss alternative approaches to neural computation such as symbolic and neuroscientific ones. Patreon (bonus materials + video chat):https://www.patreon.com/timothynguyen Part I. Introduction 00:00 : Preview 01:10 : Cognitive psychology 07:14 : Interdisciplinary work and Jay's academic journey 12:39 : Context affects perception 13:05 : Chomsky and psycholinguists 8:03 : Technical outline Part II. The Brain 00:20:20 : Structure of neurons 00:25:26 : Action potentials 00:27:00 : Synaptic processes and neuron firing 00:29:18 : Inhibitory neurons 00:33:10 : Feedforward neural networks 00:34:57 : Visual system 00:39:46 : Various parts of the visual cortex 00:45:31 : Columnar organization in the cortex 00:47:04 : Colocation in artificial vs biological networks 00:53:03 : Sensory systems and brain maps Part III. Approaches to AI, PDP, and Learning Rules 01:12:35 : Chomsky, symbolic rules, universal grammar 01:28:28 : Neuroscience, Francis Crick, vision vs language 01:32:36 : Neuroscience = bottom up 01:37:20 : Jay’s path to AI 01:43:51 : James Anderson 01:44:51 : Geoff Hinton 01:54:25 : Parallel Distributed Processing (PDP) 02:03:40 : McClelland & Rumelhart’s reading model 02:31:25 : Theories of learning 02:35:52 : Hebbian learning 02:43:23 : Rumelhart’s Delta rule 02:44:45 : Gradient descent 02:47:04 : Backpropagation 02:54:52 : Outro: Retrospective and looking ahead Image credits:http://timothynguyen.org/image-credits/ Further reading: Rumelhart, McClelland. Parallel Distributed Processing. McClelland, J. L. (2013). Integrating probabilistic models of perception and interactive neural networks: A historical and tutorial review

Michael Freedman is a mathematician who was awarded the Fields Medal in 1986 for his solution of the 4-dimensional Poincare conjecture. Mike has also received numerous other awards for his scientific contributions including a MacArthur Fellowship and the National Medal of Science. In 1997, Mike joined Microsoft Research and in 2005 became the director of Station Q, Microsoft’s quantum computing research lab. As of 2023, Mike is a Senior Research Scientist at the Center for Mathematics and Scientific Applications at Harvard University. Patreon (bonus materials + video chat): https://www.patreon.com/timothynguyen In this wide-ranging conversation, we give a panoramic view of Mike’s extensive body of work over the span of his career. It is divided into three parts: early, middle, and present day, which respectively include his work on the 4-dimensional Poincare conjecture, his transition to topological physics, and finally his recent work in applying ideas from mathematics and philosophy to social economics. Our conversation is a blend of both the nitty-gritty details and the anecdotal story-telling that can only be obtained from a living legend. I. Introduction 00:00 : Preview 01:34 : Fields Medalist working in industry 03:24 : Academia vs industry 04:59 : Mathematics and art 06:33 : Technical overview II. Early Mike: The Poincare Conjecture (PC) 08:14 : Introduction, statement, and history 14:30 : Three categories for PC (topological, smooth, PL) 17:09 : Smale and PC for d at least 5 17:59 : Homotopy equivalence vs homeomorphism 22:08 : Joke 23:24 : Morse flow 33:21 : Whitney Disk 41:47 : Casson handles 50:24 : Manifold factors and the Whitehead continuum 1:00:39 : Donaldson’s results in the smooth category 1:04:54 : (Not) writing up full details of the proof then and now 1:08:56 : Why Perelman succeeded II. Mid Mike: Topological Quantum Field Theory (TQFT) and Quantum Computing (QC) 1:10:54: Introduction 1:11:42: Cliff Taubes, Raoul Bott, Ed Witten 1:12:40 : Computational complexity, Church-Turing, and Mike’s motivations 1:24:01 : Why Mike left academia, Microsoft’s offer, and Station Q 1:29:23 : Topological quantum field theory (according to Atiyah) 1:34:29 : Anyons and a theorem on Chern-Simons theories 1:38:57 : Relation to QC 1:46:08 : Universal TQFT 1:55:57 : Witten: Donalson theory cannot be a unitary TQFT 2:01:22 : Unitarity is possible in dimension 3 2:05:12 : Relations to a theory of everything? 2:07:21 : Where topological QC is now III. Present Mike: Social Economics 2:11:08 : Introduction 2:14:02 : Lionel Penrose and voting schemes 2:21:01 : Radical markets (pun intended) 2:25:45 : Quadratic finance/funding 2:30:51 : Kant’s categorical imperative and a paper of Vitalik Buterin, Zoe Hitzig, Glen Weyl 2:36:54 : Gauge equivariance 2:38:32 : Bertrand Russell: philosophers and differential equations IV: Outro 2:46:20 : Final thoughts on math, science, philosophy 2:51:22 : Career advice

Marcus Hutter is an artificial intelligence researcher who is both a Senior Researcher at Google DeepMind and an Honorary Professor in the Research School of Computer Science at Australian National University. He is responsible for the development of the theory of Universal Artificial Intelligence, for which he has written two books, one back in 2005 and one coming right off the press as we speak. Marcus is also the creator of the Hutter prize, for which you can win a sizable fortune for achieving state of the art lossless compression of Wikipedia text. Patreon (bonus materials + video chat): https://www.patreon.com/timothynguyen In this technical conversation, we cover material from Marcus’s two books “Universal Artificial Intelligence” (2005) and “Introduction to Universal Artificial Intelligence” (2024). The main goal is to develop a mathematical theory for combining sequential prediction (which seeks to predict the distribution of the next observation) together with action (which seeks to maximize expected reward), since these are among the problems that intelligent agents face when interacting in an unknown environment. Solomonoff induction provides a universal approach to sequence prediction in that it constructs an optimal prior (in a certain sense) over the space of all computable distributions of sequences, thus enabling Bayesian updating to enable convergence to the true predictive distribution (assuming the latter is computable). Combining Solomonoff induction with optimal action leads us to an agent known as AIXI, which in this theoretical setting, can be argued to be a mathematical incarnation of artificial general intelligence (AGI): it is an agent which acts optimally in general, unknown environments. The second half of our discussion concerning agents assumes familiarity with the basic setup of reinforcement learning. I. Introduction 00:38 : Biography 01:45 : From Physics to AI 03:05 : Hutter Prize 06:25 : Overview of Universal Artificial Intelligence 11:10 : Technical outline II. Universal Prediction 18:27 : Laplace’s Rule and Bayesian Sequence Prediction 40:54 : Different priors: KT estimator 44:39 : Sequence prediction for countable hypothesis class 53:23 : Generalized Solomonoff Bound (GSB) 57:56 : Example of GSB for uniform prior 1:04:24 : GSB for continuous hypothesis classes 1:08:28 : Context tree weighting 1:12:31 : Kolmogorov complexity 1:19:36 : Solomonoff Bound & Solomonoff Induction 1:21:27 : Optimality of Solomonoff Induction 1:24:48 : Solomonoff a priori distribution in terms of random Turing machines 1:28:37 : Large Language Models (LLMs) 1:37:07 : Using LLMs to emulate Solomonoff induction 1:41:41 : Loss functions 1:50:59 : Optimality of Solomonoff induction revisited 1:51:51 : Marvin Minsky III. Universal Agents 1:52:42 : Recap and intro 1:55:59 : Setup 2:06:32 : Bayesian mixture environment 2:08:02 : AIxi. Bayes optimal policy vs optimal policy 2:11:27 : AIXI (AIxi with xi = Solomonoff a priori distribution) 2:12:04 : AIXI and AGI. Clarification: ASI (Artificial Super Intelligence) would be a more appropriate term than AGI for the AIXI agent. 2:12:41 : Legg-Hutter measure of intelligence 2:15:35 : AIXI explicit formula 2:23:53 : Other agents (optimistic agent, Thompson sampling, etc) 2:33:09 : Multiagent setting 2:39:38 : Grain of Truth problem 2:44:38 : Positive solution to Grain of Truth guarantees convergence to a Nash equilibria 2:45:01 : Computable approximations (simplifying assumptions on model classes): MDP, CTW, LLMs 2:56:13 : Outro: Brief philosophical remarks

Richard Borcherds is a mathematician and professor at University of California Berkeley known for his work on lattices, group theory, and infinite-dimensional algebras. His numerous accolades include being awarded the Fields Medal in 1998 and being elected a fellow of the American Mathematical Society and the National Academy of Sciences. Patreon (bonus materials + video chat): https://www.patreon.com/timothynguyen In this episode, Richard and I give an overview of Richard's most famous result: his proof of the Monstrous Moonshine conjecture relating the monster group on the one hand and modular forms on the other. A remarkable feature of the proof is that it involves vertex algebras inspired from elements of string theory. Some familiarity with group theory and representation theory are assumed in our discussion. I. Introduction 00:25: Biography 02:51 : Success in mathematics 04:04 : Monstrous Moonshine overview and John Conway 09:44 : Technical overview II. Group Theory 11:31 : Classification of finite-simple groups + history of the monster group 18:03 : Conway groups + Leech lattice 22:13 : Why was the monster conjectured to exist + more history 28:43 : Centralizers and involutions 32:37: Griess algebra III. Modular Forms 36:42 : Definitions 40:06 : The elliptic modular function 48:58 : Subgroups of SL_2(Z) IV. Monstrous Moonshine Conjecture Statement 57:17: Representations of the monster 59:22 : Hauptmoduls 1:03:50 : Statement of the conjecture 1:07:06 : Atkin-Fong-Smith's first proof 1:09:34 : Frenkel-Lepowski-Meurman's work + significance of Borcherd's proof V. Sketch of Proof 1:14:47: Vertex algebra and monster Lie algebra 1:21:02 : No ghost theorem from string theory 1:25:24 : What's special about dimension 26? 1:28:33 : Monster Lie algebra details 1:32:30 : Dynkin diagrams and Kac-Moody algebras 1:43:21 : Simple roots and an obscure identity 1:45:13: Weyl denominator formula, Vandermonde identity 1:52:14 : Chasing down where modular forms got smuggled in 1:55:03 : Final calculations VI. Epilogue 1:57:53 : Your most proud result? 2:00:47 : Monstrous moonshine for other sporadic groups? 2:02:28 : Connections to other fields. Witten and black holes and mock modular forms.