Show Notes
This week, I invited my friend Simon on the show to talk about AI infrastructure. This is a very interesting topic to think about. We talked about the reason for Nvidia’s market dominance and the problems that Intel and AMD have faced in developing their own market place. We talked about the number of chips involved and the challenges in building large data centers. Here is a chart of the Hopper deliveries for 2024. Keep in mind that Google also has their own TPU for inference. Here is a chart of Chinese Hyperscaler’s purchases of H20 in 2024. But keep in mind that they do rent compute from public/state built data centers that use smuggled in H100/H200 as well as Ascend chips. Here is the deliveries of H100/H800 in 2023. So if we just consider the Hopper deliveries from 2023 and 2024, the American hyperscalers have quite the computation advantage over Chinese hyperscalers. Although, the gap is much smaller if we factor in all the smuggled in chips. As I’ve discussed before, China had them everywhere to the point where they were sitting idle in many cases. I also asked Simon about the amount of chips needed to do inference. How was Tencent able to have enough compute for 8 million DeepSeek R1 requests at the same time with much less compute than what Google had. He sent me this chart of where the results improve logarithmically in compute for reasoning. So if you look at green line, going from 4 generations to 16 generations improved accuracy by about 15% and then going from 16 generations to 64 generations improved the result by another 5%. If we look at Google search results, they appear to be giving progressively better AI results on top. That is likely from running through and generation more tokens on their reasoning models. So you can choose to serve 8 million prompts at same time with vastly less compute, but you will also have inferior results. although you will also reach diminishing returns pretty soon. Here is a chart of rental cost from various Nvidia chips in China. Despite the increased demand for inference post DeepSeek, the cost of all Nvidia rental cost continue to drop, so they are likely not facing a crunch for computation yet.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit tphuang.substack.com
More from China Tech Talk
This week, I welcomed Ben Norton onto my show. I have been following Ben for a while on his youtube channel and X account. When I was in Beijing over the summer, I found out from some mutual contacts that Ben is in fact studying in Tsinghua University. I got in touch with him and we just couldn’t find a time to meet up. Fortunately, I was able to get Ben on my show this past week and it was great discussing with him about the topic of “moving to China and living in China”. That’s one thought I had when I was in Beijing. I am now a part time resident here and it’s pretty good place to live in. We talked about living in China, moving to China and the pros & cons of China. There is certainly a lot of things we agreed on. We actually, chatted for a bit afterward too. I hope I can meet up with Ben next summer when i’m there or if he ever visits the Northeast.
This week, I welcomed back my friend Snek on the show to talk about something that will inevitably effect the entire global economy, but especially the tech sector, which is the bond meltdown and rising oil prices across the board. I think we are at the point for many geopolitical reasons, foreign central banks are no longer price insensitive buyers of US Treasuries and this is part of what’s leading to surging yield globally. Of course, the surging bond issuance is another reason. now for the AI buildout, surging bond yield is a huge problem since massive amount of issuance will need to happen. Oracle is having some real problem with that. Same with Softbank. It’s gotten so bad for Oracle that they gave Tencent a huge discount on that data center compute lease because Tencent is able to pay 30% up front. You know things are bad when that happens. Let’s take Anthropic as an example. If you borrow $200B over a few years to pay for your data center build out and compute needs and you are facing almost junk bond ratings and 10 year yield is well over 5%. Then, your own yield could be close to 10% like it is getting there for softbank. 10% on $200B is $20B on interest per year. That changes the dynamic completely when it comes to margins on rest of your business. So, what’s going on in the debt market really matters. We have been too used to record low interest rate. That has allowed many bad decisions and investment over time. Now, we are essentially getting back to a more historically sane interest rate, but US government is $40T in debt. Which at 5% yield means $2T in just interest payment per year. So, that is what we are dealing with here.
I did a very early morning quick take on my trip to China. I’m sure there will be more on this since so much stuff happened while I was there. But I really wanted to just the message that China is becoming a really livable place and people are moving there for the quality of life. I met several people in Beijing this time who moved there from America from different reasons. It was interesting to hear about why they did it. I spoke about my own thoughts on what I enjoyed about living in Beijing and why I would consider moving there more permanently. I think there are still many reasons to not move to China. For example, the intensity of life and work is much higher there than here. The competition for everything is so much higher and this drives people toward addictions that are all about the short term consumerism satisfaction like looking at micro drama on their phones and snacking/drinking milk tea all the time. If I had to talk about what I think is problematic in Chinese society, the phone addiction and increased short termism seem to be the most obvious ones. And those are also problems that likely will get worse with more competition and AI. But in many ways, the quality of live in Beijing is also just so much better than here.
I had a chance to go over Tau Scaling and reusable rocket with Lei. As Lei is the person who shares my point of view the closest on Tau Scaling and Huawei’s efforts on this front, we really spent a lot of time explaining why what Huawei did in their design and production is quite special. Even if transistor cell itself can continue to scale smaller, that does not apply to memory cell sizes or analog components like copper connections and such. This episode was recorded before I went to China. Now that I’m back and Kirin 9050 has been unveiled, it’s a good time to publish my discussions on it.
I am currently in Beijing and have the pleasure of getting together with some friends in Tsinghua University in the AI school. I was welcomed by Alex and Qian Yu. We sat down talked a lot about the academia, AI research in China, getting talent to come to China and future of AI both in LLM and embodied AI. It’s quite the interesting experience to come here and see Zai in one corner, Unigroup building in another corner, the school for automation (using unreal new technology), Galbot robot booth, Infinigence, Vidu AI, Robotera and many other AI and semi startups. Pay attention to the tips they have about the transitioning toward Ascend from Nvidia chips by researchers. Also, check the part about researchers in China finally finding Kimi K3 to be good enough for vibe coding compared to GPT and Claude. There are still some concerns about inference sped for Kimi. I think they need to get more compute.