ml.cu

Today I tried to go through the /careers and /blog pages of various ai labs & startups because job postings reveal what labs believe is scarce before it becomes a course or a textbook chapter, haha. As I increasingly map the landscape, it seems that learning the common substrate, understanding what every major layer does and then going deep somewhere might be the way forward. Everything boils down to correctness, speed and scale BUT now we got a meta layer too. I'll write about that later on. My (interest) is in research engineering, hence I'll write from that perspective.

End objective is to have the ability to read a paper, inspect an implementation and understand why an experiment behaved differently from expectation, and in the process of doing so, eventually give an ambiguous problem some tangible shape.

My focus was on systems programming, and still don't know how a CNN works -- currently I work on evals & harnesses (domain specific). Synthetic fileds like ML, barely have any defined pathway. That's so much fun.

I read an article yesterday...well, now I wish to expand my territory.

1. I would love to start with Stanford CS336 Language Modeling from Scratch, because it covers the whole stack from implementing a transformer to writing triton kernels, profiling, evaluating, post training and what not.
2. After this, I would like to go through the technical report of DeepSeek V3 to understand how a frontier training system works. It's very likely I won't understand that, so I think HuggingFace's Smol Training Playbook will come handy.
3. I might look into these books: How to Scale your Model & Ultrascale Playbook
4. Then there is the Post Training Agents youtube playlist by HuggingFace. Modal and Prime Intellect will be in my radar, for obvious reasons.
5. Well, then we have evals. I already work on this. There are some good articles (like this) written by Anthropic. Essentially it's just building your own task, grader, environment and doing failure analysis. I really hate reading through rollouts though. This might come useful, HF's also got a great guide.
6. Harnesses...I have a lot to write about it. In short the topology of the task drives a lot of design decisions. OpenAI has got a good writeup, but building one for different feedback densities is heck of a game.
7. For inference and performance engineering, I'll go with whatever resources GPU MODE & Wafer has already compiled. There are other playgrounds too, like Tensara, LeetGPU, Tensortonic, and a few more aswell. Baseten has some amazing articles on this topic. Agentic Kernel in Production is one of those. Yes, this field is going more and more autonomous, verification is only the bottleneck. And Wafer is doing an amazing job.
8. I'm too dumb for compilers (yet) but if I ever need to understand them, I'd back track after going through the career page of Modular & the PyTorch docs.
9. For serving and infrastructure, I think going through the bottlenecks faces by Modal, Fireworks, Baseten and Together would be cool. Lots of distributed systems related literature.
10. Hardware is not my forté, Cerebras is good, right?
11. Autoresearch. Deserves a special mention. This combines agents, search, evals, systems and whatever domain is being optimized together. Meta-something. The frontier is beginning to optimize the research process itself.

I also came across this book on building LLMs, WandB has a guide on training LLMs, then I hear about RLMs and Meta Harnesses. Why would Microsoft have a team called Froggy? What is a kernel anyway? Hot chips? Why does CERN have a full length video on training LLMs?

Make things improve themselves.

I'm fed up with college, and how it's trying to bottleneck my progress. All I understand is that, to accelerate scientific progress, we need AI. We need to generate intelligence, build the platform to execute it, and then amplify it. Then use intelligence to improve all three. Open science and open intelligence. That's it.

I, the stupid intern, might read agent rollouts, inspect behaviour, check what the grader says, tweak the harness, hill climb the benchmark again. That's evals! Agent failed? Bad task? Weak verifier? Sparse reward? Reward hacking? Context bloated? Tool call broke? State contaminated? Maybe the whole multi agent topology is stupid. Supervisor? Islands? Swarms? Population search? Who talks to whom? Who gets compute? Who gets killed? Fine, let's change. Still bad? Maybe the environment is the problem. Better sandbox! Faster resets! More parallel copies! Better rewards, partial credit, curriculum, self play (yes, what??) synthetic tasks. Whoops! post training! Now I need rollouts, lots of them. Stragglers? Async workers? Long horizon credit assignment? GPUs sitting idle because env are slow? No data? Generate synthetic data! filter, dedup-rank, balance & ask which samples are even worth? Still nothing? Maybe the model simply cannot do it. Time to pretrain. Better data? Better architecture? Dense or MoE? Long context? Better optimizer? Scaling laws? Loss spikes? Expert collapse? Doesn't fit? More GPUs! Now distributed trainingn!! Sharding, tensor parallel? pipeline parallel! expert parallel?? checkpointing, communication, AllReduce, all-to-all, what lol? Done? Need inference. Prefill, decode, KV cache, batching, speculative decoding, quantization. Throughput improved but latency got worse? Great, now serving! Routing, autoscaling, model placement, load balancing, tail latency, queuing, idle GPUs here, overloaded GPUs?? Profiler says one operator dominates? Kernel time! CUDA, Triton, fusion, tiling, memory bandwidth, Tensor Cores, occupancy. Nice, kernel is 25% faster! End to end got slower? Of course. Runtime changed. Shapes changed too! Compiler in the way? Graph capture, fusion, lowering, codegen. Still slow? Hardware! HBM, interconnects, racks, power, cooling. Fix that, inference is cheaper, which means more rollouts, which means more RL data! which means a better policy! which means different agent behaviour! which means new trajectories! new failure modes, new evals, which means I stay employed!! So I'm back reading the rollout again. Why am I manually dealing with the entire mess? Let agents generate hypotheses, run experiments, allocate compute, mutate code, optimize kernels, synthesize data, train models, deploy them, evaluate them, and feed failures back into the next round. Did that work? No? Fine. Back to the rollout.

YES, it's an entire loop-ity loop. Yes, I have my mid semester exams in 20 days, I don't want to go back there.