Understanding LLaMA2 on When Moore's Law Ends

Understanding LLaMA2 Part 1 Model Architecture

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

Here I’m capturing the details of llama’s model architecture use PlantUML component diagram, using the following 2 GitHub repos as references

In the following diagram

The “gray boxes” are tensors with their shapes in parentheses
The “colored round dots” are operations with their categories include
- “memops”: memory operations
- “module”: PyTorch module operators
- “compute”: non-PyTorch computational operators
The “grouping boxes” are to make the architecture modularized close to the original source code
The “yellow boxes” are my in-depth comments and understanding of different parts of the architecture

In order to make the diagram more readable, here I choose to use abbreviations of parameters and hyper-parameters, which are explained on the upper-right corner of the diagram.

Understanding LLaMA2 Part 2 KV Cache

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

Following up with Understanding LLaMA2 Part 1 Model Architecture, this diagram explains LLaMA model architecture with KV Cache support. We follow the same legend as well as the abbreviations.

Understanding LLaMA2 Part 3 PyTorch Implementation

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

https://github.com/jimwang99/understanding-llama2/tree/main/pytorch

Above GitHub repo is an implementation of LLaMA2 and test-case use TinyStories in PyTorch

Example output

------ tinystories110M ------
Once upon a time, there was a little girl named Lily. She loved to play with her toys, especially her favorite teddy bear. One day, Lily's mommy gave her a big box to play with. Lily was so happy and she jumped up and down with joy.
Lily's mommy said, "Be careful, Lily. Don't break the box." Lily nodded her head and promised to be careful. She opened the box and saw that it was empty. "Mommy, can we fill the box with my toys?" Lily asked.
"Of course, sweetie," her mommy replied. They started to fill the box with all of Lily's toys. Suddenly, Lily's teddy bear fell out of the box and onto the floor. "Oh no! Teddy fell out of the box!" Lily cried.
"Don't worry, Lily. We'll pick him up and put him back in the box," her mommy said. They picked up the teddy bear and put him back in the box. Lily was so happy that her toys were safe in the box and she could play with them again.
------- the end ------

Understanding LLaMA2 Part 4 ExecuTorch Runtime

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

I was involved in the early ExecuTorch definition phase and had used its predecessor Lite Interpreter extensively in work. I really like this idea and its design. This is a great effort among multiple industry leading companies. So I’m trying the open-source version with LLaMA2.

ExecuTorch Overview #

What is ExecuTorch?#

ExecuTorch is an end-to-end solution for enabling on-device inference capabilities across mobile and edge devices including wearables, embedded devices and microcontrollers. It is part of the PyTorch Edge ecosystem and enables efficient deployment of PyTorch models to edge devices. Key value propositions of ExecuTorch are:

Understanding LLaMA2 Part 5 Training with TinyStories

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

How to train a transformer based model, like LLaMA2, from scratch? Andrej Karpathy has open-sourced llama2.c project on GitHub. My learning process is “duplicate and rewrite”, which involves following the example but rewrite the code completely in my own coding style and language. I did the same and along the way I’ve learned something new. In this blog post, I’m going to break down the training code rewritten by myself, line by line, explain what and most importantly why.

Understanding LLaMA2 Part 6 Positional Encoding

Mon, 01 Jan 0001 00:00:00 +0000

#software #ai #llm #open-source

Understanding LLaMA2 on When Moore's Law Ends

Understanding LLaMA2 Part 1 Model Architecture

Understanding LLaMA2 Part 2 KV Cache

Understanding LLaMA2 Part 3 PyTorch Implementation

Understanding LLaMA2 Part 4 ExecuTorch Runtime

ExecuTorch Overview#

What is ExecuTorch?#

Understanding LLaMA2 Part 5 Training with TinyStories

Understanding LLaMA2 Part 6 Positional Encoding

ExecuTorch Overview #