Running Meta Llama on Windows

For this tutorial, we will be using Meta Llama models already converted to Hugging Face format. However, if you’d like to download the original native weights, click on the "Files and versions" tab and download the contents of the original folder.

If you prefer, you can also download the original weights from the command line using the Hugging Face CLI:

pip install huggingface-hub

huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --include "original/*" --local-dir meta-llama/Llama-3.1-8B-Instruct

Running the model

In this example, we will showcase how you can use Meta Llama models already converted to Hugging Face format using Transformers. To use the model with Transformers, we will be using the pipeline class from Hugging Face. We recommend that you use a Python virtual environment for running this demo. In this demo, we are using Miniconda, but you can use any virtual environment of your choice.

pip install -U transformers --upgrade

pip install accelerate

For our script, open the editor of your choice, and create a Python script. We’ll first add the imports that we need for our example:

import transformers
import torch
from transformers import AutoTokenizer

Let's define the model we’d like to use. In our demo, we will use the 8B instruct model which is fine tuned for chat:

model = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model)

To use our model for inference:

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    'I have tomatoes, basil and cheese at home. What can I cook for dinner?\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    truncation = True,
    max_length=400,
)

Finally, we add the following to provide input, and information on how to run the pipeline:

for seq in sequences:
    print(f"Result: {seq['generated_text']}")

pip install -U "huggingface_hub[cli]"

followed by

huggingface-cli login

python llama3-hf-demo.py