Try Mixtral-8x7B-v0.1 on Google Colab

I tried “Mixtral-8x7B-v0.1” of “Mistral AI” with “Google Colab”, so I summarized it.

Platform: Google Colab Pro/Pro+ A100.

table of contents

  1.  Mixtral-8x7B-v0.1
  2.  Mixtral-8x7B-v0.1 model
  3. Execution on Colab

1. Mixtral-8x7B-v0.1

“ Mixtral 8x7B “ is a high quality “SMoE” (sparse mixture-of-experts) model developed by “ Mistral AI “. Inference is 6x faster and outperforms Llama2 70B on most benchmarks.

You can read more about this model here in detail

Introducing Mixtral 8x7B: Revolution in AI Language Models (Better then CHATGPT3.5 & LLAMA-70B)
Exploring the Breakthroughs in Language Processing with the Advanced SMoE Architecture

2. Mixtral-8x7B-v0.1 model

“Mixtral-8x7B-v0.1” is currently available in only two models.

・mistralai/Mixtral-8x7B-v0.1 : Base model
・mistralai/Mixtral-8x7B-Instruct-v0.1 : Instruction model

3. Execution on Colab

Here are the steps to run it in Colab:

(1) Open the Colab notebook and select “A100” for “GPU” in the menu “Edit → Notebook Settings”.

(2) Package installation.

# Let's install packages
!pip install -U transformers sentencepiece accelerate bitsandbytes

(2) Tokenizer and model preparation.
This time, I loaded “ mistralai/Mixtral-8x7B-Instruct-v0.1 “ with 4bit quantization . It took about 20 minutes.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# prepare tokenizer
tokenizer = AutoTokenizer.from_pretrained(
"mistralai/Mixtral-8x7B-Instruct-v0.1",
)
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mixtral-8x7B-Instruct-v0.1",
torch_dtype=torch.bfloat16,
load_in_4bit=True,
device_map="auto",
trust_remote_code=False,
)

(3) Performing inferences.


messages = [
{"role": "user", "content": "Who is the cutest in Madoka Magica?"},
]


with torch.no_grad():
token_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
output_ids = model.generate(
token_ids.to(model.device),
temperature=0.5,
do_sample=True,
top_p=0.95,
top_k=40,
max_new_tokens=256,
)
output = tokenizer.decode(output_ids[0][token_ids.size(1) :])
print(output)
The concept of "cute" is subjective and differs from person to person. In the anime series Madoka Magica, different characters may appeal to different people for different reasons. Some fans may find Madoka Kaname the cutest due to her kind and innocent personality, while others may find Homura Akemi's determination and resilience endearing. Still, some may find Kyubey's quirky and energetic personality the most appealing. In the end, which Madoka Magica character is the cutest is a matter of personal preference. </s>

Leave a Reply