Integration guides

Code Llama

Code Llama is an open-source family of LLMs based on Llama 2 providing SOTA performance on code tasks. It consists of:

Foundation models (Code Llama)
Python specializations (Code Llama - Python), and
Instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each.

See the recipes here for examples on how to make use of Code Llama.

The following diagram shows how each of the Code Llama models is trained:

(Fig: The Code Llama specialization pipeline. The different stages of fine-tuning annotated with the number of tokens seen during training)

One of the best ways to try out and integrate with Code Llama is using Hugging Face ecosystem by following the blog here, which has:

Demo links for all versions of Code Llama
Working inference code for code completion
Working inference code for code infilling between code prefix and suffix as inputs
Working inference code to do 4-bit loading of the 34B model so it can fit on consumer GPUs
Guide on how to write prompts for the instruction models to have multi-turn conversations about coding
Guide on how to use Text Generation Inference for model deployment in production
Guide on how to integrate code autocomplete as an extension with VSCode
Guide on how to evaluate Code Llama models

If the model does not perform well on your specific task, for example if none of the Code Llama models (7B/13B/34B/70B) generate the correct answer for a text to SQL task, fine-tuning should be considered. This is a complete guide and notebook (here) on how to fine-tune Code Llama using the 7B model hosted on Hugging Face. It uses the LoRA fine-tuning method and can run on a single GPU.

As shown in the Code Llama References (here), fine-tuning improves the performance of Code Llama on SQL code generation, and it can be critical that LLMs are able to interoperate with structured data and SQL, the primary way to access structured data - we are developing demo apps in LangChain and RAG with Llama 2 to show this.

Compatible extensions

In most of the cases, the simplest method to integrate any model size is through ollama, occasionally combined with litellm. Ollama is a program that allows quantized versions of popular LLMs to run locally. It leverages the GPU and can even run Code Llama 34B on an M1 mac. Litellm is a simple proxy that can serve an OpenAI style API, so it's easy to replace OpenAI in existing applications, in our case, extensions

Continue

This extension can be used with ollama, allowing for easy local only execution. Additionally, it provides a simple interface to 1/ Chat with the model directly running inside VS Code and 2/ Select specific files and sections to edit or explain.

This extension is an effective way to evaluate Llama because it provides simple and useful features. It also allows developers to build trust, by creating diffs for each proposed change and showing exactly what is being changed before saving the file. Handling the context for the LLM is easy and relies heavily on keyboard shortcuts. It's important to note that all the interactions with the extension are recorded in jsonl format. The objective is to provide data for future fine tuning of the models based on the feedback recorded during real world usage as well.

Steps to install with ollama

Install ollama and pull a model (e.g. ollama pull codellama:13b-instruct)
Install the extension from Visual Studio Code marketplace
Open the extension and click on the + sign to add models
Select Ollama as a provider
In the next screen, select the model and size pulled from with ollama
Select the model in the convo and start using the extension

Steps to install with TGI

For better performance or usage in non-compatible hardware, TGI can be used in a server to run the model. For example, ollama on Intel Macs is too slow to be useful, even with the 7B models. On the contrary, M1 macs can run the 34 Meta Code Llama models quickly. For this, you should have TGI running on a server with appropriate hardware, as detailed in this guide. Once Continue.dev is installed, follow these steps:

Open the configs with /config
Use the HuggingFaceTGI class and pass your instance URL in the server_url parameter:
Assign a name to it and save the config file.

llm-vscode

This extension from Hugging Face provides an open alternative to the closed sourced GitHub Copilot, allowing for the same functionality, context based autocomplete suggestions, to work with open source models. It works out of the box with a HF Token and their Inference API but can be configured to use any TGI compatible API. For usage with a self-hosted TGI server, follow these steps:

Install llm-vscode from the marketplace
Open the extension configs
Select the correct template for the model published in your TGI instance in the Config Template field. For testing, used the one named codellama/CodeLlama-13b-hf
Pass in the URL to your TGI instance in the Model ID or Endpoint field.
To avoid rate limiting messages, login to HF by providing a read only token. This was necessary even for a self-hosted instance.

It currently does not support local models unless TGI is running locally. It would be great to add ollama support to this extension, as it would accelerate the inference with the smaller models by avoiding the network.

Was this page helpful?

Yes

On this page

Code Llama

Compatible extensions

Table Of Contents

Code Llama

Compatible extensions