Meta’s open-source Llama model family has recently welcomed a new member, Code Llama, a specialized model designed for code generation. Building upon the foundation of Llama 2, Code Llama undergoes further fine-tuning using a specific code dataset. Meta emphasizes that the open-source license for Code Llama, similar to Llama 2, allows for free usage in both research and commercial applications. With a published research paper titled “Code Llama: Open Foundation Models for Code” spanning 47 pages and authored by 25 individuals, the unveiling of Code Llama has attracted significant attention.
Code Llama, available in three versions with parameter sizes of 7B, 13B, and 34B, supports various programming languages including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash. It boasts stable context generation for up to 100,000 tokens and even demonstrates superior performance compared to GPT-3.5 on the HumanEval and MBPP datasets.
How does Code Llama work? With its robust coding capabilities, Code Llama can generate code based on code and natural language prompts. Not only does it assist in code completion, but it also aids in code debugging. By leveraging 500B of code tokens and code-related data, all three parameter versions of Code Llama undergo training. For added versatility, the 7B and 13B base and instruction models undergo Fill-in-the-Middle (FIM) training, enabling tasks such as out-of-the-box code completion.
Meta recommends utilizing the Code Llama – Instruct variant for code generation due to its fine-tuning for better comprehension of input prompts. However, they caution against using Code Llama or Code Llama – Python for general natural language tasks, as these models are tailored specifically for code-related tasks. Code Llama models require adherence to licensing and usage policies.
Code Llama’s performance surpasses not only LLM, an open-source model specific to code-related tasks, but also Meta’s own Llama 2. Code Llama 34B achieves impressive scores of 53.7% on the HumanEval benchmark and 56.2% on MBPP, outperforming other state-of-the-art open-source solutions and rivaling ChatGPT.
With the introduction of Code Llama, Meta aims to inspire other researchers to create innovative tools for research and commercial products based on the foundation of Llama 2. The emergence of Code Llama fills a gap left by the less code-focused Llama 2, showcasing the expansive possibilities that lie in the integration of AI and coding.
Frequently Asked Questions
Q: Can Code Llama generate code in multiple programming languages?
A: Yes, Code Llama supports various programming languages including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.
Q: What is the maximum context length that Code Llama can handle?
A: Code Llama can generate stable results for context lengths of up to 100,000 tokens.
Q: Is Code Llama suitable for general natural language tasks?
A: Code Llama is specifically designed for code-related tasks and is not recommended for general natural language tasks. However, the Code Llama – Instruct variant can generate useful and safe answers based on its fine-tuning for input prompts.
Q: How does Code Llama compare to other models like GPT-3.5?
A: Code Llama has demonstrated superior performance compared to GPT-3.5 on the HumanEval and MBPP datasets.
Q: Is Code Llama secure?
A: Meta has taken numerous security measures and conducted quantitative evaluations of Code Llama’s risk of generating malicious code. The results indicate that Code Llama’s responses are safer compared to ChatGPT (GPT3.5 Turbo).
Source: AI.Meta