Google released a large, open-source language model based on the technology used to build Gemini that is powerful yet lightweight, optimized for use in resource-constrained environments such as a laptop or cloud infrastructure.
Gemma can be used to build a chatbot, a content generation tool, and pretty much anything else that a language model can do. This is the tool SEOs have been waiting for.
It is released in two versions, one with two billion parameters (2B) and another with seven billion parameters (7B). The number of parameters indicates the complexity and potential capacity of the model. Models with more parameters can achieve better language understanding and generate more sophisticated responses, but they also require more resources to train and run.
The goal of Gemma’s launch is to democratize access to state-of-the-art artificial intelligence that is trained to be safe and responsible out of the box, with a set of tools to further optimize it for safety
Gem of DeepMind
The model is developed to be lightweight and efficient, making it ideal for getting into the hands of more end users.
Google’s official announcement made the following key points:
“We’re releasing model weights in two sizes: Gemma 2B and Gemma 7B. Each size is released with pre-trained and instruction-tuned variants. A new responsible generative AI toolkit provides essential guidance and tools for building applications Safer AI with Gemma We provide toolchains for supervised inference and tuning (SFT) in all major frameworks: JAX, PyTorch and TensorFlow using native Keras 3.0 Ready-to-use Colab and Kaggle notebooks, along with integration with popular tools like Hugging Face, MaxText, NVIDIA NeMo and TensorRT-LLM make it easy to get started with Gemma Pre-trained and tuned Gemma models with instructions can run on your laptop, workstation or Google Cloud for easy deployment on Vertex AI and Google Kubernetes Engine (GKE) Optimizing multiple AI hardware platforms ensures industry-leading performance, including NVIDIA GPUs and Google Cloud TPUs. The terms of use allow responsible commercial use and distribution for all organizations, regardless of size.”
Gemma’s analysis
According to an analysis by Awni Hannun, a machine learning research scientist at Apple, Gemma is optimized to be highly efficient so that it is suitable for use in low-resource environments.
Hannun observed that Gemma has a vocabulary of 250,000 (250k) tokens compared to 32k for comparable models. The importance of this is that Gemma can recognize and process a wider variety of words, enabling her to manage tasks with complex language. Their analysis suggests that this extensive vocabulary improves the model’s versatility across different types of content. He also thinks he can help with math, code and other modalities.
It was also noted that the “embedded weights” are massive (750 million). Embedding weights are a reference to parameters that help map words to representations of their meanings and relationships.
An important feature he noted is that embedding weights, which encode detailed information about word meanings and relationships, are used not only to process the input part, but also to generate the output of the model. This sharing improves the model’s efficiency by allowing it to make better use of its understanding of the language when producing text.
For end users, this means more accurate, relevant and context-appropriate responses (content) to the model, improving their use in content generation, as well as for chatbots and translations.
he he tweeted:
“Vocabulary is massive compared to other open source models: 250K vs 32k for Mistral 7B
Maybe it helps a lot with math / code / other mods with a heavy tail of symbols.
Also the insertion weights are large (~750M parameters), so they are shared with the output header.”
In a follow-up tweet he also noted an optimization in training that results in potentially more accurate and refined model responses, as it allows the model to learn and adapt more effectively during the training phase.
he he tweeted:
“The RMS standard weight has a unit offset.
Instead of “x * weight” they do “x * (1 + weight)”.
I guess this is a training optimization. Normally the weight is initialized to 1, but it is likely that they will be initialized close to 0. Similar to all other parameters.”
He followed that there are more optimizations in data and training, but that these two factors are the ones that stand out in particular.
Designed to be safe and responsible
An important key feature is that it is designed from the ground up to be secure, making it ideal for deploying for use. The training data was filtered to remove personal and sensitive information. Google also used reinforcement learning from human feedback (RLHF) to train the responsible behavior model.
It was further refined with new manual equipment, automated testing, and checked for capabilities for unwanted and dangerous activities.
Google also released a set of tools to help end users further improve security:
“We are also launching a new one Responsible Generative AI Toolkit together with Gemma to help developers and researchers prioritize building safe and responsible AI applications. The toolkit includes:
Security classification: We provide a new methodology for building robust security classifiers with minimal examples. Debugging: A model debugging tool helps you investigate the Gem’s behavior and troubleshoot potential problems. Guidance: You can access best practices for modelers based on Google’s experience in developing and implementing large language models.”
Read Google’s official announcement:
Gemma: Presentation of new state-of-the-art open models
Featured image by Shutterstock/Photo For Everything
[ad_2]
Source link
