Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models

Introduction

This technical report details comprehensive evaluations of the Reka models (Core, Flash, Edge) on language and vision tasks along with discussions on development, benchmark design, and the training pipeline.

Reka Edge and Flash are dense models with 7B and 21B parameters, respectively. Our evaluation shows that these models are state-of-the-art for their compute class, often surpassing models much larger. Meanwhile, the current version of Reka Core approaches many of the best frontier models (OpenAI, 2023; Google, 2023; Google et al., 2023; Anthropic, 2024). It excels in both automated base model evaluations and blind third-party human evaluations. Figure 1 compares Reka models against proprietary large language models (LLM) APIs. We plot the price against performance, using MMLU score as an approximate indicator of model quality. All Reka models are positioned either on or beyond the Pareto frontier.

Reka Core approaches the performance levels of GPT-4V (OpenAI, 2024) on MMMU (Yue et al., 2024), VQAv2, and third-party multimodal chat evaluation. Meanwhile, Reka Core surpasses all Claude 3 models (Opus, Sonnet, Haiku) (Anthropic, 2024) on multimodal chat human evaluation. On video question answering (Perception-test (Pătrăucean et al., 2023)), both Reka Flash and Core outperform Gemini Ultra (Google, 2023). On language benchmarks, Reka Core achieves $83.2$ MMLU score and competitive GSM8K, HumanEval, and GPQA scores compared to other frontier models. On text-only chat, blind human evaluation shows that Reka Core outperforms GPT-4 ($0613$) and ranks third on our internal ELO leaderboard (right after GPT-4 Turbo and Claude 3 Opus).

Meanwhile, our Edge (7B) model surpasses the current state-of-the-art models of this compute class, outperforming both Gemma 7B (Gemma et al., 2024) and Mistral 7B (Jiang et al., 2023). Additionally, the Flash (21B) model, aside from outperforming GPT-3.5 Turbo, also outperforms much larger state-of-the-art models such as Grok-1 (xAI, 2023), Mistral Medium (Touvron et al., 2023) and Gemini Pro 1.0 (Google, 2023). On multimodal evaluations, Flash outperforms both Claude 3 Opus and Sonnet (Anthropic, 2024) on multimodal chat and matches the Sonnet model on MMMU (Yue et al., 2024). All in all, the Edge & Flash models are extremely powerful models on a compute-class basis.

In addition to comprehensive evaluations and benchmark evaluations on both language and vision (video + image) tasks, this report also shares some interesting technical details and behind-the-scenes of training large multimodal models as a startup. Areas discussed include infrastructure, data pipeline, compute, annotation pipelines, and more. Finally, artifacts of our models (playground/chat, developer platform) can be found in the following resource table (Table 1).

Model

This section briefly describes the technical details behind these models.

The training data comprises a mixture of publicly available and proprietary/licensed datasets with a dataset knowledge cutoff of November 2023. The dataset ingested by our model comprises of text, images, videos, and audio clips. Reka Flash and Reka Edge were trained on approximately $5$ trillion and $4.5$ trillion extensively deduplicated and filtered language tokens, respectively. While the classification of corpora is not strictly defined to one class or category, approximately $25\%$ of our pretraining data is code related, and ${30}\%$ are STEM related. Approximately $25\%$ of the data is web crawl. About $10\%$ of our data has some relation to math. Overall mixture rates generally follow a principle of prioritizing unique tokens but are hand-adjusted using signal from a limited number of small scale ablations.

Multilingual Data: Approximately $15\%$ of our pretraining data is explicitly (and deliberately) multilingual, comprising $32$ diverse languages tier-weighted (roughly by frequency in the wild). Beyond these explicitly up-weighted languages, we also train on the entire multilingual Wikipedia comprising of 110 languages so we expect a baseline performance for most languages. It is worth noting that these tiers reflect pretraining capability and not necessarily downstream post-training induced capabilities of the final model. To be concrete, these are meaningful to estimate the potential of a particular language, given suitable supervised fine tuning data. Languages included during pretraining are shown below.

Multimodal Data: The multimodal training data comprises large collections of images, videos, documents, and webpages. The chosen data mixture is carefully optimized for quality, diversity, and scale.

2 Architecture & Modeling

This section introduces training details, model architecture, and context length details.

Our overall architecture (Figure 2) is a modular encoder-decoder architecture supporting text, image, video, and audio inputs. For now, our model only supports text outputs. The backbone Transformer model is based on the ’Noam’ architecture, i.e., it uses SwiGLU (Shazeer, 2020), Grouped Query Attention (Ainslie et al., 2023; Shazeer, 2019), Rotary positional embeddings (Su et al., 2021) and RMSNorm (Zhang and Sennrich, 2019). Architecturally, this is similar to the PaLM architecture (Chowdhery et al., 2022) but without parallel layers. Reka Flash and Edge uses a sentencepiece vocab of 100K based on tiktoken (e.g., GPT-4 tokenizer). We add sentinel tokens for masking spans, i.e., and other special use cases such as tool-use that are beyond the scope of this technical report. Pretraining uses a curriculum that goes through multiple stages with different mixture distributions, context lengths, and objectives. The current version of this model is a dense model. Models are trained with bfloat16.

Our standard models have a context length of 8K for our regular models. Reka Flash and Reka Core have $128$K for long context models for retrieval and long document tasks. All our models pass needle-in-the-haystack (passkey retrieval) for the context they support. Based on these tests, our 128K models seem to extrapolate to $256$K context length (but not beyond). For long context training, in addition to instruction tuning data we collect, we synthetically create supervised fine tuning data using our own suite of models by conditioning on long documents found in pretraining corpus using a technique we call reverse instruction tuning from long documents.

3 Compute & Infrastructure

Our family of Reka models was trained predominantly on Nvidia H100s using Pytorch (Paszke et al., 2019). Our setup comprises of clusters from a mixture of vendors with our peak compute being approximately $2.5$K H$100$s and $2.5$K A$100$s. Our peak number of clusters is $6$. About more than $90\%$ of our compute came online in mid-December 2023. Reka Flash and Edge were trained on several hundreds of H100s across a period of several weeks. Our pretraining process was relatively smooth with very few loss spikes despite very aggressive learning ratesModels trained at the edge of stability turn out stronger. See https://x.com/m__dehghani/status/1686056450081337344. even for much larger models. Figure 3 shows the training loss for Reka Core. To improve the I/O of our clusters, especially for scalable training with multimodal inputs, we used the Ceph filesystem for distributed and scalable data storage across nodes which improved I/O substantially but came with maintenance overheads.

Generally, we find great unreliability when it comes to GPU nodes which often fail due to hardware errors or connection issues. Moreover, reliability among providers is generally of high variance. For more details, refer to Tay (2024). To expand upon Tay (2024), we report the average number of node failures across four anonymized providers, as shown in Table 4. Since the likelihood of node failures is influenced by the number of nodes concurrently used for training, we report estimated failure rates for different configurations.

Aside from variances across clusters and providers, providers could also have high variance across time periods. For example, many compute providers have clusters that behave very differently in the first few weeks of handover or whenever the cluster undergoes a big change. Hence, we also compare the node failure rates during both the early phase and stable phase. More often than not, aside from early phase of handing over a cluster, provisioning new nodes can also introduce a new chaotic era that can last a few days or weeks. In general, we determined that a key factor influencing the difference between the early and stabilized phases is whether the cluster was actively used for distributed training by previous customers.

We built a custom inference stack for text and multi-modality running on a combination of A10s and A100s. We use Kubernetes as the underlying orchestration engine and manage several large clusters across different regions.

4 Post-Training

This section describes the post-training process which involves aligning, instruction tuning the model.

After pretraining, our models are then instruction tuned (Wei et al., 2021; Ouyang et al., 2022; Chung et al., 2024) for multiple epochs using strong regularization. As for SFT data, we train on a mixture of datasets that include our proprietary and publicly available data. After SFT, models are then aligned with RLHF, specifically PPO (Schulman et al., 2017), using the same family of Reka models as the reward model. Our models go through a couple of rounds of RLHF in total. Moreover, our post-training process considers tool-use, function calling and web search, which is out of scope for this technical report.

We collect data using external data collection companies and provide them with a user interface for annotating both text-only and multimodal data. We create an annotation UI for both collecting data and/or sending examples to human raters for blind human evaluation. This software also supports annotating for individual pointwise quality and also side-by-side (pairwise) evaluations. Our annotation software supports images, videos, and text-only prompts and responses. It also supports the annotation of multi-turn dialogues.

Evaluation

This section discusses the results of extensive evaluations of Reka models.

We conduct a series of language-only and multimodal (image, video input) evaluations.

We compare our models on four language model evaluations: 1) MMLU (general language understanding and question answering) (Hendrycks et al., 2021), 2) GSM8K (reasoning and arithmetic) (Cobbe et al., 2021), HumanEval (code generation) (Chen et al., 2021) and GPQA (graduate-level question answering) (Rein et al., 2023). All numbers from baselines are reported numbers in other works. MMLU is evaluated with 5-shot direct prompting for all models. For GSM8K, most models use 8-shot chain-of-thought (Wei et al., 2022) and majority voting (maj@8). For HumanEval, this is evaluated in 0-shot setup. All results from other models are reported from other works.

We compare our models using visual question answering datasets, i.e., MMMU (Yue et al., 2024), VQAv2 (Goyal et al., 2017), and Perception-Test (Pătrăucean et al., 2023) for video question answering. For Reka models, all results are 0-shot.

Table 5 reports comparisons of Reka Core against other frontier-class models. Overall, Reka Core performs competitively with other frontier-class models. On most metrics (with the exception of MMLU), it is comparable to GPT-4At least an older version, with the results mostly reported from the recent Claude 3 release. HumanEval looks too low for the Claude 3 release so we referenced the HumanEval leaderboard for this number.. In terms of overall performance and with respect to the Claude 3 series, it falls somewhere in between Opus and Sonnet. When compared to Gemini models, Reka Core has mixed outcomes, i.e., winning some and losing some. Reka Core outperforms Gemini Pro 1.5 on several benchmarks (MMLU, GSM8K, HumanEval) but is outperformed on GPQA and MMMU. Notably, Reka Core and Flash outperform Gemini Ultra (and Pro 1.5) on video question answering. Reka Core is still improving so we expect better results in the near future.

2 Chat Model Evaluation

We conduct a blind evaluation with human raters from a third party data provider company. We consider two setups: 1) multimodal chat, where the user asks a question about an image, and 2) text-only chat. We next detail our evaluation protocol and present results for each setting.

For each annotation instance, human raters are given a prompt along with a maximum of 4 generations from different models, and asked to rate the answers according to the guidelines provided. Given that the number of models in our evaluation is higher than 4, we collect multiple such annotations for each prompt, each with a different subset of models. The pairing of models is decided randomly for each prompt, with all combinations being equally likely. We compute ELO scores following Askell et al. (2021), where we only consider pairwise comparisons where annotators express a preference stronger than the weakest available.

We design our evaluation dataset to cover a diverse set of prompts. The following table details the composition of our text-only evaluation set, which comprises 1K+ prompts:

Similarly, the following table reports the categories covered by our multimodal evaluation set:

2.2 Multimodal Chat Evaluation

We next report the results of our multimodal chat evaluation in comparsion with GPT4-V, Claude 3, Gemini Pro, IDEFICS 80B, Adept Fuyu 8B, and the strongest Llava 1.6B model:

We find that Reka Core outperforms all models except GPT4-V by a substantial margin. Reka Flash ranks next, performing marginally better than Claude 3 Opus. Reka Edge outperforms IDEFICS 80B and Adept Fuyu 8B by a large margin, approaching the performance of Gemini Pro and the largest Llava 1.6 model.

2.3 Text-only Chat Evaluation

We compare our models against different versions of GPT, Claude 3, Llama 2 Chat, and Gemini Pro (API version), and report our results next:

We find that Reka Core ranks competitively on our ELO leaderboard, outperforming Claude 3 Sonnet and GPT-4, and it is only surpassed by GPT-4 Turbo and Claude 3 Opus. Reka Flash obtains strong results for its size, beating GPT-3.5 Turbo, Gemini Pro and the much larger Llama 2 Chat 70B.

2.4 Model development and automatic evaluation using Reka Core

We leverage the frontier-class capabilities of Reka Core for model selection and development and show an example of how we use it for multimodal chat. We ask Reka Core to simulate human judgement by rating a response with respect to a prompt and a reference answer. In short, $f(\text{prompt},\text{model\_output},\text{reference\_answer})\in$. We find that Reka Core rankings across models correlate to human judgement despite the gap between pointwise and pairwise (arena style) evaluations. Our general workflow is that we perform lightweight and simple pointwise evaluations for continuous sanity checks before sending our models for third party blind human evaluations.

Figure 4 reports the Reka Core scores we obtain right before producing Table 8. Despite Reka Core evaluations being pointwise, we find that it is able to accurately approximate the final rankings. Here, the only key difference is that Reka Flash and Claude Opus have flipped rankings. In practice, these models may be very similar in performance that it could go either way. In Table, 8, we also note that Reka Flash and Claude Opus have very similar win rates and ELO scores, which is well reflected by their Reka Core scores being very close as well. Overall, we find that Reka Core is quite a good approximator of final human evaluation outcomes.

3 Cross-lingual Evaluations

We conduct experiments on a suite of general multilingual benchmarks such as multilingual commonsense (XStoryCloze (Lin et al., 2022)), causal reasoning (XCOPA (Ponti et al., 2020)), question answering (Belebele (Bandarkar et al., 2023), XQuAD (Artetxe et al., 2019), TydiQA (Clark et al., 2020)). For all datasets, we report the mean across all languages. We compare our models with Llama 2 70B (Touvron et al., 2023), GPT-3.5 and GPT-4. All evaluations are zero-shot generative except XStoryCloze which uses log-likehood evaluation.

Table 11 reports our evalsWe do not run evals for GPT models on XStoryCloze because we use logprobs. As for Belebele, we hit our credit threshold just evaluating on this large evaluation dataset so we stopped. on multilingual benchmarks. Generally we find that Reka Core outperforms all baselines reliably on most tasks (except GPT-4 where it is mixed). Specifically, Reka Core outperforms GPT-4 on XCOPA, XQuAD, TydiQA but is outperformed on XWinograd and TydiQA (w/o context). Meanwhile, Core outperforms Flash on all benchmarks. Both Flash and Core outperforms Llama-2 70B and GPT-3.5. Finally, Figure 5 shows the language breakdowns of Core vs GPT-4.

4 Long Context Question Answering

We conduct a series of evaluations on long context question answering. We use internal benchmarks in two domains: (1) movie plots and (2) ToS (terms-of-service) contract with contexts in the ballpark of $100$K tokens. Both datasets are question answering tasks where the task is to answer questions given a long document. We compare with Claude 3 (Haiku and Sonnet).

Table 12 reports results on long context question answering using internal evaluation datasets. Overall we show that Flash and Core are both competitive to the latest Claude 3 models.

5 Medical Reasoning

We compare our Reka models against state-of-the-art domain-specific medical models such as Meditron (Chen et al., 2023) and Med-PaLM-2 (Singhal et al., 2023). We also compare with GPT-4 reported from (Singhal et al., 2023). We compare on three benchmarks: MedMCQA, PubMedQA and MMLU (Medical). MMLU medical is a macro-average over clinical knowledge, medical genetics, anatomy, professional medicine, college biology and college medicine.

Table 13 reports results on medical tasks. Meditron and Med-PaLM-2 are specialized models for medicine. Our results show that Reka Core is competitive with some of the best frontier models and specialized models in medicine. Firstly, Reka Flash and Core outperforms the Meditron series. Secondly, Reka Core outperforms both Med-PaLM-2 and GPT-4 on MedMCQA. However, it is outperformed on PubMedQA. Finally, on MMLU (Medical), Reka Core outperforms Med-PaLM-2 and is slightly behind GPT-4. Overall, on average, Reka Core outperforms Med-PaLM-2 and is approximately similar to GPT-4 on medical tasks.

6 Detailed comparisons of Edge and Flash

We report detailed results of Reka Edge and Flash against other models of similar compute class. Notably, both Edge and Flash have been improved quite substantially since the initial release in Feb. Hence, numbers have been upgraded since their first appearances.

We report results of Reka Edge against other 7B models such as Llama 2 (Touvron et al., 2023), Mistral (Jiang et al., 2023) and Gemma (Gemma et al., 2024).

Table 14 reports results of Reka Edge against other 7B models (Gemma, Mistral, Llama). We observe that Reka Edge has an edge against all other models (no pun intended). It outperforms Mistral 7B and Llama 7B on all 8 benchmarks. As for Gemma, it outperforms Gemma for all benchmarks except MATH. Overall, Reka Edge is a super strong model at 7B scale.

6.2 Reka Flash results

Given that there are not many good models around the same compute class as Reka Flash, we compare Reka Flash with models that are much larger. Specifically, Llama 2 70B (Touvron et al., 2023), Gemini Pro 1.0 (Google, 2023), Mistral Medium (Touvron et al., 2023) and Grok-1 (xAI, 2023).

Table 15 reports results of Flash (21B) against other models of larger compute class. All competitors are approximately around 70B parameters with the exception of Grok-1 which is a sparse model with 314 billion parameters. We see that Flash outperforms (or is competitive to) all competitors on most benchmarks despite being much smaller.

Conclusion

We introduce a new series of powerful multimodal models, namely Reka Core, Flash, Edge. Reka Flash and Edge sets a new state-of-the-art on a compute-class basis, often delivering massive outsized value for their scale. Our Core model approaches frontier-class models on both human evaluation and automatic benchmarks. Reka Core is still improving so we expect to see even more improvements in the medium term. The field of large language models (Radford et al., 2018; Brown et al., 2020; Devlin et al., 2018; Raffel et al., 2019; Chowdhery et al., 2022; Hoffmann et al., 2022) is still nascent but moving very quickly. With that comes the trade-off of significant amount of noise in the landscape. We hope this technical report shows the rigor of what it takes to build frontier-class models from scratch given limited resources.

Appendix

In Table 16, we report our category-level scores in MMMU (Yue et al., 2024) for Reka Core.

2 Historic versioning, changelog and timeline of Reka Chat models

We include the version history of Reka models to easily refer to them across this tech report.

References