This week's Java roundup for October 16th, 2023, features news from OpenJDK, JDK 22, BellSoft, Oracle VS Code extension, WildFly 30, Payara Platform, MicroProfile 6.1, EclipseCon and releases for GraalVM Native Build Tools,

Java News Roundup: Unnamed Variables and Patterns, WildFly 30, MicroProfile 6.1, Payara Platform

Hardware made in Christchurch will be used in the first deep-space mining mission. And it’s one that will help shape things back on Earth. Palladium, used in the manufacture of emission-reducing catalytic converters for cars,

Deep-space mining: Dawn Aerospace delivers propulsion system for AstroForge’s first asteroid mission

Alexis Ohanian is a billionaire from the United States, well-known as the co-founder of Reddit, one of the internet's most popular social media and discussion platforms. Apart from Reddit, Alexis co-founded a venture capital firm called Initialized Capital in 2011.

Understanding Alexis Ohanian's net worth: What is his valuation?

Andrew Sather, principal at Initialized Capital, on why he invested in AstroForge, a US asteroid-mining startup that will use propulsion systems supplied by Christchurch's Dawn Aerospace. Video / Astr

AstroForge's plan to mine near-Earth asteroids

Over a third of consumers said they were likely to splurge in multiple categories, as household debt sits at record highs.

Nearly 7 in 10 Shoppers Likely to Overspend This Holiday Season, Survey Finds

Gulf Cooperation Council (GCC) and Pakistan initialized a free trade agreement, the GCC said on messaging platform X on Thursday. The agreement was signed by GCC Secretary-General Jasem al-Budaiwi and Pakistani trade minister Gohar Ejaz.

Gulf Cooperation Council, Pakistan Sign Free Trade Agreement

Gulf Cooperation Council (GCC) and Pakistan "initialized" a free trade agreement, the GCC said on messaging platform X on Thursday.The agreement was signed by GCC Secretary-General Jasem al-Budaiwi and Pakistani trade minister Gohar Ejaz.

Gulf Cooperation Council says it and Pakistan 'initialize' free trade agreement

If you are looking for the best nicknames for boys and girls, this post is for you. Millions of people search the internet, but there are some websites that offer some nickname ideas for boys and girls, but in this post, I am going to share 1000+ nicknames ...

366 + Best Nicknames For boys and Girls (New & Latest) 2023

Since its launch, the MetaTrace Quest platform has accumulated over 250,000 registered users, meaning a total of at least 250,000 Trace Wallets have been initialized and created. The platform launched nearly three weeks ago and has experienced exceptional recognition since its debut.

MetaTrace Has Reached 250,000 Users on the Quest Platform

If the drive that fails to initialize is empty or contains no essential data, you should format it. Then, you can create a new partition by converting its format to GPT and assigning the new volume. It is the most recommended method for resolving the issue and has worked for many users. To accomplish that, follow these steps:

How to Fix the “Disk Unknown Not Initialized” Issue on Windows

Initialized

initialized

Ending The Crypto Winter with Garry Tan and Brett Gibson of Initialized Capital

Direct Nash Optimization Beats Bigger Models with Better Data

Authors:
(1) Corby Rosset, Microsoft Research and Correspondence to corbyrosset@microsoft.com;
(2) Ching-An Cheng, Microsoft Research;
(3) Arindam Mitra, Microsoft Research;
(4) Michael Santacroce, Microsoft Research;
(5) Ahmed Awadallah, Microsoft Research and Correspondence to hassanam@microsoft.com;
(6) Tengyang Xie, Microsoft Research and Correspondence to tengyangxie@microsoft.com. Authors: Authors: (1) Corby Rosset, Microsoft Research and Correspondence to corbyrosset@microsoft.com; (2) Ching-An Cheng, Microsoft Research; (3) Arindam Mitra, Microsoft Research; (4) Michael Santacroce, Microsoft Research; (5) Ahmed Awadallah, Microsoft Research and Correspondence to hassanam@microsoft.com; (6) Tengyang Xie, Microsoft Research and Correspondence to tengyangxie@microsoft.com. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries 2 Preliminaries 2.1 RLHF Based on Reward Models 2.1 RLHF Based on Reward Models 2.2 RLHF with General Preferences 2.2 RLHF with General Preferences 3 Direct Nash Optimization and 3.1 Derivation of Algorithm 1 3 Direct Nash Optimization and 3.1 Derivation of Algorithm 1 3.2 Theoretical Analysis 3.2 Theoretical Analysis 4 Practical Algorithm – Iterative Contrastive Self-Improvement 4 Practical Algorithm – Iterative Contrastive Self-Improvement 5 Experiments and 5.1 Experimental Setup 5 Experiments and 5.1 Experimental Setup 5.2 Results and Analysis 5.2 Results and Analysis 6 Related Work 6 Related Work 7 Conclusion and References 7 Conclusion and References Appendix Appendix A Extension to Regularized Preferences A Extension to Regularized Preferences B Detailed Proofs B Detailed Proofs C Additional Experimental Details C Additional Experimental Details Abstract This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of “point-wise” rewards (such as that of the Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remain tethered to the reward maximization framework. Recently, a new wave of research sidesteps the reward maximization presumptions in favor of directly optimizing over “pair-wise” or general preferences. In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. Because DNO is a batched on-policy algorithm using a regression-based objective, its implementation is straightforward and efficient. Moreover, DNO enjoys monotonic improvement across iterations which helps it improve even over a strong teacher (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7% → 33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4. Our ablation studies analyze critical design decisions surrounding the choice of preference pairs, and the use of LLMs-as-preference-annotators. These results underscore the promise of DNO in the LLMs post-training, as well as offer actionable insights for the AI research community. 1 Introduction The field of artificial intelligence is evolving towards advanced models that can understand, reason, follow complex instructions, and create nuanced content, while aligning with human values and preferences. Large Language Models (LLMs) (e.g., Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023; OpenAI et al., 2023) have demonstrated remarkable capabilities in generating human-like text, answering questions, and coding, yet they still face challenges in tasks that require a high degree of reliability, safety, and ethical alignment. To address these challenges, fine-tuning LLMs using Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Ouyang et al., 2022) has demonstrates strong potential for making LLMs more helpful by aligning them with human values. The RLHF framework has been long studied in the context of preference-based reinforcement learning (RL) or RL from human preferences (e.g., Knox and Stone, 2008; Akrour et al., 2012; Griffith et al., 2013; Wirth et al., 2017; Christiano et al., 2017). The conventional methods for RLHF typically assume that the preference is determined by a scalar reward function through some model, such as the frequently used Bradley-Terry (BT) model (Bradley and Terry, 1952).[1] RLHF then optimizes toward the preference in a two-step procedure: reward learning, and policy optimization (through RL) to maximize the learned reward. Under certain conditions, the two-step procedure can be streamlined into a single-step contrastive learning approach (Rafailov et al., 2023), eliminating the need for explicit reward learning. Algorithms of this kind (e.g., Rafailov et al., 2023, DPO) leverage the insight that a policy can be expressed equivalently by an “internal reward function” that the policy is optimal to, so they reduce the RLHF problem to regressing the policy’s internal reward function to that of the preference model. These algorithms are originally offline, and boast enhanced stability and ease of optimization. Nonetheless, two-step RLHF algorithms and their single-step contrastive variants still fundamentally rely on the reward maximization framework, wherein reward-based preferences are governed by, e.g., the BT model. The reward maximization framing poses a major limitation. Reward functions, defined to output a scalar score r(x, y) for a single response y to input x, cannot express general preferences y ≻ y ′ | x between a pair of outputs in all cases, e.g., intransitive or cyclic preferences (Elo, 1978). Hence, LLMs trained under reward maximization cannot always align with human preference. Furthermore, recent works show that even in settings where preferences can be perfectly expressed under the reward-based BT models, optimizing towards rewards yields problematic behaviors; we refer the reader to Bertrand et al. (2023); Azar et al. (2023); Munos et al. (2023) for more details. Lastly, reward functions in practice can quickly become “stale” as the distribution of the policy shifts under training (Ross et al., 2011; Cheng et al., 2023; Azar et al., 2023; Munos et al., 2023) – leaving them vulnerable to “reward hacking” (Amodei et al., 2016) We are motivated to overcome two separate challenges: the limited expressivity of reward-based RLHF, and the lack of clarity on how to scale up optimizing with respect to general preferences. Recent advances in reward-based optimization e.g., DPO, already have efficient and scalable implementations – we seek a similarly efficient solution under the framework of general preferences. We propose a provable and scalable RLHF algorithm – Direct Nash Optimization (DNO) (Algorithm 1) that achieves the best of both worlds, combining the scalability of contrastive objectives with the theoretical soundness of general preference optimization. DNO is designed as a batched on-policy algorithm with a regression-based learning objective; this design choice makes DNO stable and scalable, striking a balance between deployment efficiency and adaptability. Direct Nash Optimization Direct Nash Optimization We summarize at a high level the key ingredients and insights of DNO below. To address the issue that reward functions cannot express general preferences, we leverage recent insights that the notion of reward of ought to be expressed as expected win-rates with regard to a general preference function.[2]


To address the issue found in previous work that optimizing this more general objective with online algorithms is sample-inefficient or unstable, we decompose the learning procedure into a sequence of “batched on-policy” iterations, wherein each step instead optimizes a simple regression objective.


The regression objective (we choose binary cross-entropy) aligns the “internal reward function” of the policy to the expected win-rate compared with itself (as defined in Line 3 of Algorithm 1). By sampling outputs from the current policy to use for training (i.e., “self-play”), this procedure incentivizes self-improving behavior.


Our framework is general enough to admit off-policy samples into training, importantly, those from a more powerful teacher (See choice of µ1 and µ2 in Algorithm 1).


Furthermore, to ensure stability and computational efficiency, we propose a filtering scheme such that the reward regression is only performed on preference pairs with a sufficiently large margin (for theoretical explanation, see Section 4; in practice, see Section 5.2).


DNO repeats this procedure for multiple iterations to let the policy optimize toward the general preference. Since each step involves a regression problem it can be easily implemented at scale. To address the issue that reward functions cannot express general preferences, we leverage recent insights that the notion of reward of ought to be expressed as expected win-rates with regard to a general preference function.[2] To address the issue that reward functions cannot express general preferences, we leverage recent insights that the notion of reward of ought to be expressed as expected win-rates with regard to a general preference function.[2] To address the issue found in previous work that optimizing this more general objective with online algorithms is sample-inefficient or unstable, we decompose the learning procedure into a sequence of “batched on-policy” iterations, wherein each step instead optimizes a simple regression objective. To address the issue found in previous work that optimizing this more general objective with online algorithms is sample-inefficient or unstable, we decompose the learning procedure into a sequence of “batched on-policy” iterations, wherein each step instead optimizes a simple regression objective. The regression objective (we choose binary cross-entropy) aligns the “internal reward function” of the policy to the expected win-rate compared with itself (as defined in Line 3 of Algorithm 1). By sampling outputs from the current policy to use for training (i.e., “self-play”), this procedure incentivizes self-improving behavior. The regression objective (we choose binary cross-entropy) aligns the “internal reward function” of the policy to the expected win-rate compared with itself (as defined in Line 3 of Algorithm 1). By sampling outputs from the current policy to use for training (i.e., “self-play”), this procedure incentivizes self-improving behavior. Our framework is general enough to admit off-policy samples into training, importantly, those from a more powerful teacher (See choice of µ1 and µ2 in Algorithm 1). Our framework is general enough to admit off-policy samples into training, importantly, those from a more powerful teacher (See choice of µ1 and µ2 in Algorithm 1). Furthermore, to ensure stability and computational efficiency, we propose a filtering scheme such that the reward regression is only performed on preference pairs with a sufficiently large margin (for theoretical explanation, see Section 4; in practice, see Section 5.2). Furthermore, to ensure stability and computational efficiency, we propose a filtering scheme such that the reward regression is only performed on preference pairs with a sufficiently large margin (for theoretical explanation, see Section 4; in practice, see Section 5.2). DNO repeats this procedure for multiple iterations to let the policy optimize toward the general preference. Since each step involves a regression problem it can be easily implemented at scale. DNO repeats this procedure for multiple iterations to let the policy optimize toward the general preference. Since each step involves a regression problem it can be easily implemented at scale. Theoretically, we prove DNO converges to the intended Nash equilibrium on average, and that it can improve monotonically across iterations (see Section 3.1). Furthermore, our finite-sample analysis shows that approximation error at any iteration between the learned policy and the target is tightly bounded (Theorem 1). On the practical side, we provide a scalable implementation of DNO (Algorithm 2): an iterative self-improving algorithm with contrastive updates, which approximates Algorithm 1 under several critical design choices. Those choices include: sampling multiple online outputs from the policy being trained, using GPT-4 as the preference oracle, comparing onpolicy samples to GPT-4’s own (teacher) outputs, and training only on pairs with “large margin” (for theoretical explanation, see Section 4; in practice, see Section 5.2). The primary distinction of our work over related works of Nash-MD (Munos et al., 2023) and SPO (Swamy et al., 2024) is that they both exhibit sample efficiency issues (two timescale updates or sample-inefficient RL steps), and both use purely on-policy samples. We resolve the efficiency issue with a sample-efficient objective that works in practice, and DNO is more flexible to incorporate off-policy samples from e.g., a powerful teacher. Most importantly, DNO works in practice – we provide comprehensive empirical evaluations, resulting in state-of-the-art performance: • The resulting 7B parameter Orca-2.5 model, aligned using the practical implementation of DNO (Algorithm 2), achieves the state-of-the-art win-rate of any 7B model, exceeding 33% against GPT-4-Turbo beyond on the AlpacaEval 2.0, even after controlling for length. This is an over 26% absolute gain (7%→33%) compared to the initialized model. It outperforms several recent advanced closed-source models, including Mistral Large and GPT-4-0613, as well as open-source models with far more (10×) parameters, such as Self-Rewarding LM (Yuan et al., 2024) which has 70B parameters. • Our thorough ablation studies in Section 5.2 examine critical design touchpoints surrounding choice of loss function (supervised finetuning or contrastive), training paradigm (with or without on-policy samples), preference annotator quality (large margin or not), and training pair construction (self-play, teacher-vs-student, etc). Our findings highlight that carefully-crafted methods encoded in Algorithm 2 lead to substantial gains. • We show some examples of outputs across iterations which demonstrate qualitative improvements such as better addressing nuanced issues and presumptious questions (Table 5), better organization and clarity while refraining from making misleading statements (Table 6), and higher information density in answers (Table 7). We hope that the results presented herein will provide clarity to the community regarding the use of AI feedback for post-training LLMs. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] We use “reward model” to denote a framework that translates preferences into rewards, e.g., Bradley-Terry, while “reward function” is a (possibly learned) function that outputs reward scalars.

The Art of Arguing With Yourself—And Why It’s Making AI Smarter

Umění hádat se sebou - a proč to dělá AI chytřejší

Kunsten at argumentere med dig selv - og hvorfor det gør AI smartere

El arte de discutir contigo mismo y por qué está haciendo que la IA sea más inteligente

自分と議論する芸術 - そしてなぜAIを賢くするのか

Pamba la Pamba la Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100% ya Pamba la 100%

The Art of Argumenting With Yourself – ແລະວິທີທີ່ມັນເຮັດໃຫ້ AI smarter

Уметноста да се расправате со себеси – и зошто ја прави вештачката интелигенција попаметна

A arte de discutir consigo mesmo – e por que está tornando a IA mais inteligente

Уметност расправљања са собом - и зашто то чини ИИ паметнијим

اپنے آپ کے ساتھ جھگڑا کرنے کا فن - اور کیوں یہ AI کو ذہین بناتا ہے

Authors:
(1) Rohit Saxena, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg;
(2) RFrank Keller, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg. Table of Links Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 4. Scene Saliency Classification Model Using the set of generated silver-standard labels for scene saliency, we train a neural network-based classification model to predict scene saliency. We formulate this task as a sequence labeling task where the model takes a sequence of scenes M = {S1, S2, ..., SN} as input and predicts a sequence of binary labels Y = {y1, y2, ..., yN } denoting whether a scene is salient. The model consists of two components, as shown in Figure 1. The first component is a scene encoder which computes scene representations by concatenating the sentences in the scene and encodes them using a pre-trained language model. Next, to learn contextual scene representation across the whole movie, we further encode the scene embeddings generated by the scene encoder using a transformer (Vaswani et al., 2017) block (L layers stacked), with unmasked self-attention initialized with random weights (Liu and Lapata, 2019). To preserve the sequence of the scenes, we add positional encodings to scene representations obtained from the first component. The final contextualized representation of the scenes is then used to classify whether scenes are salient or not. The model is trained for binary sequence labeling using the binary cross-entropy loss. 4.1 Dataset To train the saliency model, we used the ScriptBase corpus (Gorinski and Lapata, 2015) that contains preprocessed scripts of movies with Wikipedia summaries. We removed the movies used in our gold-standard MENSA dataset from Scriptbase from the training set. This resulted in a training set containing 824 movie scripts, for which we generated silver-standard scene saliency labels using the model of Mirza et al. (2021), as previously discussed. We randomly split our gold-standard scene saliency dataset of 100 movies, using half of it for validation and the other half for testing. 4.2 Baselines Majority Class: We used predicting the majority class as a simple baseline for classification. The dataset is highly imbalanced, with non-salient being the majority class. Supervised Bi-LSTM: For a supervised baseline, we used a bi-directional LSTM (Bi-LSTM) to learn contextual representation for the classification of scene saliency. Again, we computed scene representations by concatenating the sentences and encoding them using a pre-trained language model. Note that the alignment model of Mirza et al. (2021) cannot be used as a baseline for saliency classification: it requires summaries to align to movie scripts at test time. In a summarization scenario, no summaries are available at test time. 4.3 Implementation Details Our Scene Saliency Model and baseline models employed RoBERTa-large as the pre-trained scene encoder. Representation of a scene is computed using the first token’s last hidden state of the model. The movie encoder transformer block has 10 layers with 16 heads and a feedforward hidden size of 2048. As the binary scene labels in the dataset are highly imbalanced, we used weighted binary cross entropy. We employed AdamW with β1 = 0.9, β2 = 0.999 as our optimizer. The learning rate was fixed at 5e-5. For the baseline TextRank model, we performed a grid search for hyperparameters and used λ1 = 0.7, λ2 = 0.3, and K = 15% of move length. For Bi-LSTM, we used hidden dimension of size 512 followed by a fully connected layer. 4.4 Results The results of our saliency classification model and the baselines are summarized in Table 4. We report macro-averaged precision (P), recall (R), and F1 score for each model, as the labels are highly imbalanced given that only a limited number of scenes in each movie are salient. Our model outperforms the baselines and achieves 68.38, 68.13, and 68.01 on precision, recall, and F1. The results show that the majority baseline performance is equivalent to random guessing (macro-average). The unsupervised TextRank model has higher precision, recall, and F1 than the majority baseline, which indicates that it is able to correctly predict some scenes as salient based on the centrality score. Also, the high value of λ1 (see Section 4.3) signifies that the backward-looking context is more important than forward-looking context for computing scene importance. The transformer-based scene saliency model achieves better performance than other baselines, indicating the effectiveness of transformer layers in learning the context across scene representations, which is helpful in classifying scene saliency. We also found that a higher number of layers worked better for the transformer, which indicates that more layers help in capturing complex relationships in the input. See Appendix C for k-fold cross-validation on the test set. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Rohit Saxena, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg; (2) RFrank Keller, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg. Authors: Authors: (1) Rohit Saxena, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg; (2) RFrank Keller, Institute for Language, Cognition and Computation, School of Informatics, University of Edinburg. Table of Links Part 1 Part 1 Part 2 Part 2 Part 3 Part 3 Part 4 Part 4 Part 5 Part 5 Part 6 Part 6 4. Scene Saliency Classification Model Using the set of generated silver-standard labels for scene saliency, we train a neural network-based classification model to predict scene saliency. We formulate this task as a sequence labeling task where the model takes a sequence of scenes M = {S1, S2, ..., SN} as input and predicts a sequence of binary labels Y = {y1, y2, ..., yN } denoting whether a scene is salient. The model consists of two components, as shown in Figure 1. The first component is a scene encoder which computes scene representations by concatenating the sentences in the scene and encodes them using a pre-trained language model. Next, to learn contextual scene representation across the whole movie, we further encode the scene embeddings generated by the scene encoder using a transformer (Vaswani et al., 2017) block (L layers stacked), with unmasked self-attention initialized with random weights (Liu and Lapata, 2019). To preserve the sequence of the scenes, we add positional encodings to scene representations obtained from the first component. The final contextualized representation of the scenes is then used to classify whether scenes are salient or not. The model is trained for binary sequence labeling using the binary cross-entropy loss. 4.1 Dataset To train the saliency model, we used the ScriptBase corpus (Gorinski and Lapata, 2015) that contains preprocessed scripts of movies with Wikipedia summaries. We removed the movies used in our gold-standard MENSA dataset from Scriptbase from the training set. This resulted in a training set containing 824 movie scripts, for which we generated silver-standard scene saliency labels using the model of Mirza et al. (2021), as previously discussed. We randomly split our gold-standard scene saliency dataset of 100 movies, using half of it for validation and the other half for testing. 4.2 Baselines Majority Class: We used predicting the majority class as a simple baseline for classification. The dataset is highly imbalanced, with non-salient being the majority class. Majority Class: Supervised Bi-LSTM: For a supervised baseline, we used a bi-directional LSTM (Bi-LSTM) to learn contextual representation for the classification of scene saliency. Again, we computed scene representations by concatenating the sentences and encoding them using a pre-trained language model. Supervised Bi-LSTM: Note that the alignment model of Mirza et al. (2021) cannot be used as a baseline for saliency classification: it requires summaries to align to movie scripts at test time. In a summarization scenario, no summaries are available at test time. 4.3 Implementation Details Our Scene Saliency Model and baseline models employed RoBERTa-large as the pre-trained scene encoder. Representation of a scene is computed using the first token’s last hidden state of the model. The movie encoder transformer block has 10 layers with 16 heads and a feedforward hidden size of 2048. As the binary scene labels in the dataset are highly imbalanced, we used weighted binary cross entropy. We employed AdamW with β1 = 0.9, β2 = 0.999 as our optimizer. The learning rate was fixed at 5e-5. For the baseline TextRank model, we performed a grid search for hyperparameters and used λ1 = 0.7, λ2 = 0.3, and K = 15% of move length. For Bi-LSTM, we used hidden dimension of size 512 followed by a fully connected layer. 4.4 Results The results of our saliency classification model and the baselines are summarized in Table 4. We report macro-averaged precision (P), recall (R), and F1 score for each model, as the labels are highly imbalanced given that only a limited number of scenes in each movie are salient. Our model outperforms the baselines and achieves 68.38, 68.13, and 68.01 on precision, recall, and F1. The results show that the majority baseline performance is equivalent to random guessing (macro-average). The unsupervised TextRank model has higher precision, recall, and F1 than the majority baseline, which indicates that it is able to correctly predict some scenes as salient based on the centrality score. Also, the high value of λ1 (see Section 4.3) signifies that the backward-looking context is more important than forward-looking context for computing scene importance. The transformer-based scene saliency model achieves better performance than other baselines, indicating the effectiveness of transformer layers in learning the context across scene representations, which is helpful in classifying scene saliency. We also found that a higher number of layers worked better for the transformer, which indicates that more layers help in capturing complex relationships in the input. See Appendix C for k-fold cross-validation on the test set. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Smart Summaries: Teaching AI to Pick Key Scenes from Scripts

Does AI Understand Angular Development? We Put ChatGPT to the Test

Begryp AI Angular Development? Ons stel ChatGPT aan die toets

— А як вы збіраецеся пракарміць дзіця на 208 рублёў? — запытаў суддзя.

Versteht AI Angular Development? Wir setzen ChatGPT auf den Test

¿Comprende la IA el desarrollo angular?Ponemos ChatGPT a la prueba

AI は Angular Development を理解していますか? ChatGPT をテストします

გაითვალისწინეთ, თუ როგორ უნდა გააკეთოთ ChatGPT?

Алдыңғы мақалаЧТО НАДО ДЕЛАТЬ ДЛЯ РАЗВИТИЯ ПРОМЫШЛЕННОСТИ?

Понимает ли ИИ углубленное развитие? мы поставили ChatGPT на тест

Ano ang Pag-iisip ng AI ang Angular Development? Ipinapakita namin ang ChatGPT sa Test

AI có hiểu sự phát triển góc không?Chúng tôi đặt ChatGPT vào thử nghiệm

Authors:
(1) Frank Palma Gomez, from Boston University and the work done by him and Ramon during their internship in Google Research and Google DeepMind respectively;
(2) Ramon Sanabria, The University of Edinburgh and the work done by him and Frank during their internship in Google Research and Google DeepMind respectively;
(3) Yun-hsuan Sung, Google Research;
(4) Daniel Cer, Google Research;
(5) Siddharth Dalmia, Google DeepMind and Equal Advising Contributions;
(6) Gustavo Hernandez Abrego, Google Research and Equal Advising Contributions. Authors: Authors: (1) Frank Palma Gomez, from Boston University and the work done by him and Ramon during their internship in Google Research and Google DeepMind respectively; (2) Ramon Sanabria, The University of Edinburgh and the work done by him and Frank during their internship in Google Research and Google DeepMind respectively; (3) Yun-hsuan Sung, Google Research; (4) Daniel Cer, Google Research; (5) Siddharth Dalmia, Google DeepMind and Equal Advising Contributions; (6) Gustavo Hernandez Abrego, Google Research and Equal Advising Contributions. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Method 2 Method 3 Data and Tasks 3 Data and Tasks 4 Model 4 Model 5 Experiments 5 Experiments 6 Related Work 6 Related Work 7 Conclusion 7 Conclusion 8 Acknowledgements and References 8 Acknowledgements and References A Appendix A Appendix 5 Experiments We train our DE model to perform S2T, where the task is to retrieve the corresponding transcription given a speech sample. We train on the 21 languages from CoVoST-2 and evaluate our model using the S2T portion of FLEURS in 102 languages. 5.1 Speech-to-Text Retrieval Table 1 shows the average R@1 and WER for S2T for 102 languages from FLEURS. We compare against the mSLAM DE model from Conneau et al. (2023), a model trained on 426k hours of S2T data in 51 languages and fine-tuned on FLEURS training data. Our model significantly outperforms the mSLAM DE baseline in R@1 and W ER metrics despite being trained with only 1/10 of the data and having been initialized from a text-only LLM. More importantly, our model was only trained on the 21 languages in CoVoST-2 and never fine-tuned on the FLEURS training data. 5.1.1 Seen-Unseen Breakdown 5.1.1 Seen-Unseen Breakdown In Figure 2 we break down the R@1 scores based on seen and unseen languages during training. We find that our model performs best on the 20 languages that are within the training and evaluation data, but still perform significantly well on the remaining 82 unseen languages. We hypothesize this is due to the vast textual multilingual data our backbone LLM has seen during pre-training. 5.1.2 Language Group Breakdown 5.1.2 Language Group Breakdown Table 2 shows the R@1 language group breakdown for S2T on FLEURS. We find that although we only trained on 21 languages, our model significantly outperforms mSLAM DE in 13 of the 15 language groups. These results are consistent with the experiments in Hassid et al. (2023) which explore the effect of initializing speech language models from pre-trained LLMs. 5.2 Evaluating on Cross-Modal and Cross-Lingual Tasks We evaluate on S2TT to gauge the cross-modal and cross-lingual capabilities of our model. We show that we can further improve S2TT by simply training on a mixture of S2T and translation data without using any S2TT training data. 5.2.1 Zero-Shot S2TT 5.2.1 Zero-Shot S2TT Given the multi-lingual capabilities of our backbone language model, we explore if these capabilities are transferred after training our model contrastively on the S2T task. We hypothesize that our model should showcase cross-lingual and crossmodal capabilities due to the cross-modal training task and the cross-lingual capabilities of the backbone LLM. We evaluate S2TT in a zero-shot setting to assess our model’s performance retrieving English translations given a speech sample in another language. Using the FLEURS S2TT portion, we evaluate S2TT X → En in 4 languages: German, Polish, French, and Dutch. Figure 3 shows BLEU S2TT performance using S2T CoVoST-2 in 21 languages. We call this setup Transcripts in Figure 3. Our results demonstrate that even when only training our model on speech and transcriptions, we can achieve some zero-shot S2TT performance and We find that S2TT BLEU scores are considerably higher for languages present S2T training data. For example, Polish was not in the S2T training therefore its BLEU scores are the lowest. 5.2.2 Improving S2TT with MT Data 5.2.2 Improving S2TT with MT Data To further improve our model’s cross-lingual performance, we add readily available translation data from Schwenk et al. (2019) to improve S2TT. For each batch, we combine 25% translation and 75% S2T data. Figure 3 shows comparison of only training on S2T (Transcripts) and combining S2T and translation data ( Transcriptions + Translations). We find that combining S2T and translation data significantly improves the S2TT BLEU scores in all 4 languages without training on S2TT data. This finding demonstrates that we can improve our models cross-lingual performance with highly accessible translation data without needing scarce and often expensive speech-totext translation training data. 6 Related Work The success of pre-trained LLMs have motivated the application of these models in different modalities. Lakhotia et al. (2021) transformed speech into pseudo-text units to introduce the task of generative spoken language modeling. Borsos et al. (2023) introduced a framework to generate audio with long-term consistency. Consequently, Hassid et al. (2023) showed that SpeechLMs benefit from being initialized from pre-train LLMs while Rubenstein et al. (2023) demonstrated that pre-trained LLMs can be adapted to various tasks that required text and speech understanding. On the other hand, several works aim to build joint speech and text representations. Chung et al. (2021) introduced w2v-bert which combines masked language modeling and contrastive learning to create speech representations. Bapna et al. (2022) jointly pre-trains on speech and text from unsupervised speech and text data. Recently, Duquenne et al. (2023) employed separate speech and text encoders to generate embeddings in over 200 languages. Nevertheless, there is still a lack of understanding of whether joint speech and text representations can be built from a single encoder. We fill this gap by using pre-trained LLMs to jointly train on speech samples and their transcriptions to show that our approach is capable of speech-text matching in 102 languages. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Why Our Tiny Training Set Beat Giants in Cross-Lingual Speech Retrieval

Authors:
(1) Frank Palma Gomez, from Boston University and the work done by him and Ramon during their internship in Google Research and Google DeepMind respectively;
(2) Ramon Sanabria, The University of Edinburgh and the work done by him and Frank during their internship in Google Research and Google DeepMind respectively;
(3) Yun-hsuan Sung, Google Research;
(4) Daniel Cer, Google Research;
(5) Siddharth Dalmia, Google DeepMind and Equal Advising Contributions;
(6) Gustavo Hernandez Abrego, Google Research and Equal Advising Contributions. Authors: Authors: (1) Frank Palma Gomez, from Boston University and the work done by him and Ramon during their internship in Google Research and Google DeepMind respectively; (2) Ramon Sanabria, The University of Edinburgh and the work done by him and Frank during their internship in Google Research and Google DeepMind respectively; (3) Yun-hsuan Sung, Google Research; (4) Daniel Cer, Google Research; (5) Siddharth Dalmia, Google DeepMind and Equal Advising Contributions; (6) Gustavo Hernandez Abrego, Google Research and Equal Advising Contributions. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Method 2 Method 3 Data and Tasks 3 Data and Tasks 4 Model 4 Model 5 Experiments 5 Experiments 6 Related Work 6 Related Work 7 Conclusion 7 Conclusion 8 Acknowledgements and References 8 Acknowledgements and References A Appendix A Appendix Abstract Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn’t require speech data during LLM pre-training and can exploit LLM’s multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLMbased retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data. 1 Introduction LLMs have demonstrated their effectiveness in modelling textual sequences to tackle various downstream tasks (Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2023). This effectiveness has led to the development of powerful LLMs capable of modelling text in a wide range of languages. The abundance of textual data in different languages across the internet has fueled the progress of multi-lingual models (Johnson et al., 2017; Xue et al., 2020; Siddhant et al., 2022). On the other hand, speech technologies are prevalent in smartphones and personal assistants, but their language availability is relatively limited compared to the languages that LLMs support (Baevski et al., 2020; Radford et al., 2023). Various efforts have explored solutions to the speech-text data scarcity problem (Duquenne et al., 2021; Ardila et al., 2019; Wang et al., 2020). Works such as SpeechMatrix (Duquenne et al., 2022) use separate speech and text encoders to mine semantically similar utterances that are neighbors in an embedding space. However, these approaches are limiting because they require speech and text encoders that have aligned representation spaces. We posit that we can retrieve speech and text utterances by aligning both modalities within the embedding space built from a single pre-trained LLM. We take inspiration from previous works that use pre-trained LLMs to perform automatic speech recognition (ASR) and automatic speech translation (AST) (Rubenstein et al., 2023; Wang et al., 2023; Hassid et al., 2023). Our intuition is that we can perform the speech and text alignment leveraging the capabilities of text-only LLMs without requiring two separate models. In this paper, we propose converting LLMs into speech and text DE retrieval systems without requiring speech pre-training and outperform previous methods with significantly less data. By discretizing speech into acoustic units (Hsu et al., 2021), we extend our LLMs embedding layer and treat the acoustic units as ordinary text tokens. Consequently, we transform our LLM into a retrieval system via a contrastive loss allowing us to match speech and text utterances in various languages. Our contributions are the following: We build a speech-to-text symmetric DE from a pre-trained LLM. We show that our retrieval system is effective matching speech and text in 102 languages of FLEURS (Conneau et al., 2023 despite only training on 21 languages.


We show that our model exhibits cross-lingual speech and text matching without training on this type of data. At the same time, we find that cross-lingual speech and text matching is furthered improved by training on readily available machine translation data. We build a speech-to-text symmetric DE from a pre-trained LLM. We show that our retrieval system is effective matching speech and text in 102 languages of FLEURS (Conneau et al., 2023 despite only training on 21 languages. We build a speech-to-text symmetric DE from a pre-trained LLM. We show that our retrieval system is effective matching speech and text in 102 languages of FLEURS (Conneau et al., 2023 despite only training on 21 languages. We show that our model exhibits cross-lingual speech and text matching without training on this type of data. At the same time, we find that cross-lingual speech and text matching is furthered improved by training on readily available machine translation data. We show that our model exhibits cross-lingual speech and text matching without training on this type of data. At the same time, we find that cross-lingual speech and text matching is furthered improved by training on readily available machine translation data. 2 Method We train a transformer-based DE model that encodes speech and text given a dataset D = {(xi , yi)}, where xi is a speech utterance and yi is its transcription. We denote the speech and text embeddings as xi = E(xi) and yi = E(yi), respectively, where E is a transformer-based DE that encodes speech and text. 2.1 Generating Audio Tokens We convert raw speech into discrete tokens using the process in Lakhotia et al. (2021); Borsos et al. (2023). The process converts a speech query xi into an embedding using a pre-trained speech encoder. The output embedding is then discretized into a set of tokens using k-means clustering. We refer to the resulting tokens as audio tokens. We use the 2B variant of the Universal Speech Model (USM) encoder (Zhang et al., 2023) as the speech encoder and take the middle layer as the embedding for xi . Additionally, we generate audio tokens at 25Hz using k-means clustering, resulting in a set of 1024 possible audio tokens. We will refer to this as our audio token vocabulary. 2.2 Supporting Text and Audio Tokens To support text and audio tokens in our LLM, we follow the formulation of Rubenstein et al. (2023). We extend the embedding layer of a transformer decoder by a tokens, where a represents the size of our audio token vocabulary. This modification leads to an embedding layer with size (t + a) × m, where t is the number of tokens in the text vocabulary and m is the dimensions of the embedding vectors. In our implementation, the first t tokens represent text and the remaining a tokens are reserved for audio. We initialize the embeddings layer from scratch when training our model. 3 Data and Tasks Appendix A.3 details our training and evaluation datasets along with the number of languages in each dataset, the split we used, and the size of each dataset. We focus on the following retrieval tasks: Speech-to-Text Retrieval (S2T) involves retrieving the corresponding transcription from a database given a speech sample. In S2T, we train on CoVoST-2 (Wang et al., 2021) speech utterances and their transcriptions. CoVoST-2 is a large multilingual speech corpus derived from Wikipedia expanding over 21 languages and provides translation to and from English. We use FLEURS (Conneau et al., 2023) to evaluate S2T performance on 102 languages. FLEURS is an n-way parallel dataset containing speech utterances from FLoRES-101 (Goyal et al., 2021) human translations. To evaluate S2T, we report recall at 1 (R@1) rates for retrieving the correct transcription for every speech sample and word error rate (WER). Speech-to-Text Retrieval (S2T) Speech-to-Text Translation Retrieval (S2TT) attempts to retrieve the corresponding text translation of a speech sample. We use S2TT to measure the cross-lingual capabilities of our multi-modal DE retrieval system. We evaluate this capability zero-shot on X → En S2TT data of FLUERS and explore if we can further improve this capability by training on readily-available machine translation data from WikiMatrix (Schwenk et al., 2019). We pick French, German, Dutch, and Polish to English that are common across WikiMatrix and FLEURS and further discuss the amount of machine translation data used in Appendix A.3. For S2TT, we report 4-gram corpusBLEU (Post, 2018). Speech-to-Text Translation Retrieval (S2TT) 4 Model Figure 1 shows an illustration of our model. We initialize our dual encoder from PaLM 2 XXS (Google et al., 2023) and append a linear projection layer after pooling the outputs along the sequence length dimension. The embedding and linear projection layers are initialized randomly. After initializing our model from PaLM 2, we use a contrastive loss (Hadsell et al., 2006). Appendix A.1 includes more details on our training setup. We will refer to our proposed model as PaLM 2 DE. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work


Task, Datasets, Baseline


RQ 1: Leveraging the Neighbourhood at Inference
4.1. Methods
4.2. Experiments


RQ 2: Leveraging the Neighbourhood at Training
5.1. Methods
5.2. Experiments


RQ 3: Cross-Domain Generalizability


Conclusion


Limitations


Ethics Statement


Bibliographical References Related Work Related Work Related Work Task, Datasets, Baseline Task, Datasets, Baseline Task, Datasets, Baseline RQ 1: Leveraging the Neighbourhood at Inference
4.1. Methods
4.2. Experiments RQ 1: Leveraging the Neighbourhood at Inference 4.1. Methods 4.1. Methods 4.2. Experiments 4.2. Experiments RQ 2: Leveraging the Neighbourhood at Training
5.1. Methods
5.2. Experiments RQ 2: Leveraging the Neighbourhood at Training RQ 2: Leveraging the Neighbourhood at Training 5.1. Methods 5.1. Methods 5.2. Experiments 5.2. Experiments RQ 3: Cross-Domain Generalizability RQ 3: Cross-Domain Generalizability RQ 3: Cross-Domain Generalizability Conclusion Conclusion Conclusion Limitations Limitations Limitations Ethics Statement Ethics Statement Ethics Statement Bibliographical References Bibliographical References Bibliographical References 5. RQ 2: Leveraging the Neighbourhood at Training We leverage the knowledge from neighbour instances directly at the training process to improve the performance. We explore three methods: contrastive learning, single prototypical learning, and multi prototypical learning. These techniques draw inspiration from the same principles as their inference-time counterparts but serve as auxiliary loss constraints during training. Their primary aim is to improve the discriminative capability of embeddings by highlighting differences between instances with distinct rhetorical roles and similarities among instances sharing the same label. While the task-specific classification loss focuses on mapping contextualized representations to label outputs with supervision on individual instances, the methods in this section directly operate on embeddings in latent space. They exploit the interplay among instances to establish effective discriminative decision boundaries, serving as a form of regularization. 5.1. Methods 5.1.1. Contrastive Learning 5.1.1. Contrastive Learning Contrastive learning aims to bring an anchor point closer to related samples while pushing it away from unrelated samples in the embedding space. In a supervised setting, samples with the same/different labels are considered related/unrelated with respect to an anchor (Khosla et al., 2020). The loss is calculated as follows: where δ(ci , cj ) denotes 1 if ci and cj have same rhetorical label, else 0, N denotes batch size. Lengthy legal documents limits the number of documents that can be accommodated in a single batch and this raises concerns about having enough positive samples for the minority class instances within a batch for effective contrasting. To overcome this limitation, we utilize a memory bank (Wu et al., 2018), where we progressively reuse encoded representations from previous batches to compute the contrastive loss. In practice, we maintain a fixed-size representation queue for each rhetorical role. As new representations corresponding to specific labels are generated, they are enqueued into the respective queue with their gradients detached. If the queue size for a label exceeds the maximum limit, the oldest element is dequeued. When computing the contrastive loss, we use the same equation 7. However, in addition to the current batch instances, we employ all the representations stored in the memory bank for contrasting purposes, using them as both positives and negatives, based on the anchor point’s label. memory bank To incorporate the concept of context from surrounding sentences into contrastive learning, we introduce a novel discourse-aware contrastive loss. This is based on the idea that sentences in close proximity within a document, sharing the same label, should exhibit a stronger proximity in the embedding space compared to sentences with the same label but positioned farther apart in the document. To implement this concept, we introduce a penalty inversely proportional to the absolute difference in their positions. In particular, we impose a higher penalty on positive sentence pairs that are closer in the document, encouraging them to be closer in the embedding space than pairs originating from greater distances within the document. The discouse-aware loss is as follows: discourse-aware contrastive loss where β represents a penalty that considers positional information. When ci and cj come from different documents, such as cross-document positives/negatives from the memory bank or across the batch, we apply the lowest possible penalty, considering ci as the farthest sentence relative to in-document positives. We incorporate this additional contrastive loss alongside the classification loss during training. 5.1.2. Single Prototypical Learning 5.1.2. Single Prototypical Learning These both views shape the embedding space by aligning prototypes with their corresponding samples, forming distinct clusters of different labels, each centered around a specific prototype vector. 5.1.3. Multi Prototypical Learning 5.1.3. Multi Prototypical Learning Instead of using a single prototype for each label, this approach employs multiple prototypes for each label to capture the diverse variations within the sentences of the same label. To implement this, a set of M prototypes per label is randomly initialized and a diversity loss (Zhang et al., 2022) is integrated to penalize prototypes of the same label if they are too similar to each other. This ensures that prototypes of the same label are distributed across the embedding space, capturing the multifaceted nuances under each label. The Sample Centric View is also modified to ensure that each sample is in close proximity to at least one prototype among all the prototypes of the same class. Authors:
(1) Santosh T.Y.S.S, School of Computation, Information, and Technology; Technical University of Munich, Germany (santosh.tokala@tum.de);
(2) Hassan Sarwat, School of Computation, Information, and Technology; Technical University of Munich, Germany (hassan.sarwat@tum.de);
(3) Ahmed Abdou, School of Computation, Information, and Technology; Technical University of Munich, Germany (ahmed.abdou@tum.de);
(4) Matthias Grabmair, School of Computation, Information, and Technology; Technical University of Munich, Germany (matthias.grabmair@tum.de). Authors: Authors: (1) Santosh T.Y.S.S, School of Computation, Information, and Technology; Technical University of Munich, Germany (santosh.tokala@tum.de); (2) Hassan Sarwat, School of Computation, Information, and Technology; Technical University of Munich, Germany (hassan.sarwat@tum.de); (3) Ahmed Abdou, School of Computation, Information, and Technology; Technical University of Munich, Germany (ahmed.abdou@tum.de); (4) Matthias Grabmair, School of Computation, Information, and Technology; Technical University of Munich, Germany (matthias.grabmair@tum.de). This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

How AI Can Better Categorize Legal Documents by Learning from Similar Texts

This is the second article in series C++ Metaprogramming series, you can find the first article here C++ Metaprogramming: Variadic Templates & Fold Expressions. This article will unveil the practical usage of templates and constexpr for compile-time code execution. C++ Metaprogramming: Variadic Templates & Fold Expressions constexpr Introduction Compile-time calculations in C++ allow some operations to be performed during compilation instead of at runtime. This provides a number of advantages: validating data and logic before running the program, reducing overall execution time, and enhancing code safety and reliability. At the same time, of course, there is an additional cost: compilation time could increase significantly, and the code itself may become more difficult to read. In this article we will explore: Compilation mechanisms for computations available in C++ (templates and constexpr)
Examples of non-trivial use of template metaprogramming
Modern capabilities of constexpr (including objects, loops, and algorithms)
Useful features and techniques, which make developer’s life easier.
Potential pitfalls and tips on how to avoid them. Compilation mechanisms for computations available in C++ (templates and constexpr) constexpr Examples of non-trivial use of template metaprogramming Modern capabilities of constexpr (including objects, loops, and algorithms) constexpr Useful features and techniques, which make developer’s life easier. Potential pitfalls and tips on how to avoid them. Templates Metaprogramming - the foundation of compile-time calculations Historically in C++ metaprogramming emerged thanks to templates. They originally were intended for generating generic functions and classes, but over time it was discovered, that recursive templates could be used to preform quite complex computations. Classical examples Example 1: Factorial using template recursion #include <cstdio>

template <int N>
struct Factorial {
    static constexpr int value = N * Factorial<N - 1>::value;
};

template <>
struct Factorial<0> {
    static constexpr int value = 1;
};

int main() {
  printf("%d\n", Factorial<5>::value);
  return 0;
} #include <cstdio>

template <int N>
struct Factorial {
    static constexpr int value = N * Factorial<N - 1>::value;
};

template <>
struct Factorial<0> {
    static constexpr int value = 1;
};

int main() {
  printf("%d\n", Factorial<5>::value);
  return 0;
} Here, the computation of Factorial<5>::value takes place at compile time. However, this approach is difficult to read, requires specializations and overloads the compiler. Factorial<5>::value Example 2: Fibonacci Numbers #include <cstdio>

template <int N>
struct Fibonacci {
    static constexpr int value = Fibonacci<N - 1>::value + Fibonacci<N - 2>::value;
};

template <>
struct Fibonacci<0> {
    static constexpr int value = 0;
};

template <>
struct Fibonacci<1> {
    static constexpr int value = 1;
};

int main() {
  printf("%d\n", Fibonacci<10>::value);
  return 0;
} #include <cstdio>

template <int N>
struct Fibonacci {
    static constexpr int value = Fibonacci<N - 1>::value + Fibonacci<N - 2>::value;
};

template <>
struct Fibonacci<0> {
    static constexpr int value = 0;
};

template <>
struct Fibonacci<1> {
    static constexpr int value = 1;
};

int main() {
  printf("%d\n", Fibonacci<10>::value);
  return 0;
} Recursive templates are good for demonstrating ideas, but in real enterprise code they quickly become cumbersome. Drawbacks and limitations Debugging complexity: when errors occur in metacode, compiler messages can be extremely confusing.
Increased compilation time: recursive calculations generate numerous instances of templates.
Template limitations: difficulties with condition constructs and local variables. Debugging complexity: when errors occur in metacode, compiler messages can be extremely confusing. Increased compilation time: recursive calculations generate numerous instances of templates. Template limitations: difficulties with condition constructs and local variables. Nevertheless, templates metaprogramming is still successfully used today, including in the modern libraries, though more often for working with types than for purely mathematical operations. Modern approach: constexpr constexpr With the introduction of the constexpr specifier in C++11, template hacks began to take a back seat. The keyword constexpr informs the compiler that a function or an object can (and should, if possible) be evaluated at compile time. constexpr constexpr Basics of constexpr constexpr constexpr functions: They’re regular functions that can be evaluated at compile time if all arguments are known at the compile stage.
constexpr variables: Variables that are initialized by the constant expression.
Limitations:  in constexpr functions, operations that require runtime evaluation - such as a memory allocation with new or calling virtual functions - cannot be used. constexpr functions: They’re regular functions that can be evaluated at compile time if all arguments are known at the compile stage. constexpr constexpr variables: Variables that are initialized by the constant expression. constexpr Limitations:  in constexpr functions, operations that require runtime evaluation - such as a memory allocation with new or calling virtual functions - cannot be used. constexpr new Example 3: Factorial using constexpr constexpr #include <cstdio>

constexpr int factorial(int n) {
    return n <= 1 ? 1 : n * factorial(n - 1);
}

int main() {
  printf("%d\n", factorial(5));
  return 0;
} #include <cstdio>

constexpr int factorial(int n) {
    return n <= 1 ? 1 : n * factorial(n - 1);
}

int main() {
  printf("%d\n", factorial(5));
  return 0;
} This core is simpler than the template-based version and easier to read. Moreover, if factorial(5) is called with a constant, the result will be computed at compile time. factorial(5) Advanced aspects of constexpr constexpr Conditional operators (if, switch) inside constexpr functions have been available since C++14, which simplifies writing complex logic.


Loops (for, while) have also been allowed inside constexpr since C++14, which makes it easier to create initializer lists, arrays and lookup tables.


constexpr classes: you can declare class constructors and methods as constexpr, which allows creating objects at compile time and invoke their methods.
Example 4:  constexprand loops Conditional operators (if, switch) inside constexpr functions have been available since C++14, which simplifies writing complex logic. Conditional operators (if, switch) inside constexpr functions have been available since C++14, which simplifies writing complex logic. constexpr Loops (for, while) have also been allowed inside constexpr since C++14, which makes it easier to create initializer lists, arrays and lookup tables. Loops (for, while) have also been allowed inside constexpr since C++14, which makes it easier to create initializer lists, arrays and lookup tables. constexpr constexpr classes: you can declare class constructors and methods as constexpr, which allows creating objects at compile time and invoke their methods.
Example 4:  constexprand loops constexpr classes: you can declare class constructors and methods as constexpr, which allows creating objects at compile time and invoke their methods. constexpr constexpr Example 4:  constexprand loops constexpr #include <cstdio>

constexpr int sum_to_n(int n) {
    int sum = 0;
    for (int i = 1; i <= n; ++i) {
        sum += i;
    }
    return sum;
}

int main() {
  printf("%d\n", sum_to_n(15));
  return 0;
} #include <cstdio>

constexpr int sum_to_n(int n) {
    int sum = 0;
    for (int i = 1; i <= n; ++i) {
        sum += i;
    }
    return sum;
}

int main() {
  printf("%d\n", sum_to_n(15));
  return 0;
} Example 5:  constexpr and classes constexpr #include <cstdio>
#include <cmath>

struct Point {
    int x, y;

    constexpr Point(int x, int y) : x(x), y(y) {}
    constexpr float len() const { return sqrt(x * x + y * y); }
};

constexpr Point p(3, 4);

int main() {
  printf("%f\n", p.len());
  return 0;
} #include <cstdio>
#include <cmath>

struct Point {
    int x, y;

    constexpr Point(int x, int y) : x(x), y(y) {}
    constexpr float len() const { return sqrt(x * x + y * y); }
};

constexpr Point p(3, 4);

int main() {
  printf("%f\n", p.len());
  return 0;
} Useful features for compile-time computations static_assert : allows performing checks at compile-time
static_assert(sizeof(void*) == 8, "64-bit platform expected");

or using Example 4
static_assert(sum_to_n(15) == 120, "Something went wrong");



if constexpr (C++17): simplified form of conditional operators in templates
#include <iostream>

template <typename T>
void print_type_info(const T& val) {
    if constexpr (std::is_integral_v<T>) {
        std::cout << "Integer: " << val << std::endl;
    } else {
        std::cout << "Not an integer" << std::endl;
    }
}

int main() {
  print_type_info(15);
  return 0;
}

Here, the brunch that doesn’t satisfy the condition won’t be compiled at all if the check occurs at compile time.


Compile-time array and table generation: convenient for precomputing values
constexpr int squares[] = {
    1*1, 2*2, 3*3, 4*4, 5*5
};



Creating complex structures: you can declare an entire object as constexpr if all of its initialization methods are also constexpr. static_assert : allows performing checks at compile-time
static_assert(sizeof(void*) == 8, "64-bit platform expected");

or using Example 4
static_assert(sum_to_n(15) == 120, "Something went wrong"); static_assert : allows performing checks at compile-time static_assert static_assert(sizeof(void*) == 8, "64-bit platform expected"); static_assert(sizeof(void*) == 8, "64-bit platform expected"); or using Example 4 static_assert(sum_to_n(15) == 120, "Something went wrong"); static_assert(sum_to_n(15) == 120, "Something went wrong"); if constexpr (C++17): simplified form of conditional operators in templates
#include <iostream>

template <typename T>
void print_type_info(const T& val) {
    if constexpr (std::is_integral_v<T>) {
        std::cout << "Integer: " << val << std::endl;
    } else {
        std::cout << "Not an integer" << std::endl;
    }
}

int main() {
  print_type_info(15);
  return 0;
}

Here, the brunch that doesn’t satisfy the condition won’t be compiled at all if the check occurs at compile time. if constexpr (C++17): simplified form of conditional operators in templates if constexpr #include <iostream>

template <typename T>
void print_type_info(const T& val) {
    if constexpr (std::is_integral_v<T>) {
        std::cout << "Integer: " << val << std::endl;
    } else {
        std::cout << "Not an integer" << std::endl;
    }
}

int main() {
  print_type_info(15);
  return 0;
} #include <iostream>

template <typename T>
void print_type_info(const T& val) {
    if constexpr (std::is_integral_v<T>) {
        std::cout << "Integer: " << val << std::endl;
    } else {
        std::cout << "Not an integer" << std::endl;
    }
}

int main() {
  print_type_info(15);
  return 0;
} Here, the brunch that doesn’t satisfy the condition won’t be compiled at all if the check occurs at compile time. Compile-time array and table generation: convenient for precomputing values
constexpr int squares[] = {
    1*1, 2*2, 3*3, 4*4, 5*5
}; Compile-time array and table generation: convenient for precomputing values constexpr int squares[] = {
    1*1, 2*2, 3*3, 4*4, 5*5
}; constexpr int squares[] = {
    1*1, 2*2, 3*3, 4*4, 5*5
}; Creating complex structures: you can declare an entire object as constexpr if all of its initialization methods are also constexpr. Creating complex structures: you can declare an entire object as constexpr if all of its initialization methods are also constexpr. constexpr constexpr Combining Templates and constexpr constexpr Although constexpr often simplifies the task, there are situations in which templates remain necessary: constexpr Need to determine type characteristics, for example determine if a type is an array, a pointer, etc.
Need to store a compile-time list of types (using std::tuple or custom structures) Need to determine type characteristics, for example determine if a type is an array, a pointer, etc. Need to store a compile-time list of types (using std::tuple or custom structures) std::tuple Often it is more efficient and clear to perform all calculations using constexpr, and to use templates only where metaprogramming on types is truly needed. This approach strikes a balance between flexibility and ease of debugging. constexpr Pitfalls Compilation Time: Extensive use of metaprogramming and constexpr can significantly increase build times.
Expression Limitations: Not all operations are allowed in constexpr functions requiring full runtime support are prohibited.
Potential Hidden Copies: When using complex objects in constexpr functions, unexpected copies may occur if methods aren’t declared as constexpr or aren't properly optimized.
Debugging Complexity: If an error occurs inside a constexpr function, it’s not always easy to pinpoint the exact location. Compilation Time: Extensive use of metaprogramming and constexpr can significantly increase build times. constexpr Expression Limitations: Not all operations are allowed in constexpr functions requiring full runtime support are prohibited. constexpr Potential Hidden Copies: When using complex objects in constexpr functions, unexpected copies may occur if methods aren’t declared as constexpr or aren't properly optimized. constexpr constexpr Debugging Complexity: If an error occurs inside a constexpr function, it’s not always easy to pinpoint the exact location. constexpr Conclusion Conclusion Modern C++ offers developers a wide range of tools for organizing compile-time computations - from classic template-based metaprogramming to the more intuitive and flexible constexpr. When used wisely, these tools can significantly improve code performance and safety by catching entire classes of errors early. constexpr At the same time, it's important to weigh the trade-offs: will your code turn into an unreadable monolith, or will compilation times become unreasonably long? Within reasonable limits, metaprogramming and constexpr can make your project more efficient and reliable, offering tangible benefits when developing large-scale systems. constexpr Try It Yourself on GitHub If you want to explore these examples hands-on, feel free to visit my GitHub repository where you’ll find all the source files for the code in this article. You can clone the repository, open the code in your favorite IDE or build system, and experiment with constexpr to see how the compiler works with it. Enjoy playing around with the examples! GitHub repository constexpr Follow me Follow me LinkedIn LinkedIn Github Github

Stories, News & Mentions about INITIALIZED

Categories

Trending Topics

Stories, News & Mentions about INITIALIZED

Initialized’s Stories on HackerNoon

Initialized's News & Mentions

Categories

Trending Topics