publications

2024

1. Under Rev.

   Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models

   Mohammad Jalali, Azim Ospanov, Amin Gohari, and 1 more author

   Under Review, 2024

   Abs PDF Code URL

   Text-conditioned generation models are commonly evaluated based on the quality of the generated data and its alignment with the input text prompt. On the other hand, several applications of prompt-based generative models require sufficient diversity in the generated data to ensure the models’ capability of generating image and video samples possessing a variety of features. However, most existing diversity metrics are designed for unconditional generative models, and thus cannot distinguish the diversity arising from variations in text prompts and that contributed by the generative model itself. In this work, our goal is to quantify the prompt-induced and model-induced diversity in samples generated by prompt-based models. We propose an information-theoretic approach for internal diversity quantification, where we decompose the kernel-based entropy H(X) of the generated data X into the sum of the conditional entropy H(X|T), given text variable T, and the mutual information I(X;T) between the text and data variables. We introduce the \emphConditional-Vendi score based on H(X|T) to quantify the internal diversity of the model and the \emphInformation-Vendi score based on I(X;T) to measure the statistical relevance between the generated data and text prompts. We provide theoretical results to statistically interpret these scores and relate them to the unconditional Vendi score. We conduct several numerical experiments to show the correlation between the Conditional-Vendi score and the internal diversity of text-conditioned generative models.

2. NeurIPS 2024

   Towards a Scalable Reference-Free Evaluation of Generative Models

   Azim Ospanov, Jingwei Zhang, Mohammad Jalali, and 3 more authors

   The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

   Abs PDF Code URL

   While standard evaluation scores for generative models are mostly reference-based, a reference-dependent assessment of generative models could be generally difficult due to the unavailability of applicable reference datasets. Recently, the reference-free entropy scores, VENDI and RKE, have been proposed to evaluate the diversity of generated data. However, estimating these scores from data leads to significant computational costs for large-scale generative models. In this work, we leverage the random Fourier features framework to reduce the metrics’ complexity and propose the Fourier-based Kernel Entropy Approximation (FKEA) method. We utilize FKEA’s approximated eigenspectrum of the kernel matrix to efficiently estimate the mentioned entropy scores. Furthermore, we show the application of FKEA’s proxy eigenvectors to reveal the method’s identified modes in evaluating the diversity of produced samples. We provide a stochastic implementation of the FKEA assessment algorithm with a complexity linearly growing with sample size. We extensively evaluate FKEA’s numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method’s scalability and interpretability applied to large-scale generative models.

3. Under Rev.

   Identification of Novel Modes in Generative Models via Fourier-based Differential Clustering

   Jingwei Zhang, Mohammad Jalali, Cheuk Ting Li, and 1 more author

   Under Review, 2024

   Abs PDF URL

   An interpretable comparison of generative models requires the identification of sample types produced more frequently by each of the involved models. While several quantitative scores have been proposed in the literature to rank different generative models, such score-based evaluations do not reveal the nuanced differences between the generative models in capturing various sample types. In this work, we propose a method to identify modes produced by a generative model with a higher frequency in comparison to a reference distribution. FINC provides a scalable stochastic algorithm based on random Fourier features to estimate the eigenspace of kernel covariance matrices of two generative models and utilize the principal eigendirections to detect the sample types present more dominantly in each model. We demonstrate the application of the FINC method to standard computer vision datasets and generative model frameworks. Our numerical results suggest the scalability and efficiency of the developed Fourier-based method in highlighting the sample types captured with different frequencies by widely-used generative models.

4. Under Rev.

   Maximizing Rank Agreement Under Noisy Comparisons

   Mohammad Jalali, and Ramin Javadi

   Under Review, 2024

   Abs

   In this paper, we investigate the problem of ranking with maximum agreement under noisy comparisons. The input is a collection of pairwise comparisons between the elements of a ground set derived from a hidden embedded ranking which are exposed to an external noise and so may have inconsistencies. The goal is to reconstruct the ground truth with minimum error or maximize the number of satisfied comparisons. We propose three algorithms called In-Degree, PageRank, and Hybrid for solving this problem. We analyze our algorithms under a stochastic model which is a refinement of Erdős-Rényi model and each pairwise comparison is revealed independently with probability p and each revealed comparison is reversed with probability ε(noise rate). For the In-degree algorithm, we give a theoretical bound for the error rate of the obtained ranking in comparison with the ground truth. In particular we prove that when n is the number of items and p=α\ln n /n, for some constant α, then the error rate is at most O(α^-1/3(1-ε)^2/3/(1-2ε)). We also compare the performance of our algorithms with previous known methods in several experiments with synthetic and real data.

2023

1. 

   An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions

   Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia

   In 37th conference on Neural Information Processing Systems (NeurIPS), 2023

   Abs Bib PDF Supp Code Poster URL

   The evaluation of generative models has received significant attention in the machine learning community. When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model’s generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We utilize the R’enyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples.

   @inproceedings{Jalali2023rke,
     author = {Jalali, Mohammad and Li, Cheuk Ting and Farnia, Farzan},
     title = {An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions},
     year = {2023},
     booktitle = {37th conference on Neural Information Processing Systems (NeurIPS)},
     url = {https://openreview.net/forum?id=PdZhf6PiAb},
   }

2. 

   Games of GANs: game-theoretical models for generative adversarial networks

   Monireh Mohebbi Moghaddam, Bahar Boroomand*, Mohammad Jalali*, and 4 more authors

   Artificial Intelligence Review, Feb 2023

   Abs Bib PDF URL

   Generative Adversarial Networks (GANs) have recently attracted considerable attention in the AI community due to their ability to generate high-quality data of significant statistical resemblance to real data. Fundamentally, GAN is a game between two neural networks trained in an adversarial manner to reach a zero-sum Nash equilibrium profile. Despite the improvement accomplished in GANs in the last few years, several issues remain to be solved. This paper reviews the literature on the game-theoretic aspects of GANs and addresses how game theory models can address specific challenges of generative models and improve the GAN’s performance. We first present some preliminaries, including the basic GAN model and some game theory background. We then present a taxonomy to classify state-of-the-art solutions into three main categories: modified game models, modified architectures, and modified learning methods. The classification is based on modifications made to the basic GAN model by proposed game-theoretic approaches in the literature. We then explore the objectives of each category and discuss recent works in each class. Finally, we discuss the remaining challenges in this field and present future research directions.

   @article{MohebbiMoghaddam2023,
     author = {Mohebbi Moghaddam, Monireh and Boroomand*, Bahar and Jalali*, Mohammad and Zareian*, Arman and Daeijavad, Alireza and Manshaei, Mohammad Hossein and Krunz, Marwan},
     title = {Games of GANs: game-theoretical models for generative adversarial networks},
     journal = {Artificial Intelligence Review},
     year = {2023},
     month = feb,
     day = {13},
     issn = {1573-7462},
     doi = {10.1007/s10462-023-10395-6},
     url = {https://doi.org/10.1007/s10462-023-10395-6},
   }

2021

1. 

   Game of GANs: Game Theoretical Models for Generative Adversarial Networks

   Monireh Mohebbi Moghadam, Bahar Boroumand*, Mohammad Jalali*, and 3 more authors

   arXiv preprint arXiv:2106.06976, Feb 2021

   Abs arXiv Bib PDF URL

   Generative Adversarial Networks (GANs) have recently attracted considerable attention in the AI community due to its ability to generate high-quality data of significant statistical resemblance to real data. Fundamentally, GAN is a game between two neural networks trained in an adversarial manner to reach a zero-sum Nash equilibrium profile. Despite the improvement accomplished in GANs in the last few years, several issues remain to be solved. This paper reviews the literature on the game theoretic aspects of GANs and addresses how game theory models can address specific challenges of generative model and improve the GAN’s performance. We first present some preliminaries, including the basic GAN model and some game theory background. We then present taxonomy to classify state-of-the-art solutions into three main categories: modified game models, modified architectures, and modified learning methods. The classification is based on modifications made to the basic GAN model by proposed game-theoretic approaches in the literature. We then explore the objectives of each category and discuss recent works in each category. Finally, we discuss the remaining challenges in this field and present future research directions.

   @article{MohebbiMoghaddamArXiv2021,
     author = {Moghadam, Monireh Mohebbi and Boroumand*, Bahar and Jalali*, Mohammad and Zareian*, Arman and Javad, Alireza Daei and Manshaei, Mohammad Hossein},
     title = {Game of GANs: Game Theoretical Models for Generative Adversarial Networks},
     journal = {arXiv preprint arXiv:2106.06976},
     volume = {abs/2106.06976},
     year = {2021},
     eprint = {2106.06976},
   }