Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh
Published in Arxiv
A training framework for LoRA that learns accurate hierarchical low-rank representations for different sub-ranks without retraining.
Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F Shen, Xinchi Qiu, Dan Alistarh, Nicholas D Lane
Published in Arxiv
A framework that unifies low-rank optimization with infrequent synchronization.
Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis, Mher Safaryan, Dan Alistarh
Published in Arxiv
Faster implementation of Distributed Shampoo by stacking the preconditioner blocks into 3D tensors.
Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis, Max Ryabinin, Artem Chumachenko, Dan Alistarh
Published in Arxiv
FFT-based low-rank optimization for LLMs.
Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, James Martens
Published in Arxiv
Optimizers have been introduced and benchamrked with respect to how fast they reach a specific loss. In this work we hypothesize they might also have other effects, such as inducing certain biases.
Andrei Panferov, Alexandra Volkova, Ionut-Vlad Modoranu, Vage Egiazarian, Mher Safaryan, Dan Alistarh
Published in Arxiv
Scaling laws for quantization and sparsity.
Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, Dan Alistarh
Published in NeurIPS 2024
Theoretical guarantees for sparse, second order pruning.
Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, Dan Alistarh
Published in ICLR 2025
Improved low-rank optimization for LLMs (over GaLore).
Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtarik, Dan Alistarh
Published in NeurIPS 2024
Reduce the memory usage of Adam optimizer via sparsity and error feedback.
Ionut-Vlad Modoranu, Aleksei Kalinov, Eldar Kurtic, Elias Frantar, Dan Alistarh
Published in ICML 2024
Reduce the memory usage of M-FAC optimizer via sparsity, low-rank and error feedback.
Sanghyun Hong, Yiğitcan Kaya, Ionuţ-Vlad Modoranu, Tudor Dumitraş
Published in ICLR 2021 (Spotlight 🔦)
A new adversarial attack to introduce delay in the predictions of multi-exit deep neural networks.