![\"\"](\"https:\/\/www.hpcwire.com\/wp-content\/uploads\/2025\/05\/Harness-LLM-2-LLMbased_web.jpg\")
<\/a>Figure 1: Schematic of the workflow for LLM-based code conversion process. Steps in blue are managed using CodeScribe, while steps in red are manual and require developer intervention.<\/p>\nIn ongoing work, the researchers are applying CodeScribe to other applications. For example, they use CodeSource to build GPU compatibility between the Flash-X open-source multiphase simulation software and the AMRex framework for block-structured adaptive mesh refinement applications. The researchers envision CodeScribe as a valuable tool that empowers developers in scientific computing to leverage generative AI effectively.<\/p>\n
For further information, see A. Dhruv and A. Dubey, \u200b\u201cLeveraging Large Language Models for Code Translation and Software Development in Scientific Computing,\u201d accepted by the Platform for Advanced Scientific Computing conference.<\/a><\/p>\n Making LLMs More Manageable\u00a0<\/p>\n Another major challenge facing LLMs is making them accessible with significantly reduced computational resources.\u00a0Pruning has emerged as an important compression strategy to enhance memory and computational efficiency, but traditional global pruning has been impractical for LLMs because of scalability issues.<\/p>\n Researchers from Emory University and Argonne have developed SparseLLM to address this challenge.\u00a0This innovative method redefines the global pruning process into multiple local optimization subproblems coordinated by auxiliary variables. The researchers introduced an alternating optimization strategy in which some subproblems are optimized while others are kept fixed; the process is then repeated with a different subset. For the optimization, they leveraged sparsity-aware algorithms to optimize both the pruning mask selection and weight reconstruction simultaneously, ensuring minimal performance degradation (see Fig. 2).<\/p>\n \u201cBy not optimizing all the variables at the same time, we can achieve more scalable training while reducing the computational cost,\u201d said Kibaek Kim, a computational mathematician and one of the developers of SparseLLM.<\/p>\n For further information, see the paper by Guangji Bai, Yijiang Li, Kibaek Kim, and Liang Zhao, \u200b\u201cTowards Global Pruning for Pre-trained Language Models,\u201d arXivL2402.17946<\/a>; poster at NeurlPS\u00a0 2024.<\/p>\n Reasoning with LLMs<\/p>\n Whether LLMs can reason has become a highly debated issue. Some studies cite achievements in multistep planning and prediction as demonstrating LLMs\u2019 reasoning capabilities; others argue that \u200b\u201ctrue reasoning\u201d goes beyond LLMs\u2019 ability to recognize patterns and apply logical rules. In a recent study, Argonne and the University of Pennsylvania researchers joined the debate by focusing on a new aspect \u2014 LLM token biases when solving logical problems.<\/p>\n They introduced a hypothesis-testing framework to evaluate multiple commercial and open-source LLMs. They applied tests on matched problem pairs to detect performance shifts when logically irrelevant tokens, such as names or quantifiers, were altered. The results showed that many state-of-the-art LLMs fail to generalize logical reasoning across minor perturbations, suggesting they often rely on superficial token patterns rather than formal logical reasoning (see Fig. 3).<\/p>\n \u201cWe demonstrated statistically that apparent reasoning success may stem from token bias rather than actual understanding,\u201d said Tanwi Mallick, an assistant computer scientist. \u200b\u201cThe study provides new insights into the reliability of LLMs and opens avenues for future work on ways to improve LLMs\u2019 logical reasoning ability.\u201d<\/p>\n For further information, see the paper by Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth, \u200b\u201cA Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners<\/a>,\u201d in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4722\u20134756<\/a>.<\/p>\n Visualizing with an LLM<\/p>\n Creating scientific visualizations is challenging, consuming considerable time, and requiring data analysis and visualization expertise. Four researchers from Argonne\u2019s MCS Division have proposed a new approach:\u00a0synthetic software generation using an LLM. To this end, they have developed\u00a0an AI assistant, ChatVis, that allows the user to specify a chain of analysis\/visualization operations in natural language.\u00a0<\/p>\n ChatVis generates a Python script for the desired operations and iterates until the script executes correctly, prompting the LLM to revise the script as needed. Moreover, the LLMs are\u00a0not trained in esoteric visualization operations; instead, ChatVis allows commonly available LLMs, such as ChatGPT, to generate correct visualizations without retraining or fine-tuning.\u00a0<\/p>\n \u201cChatVis employs a friendly human-centric natural-language interface,\u201d said Orcun Yildiz, an assistant computer scientist. \u200b\u201cDomain scientists, managers, administrators, and decision-makers who are not visualization experts can now generate their own\u00a0high-quality visualizations.\u201d<\/p>\n The Argonne team compared visualizations against five state-of-the-art LLM models with and without ChatVis. With ChatVis, they generated all five visualizations successfully; without ChatVis, the best LLM could generate only one of the five cases correctly. Figure 4 shows how closely ChatVis matched the ground truth.<\/p>\n For the full study, see the paper by Tanwi Mallick, Orcun Yildiz, David Lenz, and Tom Peterka, \u200b\u201c<\/strong>ChatVis: Automating Scientific Visualization with a Large Language Model,\u201d in\u00a0SC24-W: Workshops of the International Conference for High-Performance Computing,\u00a0Networking, Storage and Analysis, pp. 49-55<\/a>.<\/p>\n Pruning and Privacy with LLMs\u00a0<\/p>\n The enormous size of LLM datasets means they have high computational and storage demands. Pruning can reduce model size, but most methods assume public access to the data, hindering their use in privacy-sensitive applications.<\/p>\n Researchers at Argonne and Emory University have now proposed the first federated learning framework designed specifically for pruning LLMs over distributed data silos. Called FedSpaLLM, the new framework enables clients to prune their models locally based on private datasets, each owned by a different institution. The approach allows collaboration without sharing the raw data; only the model updates (e.g., weights or parameters) are shared, thus ensuring data privacy.<\/p>\n The researchers introduce three innovations in FedSpaLLM: a specialized aggregation function to handle sparse model updates, an adaptive mask expansion technique to ensure that the global model meets the target sparsity, and a layer sampling strategy that allows clients to prune subsets of model layers based on their computational resources, enabling personalized pruning and reducing communication costs.<\/p>\n Extensive experiments show the efficacy of the approach. The key metric used with FedSpaLLM to evaluate the models was \u200b\u201cperplexity,\u201d a well-known metric considered well-suited for assessing the accuracy of compression methods and thus measuring the performance of the compressed models.\u00a0<\/p>\n \u201cPerplexity seems an odd name, but it reflects the confidence the model has in making a prediction,\u201d said Yijiang Li, a postdoctoral appointee. \u200b\u201cLower perplexity means greater confidence.\u201d<\/p>\n As shown in Fig. 5, FedSpaLLM consistently outperforms \u200b\u201cstandalone\u201d client models in achieving lower perplexity. Random pruning (not shown) suffers from perplexity, an order of magnitude higher than the standalone models and FedSpaLLM.<\/p>\n In general, as the target sparsity increases, the perplexity of the global model (e.g., FedSpaLLM) improves more than that of the client models (e.g., standalone).\u00a0<\/p>\n The results highlight the benefits of FedSpaLLM as a promising solution for resource-constrained applications where privacy is critical.<\/p>\n For further information, see the preprint by Guangji Bai, Yijiang Li, Zilinghan Li, Liang Zhao, and Kibaek Kim. \u200b\u201cFedSpaLLM: Federated Pruning of Large Language Models<\/a>,\u201d to appear in NAACL 2025.<\/p>\n<\/a>Figure 2: SparseLLM decomposes the global pruning of LLMs into manageable subproblems by leveraging the chain of modules and auxiliary variables<\/p>\n<\/a>Figure 3: Experimental results in which the perturbed problems add the names of trustworthy news agencies and universities to alter the narratives of syllogisms. LLMs tend to falsely believe that these narratives are more trustworthy and hence ignore the logical fallacy in them.<\/p>\n<\/a>Figure 4: Generated images for volume rendering. ChatVis produced a screenshot identical to the ground truth except for a different color palette because the user prompt did not specify one.<\/p>\n<\/a>Figure 5: Performance improvements in the metric perplexity from FedSpaLLM over client \u200b\u201cstandalone\u201d models.<\/p>\n