Alibaba Group Holding has introduced a computing pooling solution that it said led to an 82 per cent cut in the number of Nvidia graphics processing units (GPUs) needed to serve its artificial intelligence models.<\/p>\n
The system, called Aegaeon, was beta tested in Alibaba Cloud\u2019s model marketplace for more than three months, where it reduced the number of Nvidia H20 GPUs required to serve dozens of models of up to 72 billion parameters from 1,192 to 213, according to a research paper presented this week at the 31st Symposium on Operating Systems Principles (SOSP) in Seoul, South Korea.<\/p>\n
\u201cAegaeon is the first work to reveal the excessive costs associated with serving concurrent LLM workloads on the market,\u201d the researchers from Peking University and Alibaba Cloud wrote.<\/p>\n
Alibaba Cloud is the AI and cloud services unit of Hangzhou-based Alibaba, which owns the Post. Its chief technology officer, Zhou Jingren, is one of the paper\u2019s authors.<\/p>\n
Cloud services providers, such as Alibaba Cloud and ByteDance\u2019s Volcano Engine, serve thousands of AI models to users concurrently, meaning that many application programming interface calls are handled at the same time.<\/p>\n
However, a small handful of models such as Alibaba\u2019s Qwen and DeepSeek are most popular for inference, with most other models only sporadically called upon. This leads to resource inefficiency, with 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud\u2019s marketplace, the researchers found.<\/p>\n
Researchers globally have sought to improve efficiency by pooling GPU power, allowing one GPU to serve multiple models, for instance.<\/p>\n","protected":false},"excerpt":{"rendered":"Alibaba Group Holding has introduced a computing pooling solution that it said led to an 82 per cent…\n","protected":false},"author":3,"featured_media":312732,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[691,738,3774,64,74,79,8464,1165,50,1269,14002,1926,158,67,132,68,103],"class_list":{"0":"post-312731","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-asia","11":"tag-business","12":"tag-china","13":"tag-economy","14":"tag-hong-kong","15":"tag-lifestyle","16":"tag-news","17":"tag-opinion","18":"tag-south-china-morning-post","19":"tag-sport","20":"tag-technology","21":"tag-united-states","22":"tag-unitedstates","23":"tag-us","24":"tag-world"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/115393761281208751","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/312731","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=312731"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/312731\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/312732"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=312731"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=312731"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=312731"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}