UnitedHealth Group Inc. is testing a new system to streamline how medical claims are processed, an early example of what the company says is the potential for artificial intelligence to smooth out friction in billing.

The system, dubbed Optum Real, aims to distill health plans’ complex rules around what is covered into information that doctors and billing staff can use in real time to tell whether a claim is likely to be paid.