Nottingham Forest right-back Orr side-footed home in the 66th minute in Kutaisi after Uefa credited midfielder Gabriel Sigua with the hosts’ seventh-minute opener.
Northern Ireland substitute, Arsenal striker Ceadach O’Neill, who was making his Under-21 debut, twice threatened a late winner in the Group F fixture, brushing the crossbar with a powerful strike before seeing an added-time attempt hacked off the line.
Manager Tommy Wright gave debuts to Leicester City midfielder Ryan Donnelly and Dundalk striker Eoin Kenny.
The visitors survived an early scare at Ramaz Shengelia Stadium when Irakli Egoyan’s curling shot deflected off Tom Atcheson and struck the top of the bar.
However, Georgia took the lead in scruffy fashion from the resulting corner.
Goalkeeper Fraser Barnsley did well to keep out Luka Latsabidze’s close-range effort after Aleksandre Narimanidze diverted Giorgi Tabatadze’s delivery towards goal but the loose ball rebounded off former Cliftonville ace Shea Kearney, who was under pressure from Georgia captain Sigua, and crossed the line.
Kenny tried his luck from distance on 15 minutes, however his low shot was comfortably saved by Georgia keeper Mikheil Makatsaria.
On 53 minutes Makenzie Kirk’s header drifted wide of the target and another effort was intercepted by Makatsaria.
Former Linfield defender Orr levelled by finishing at the second attempt after his initial header was blocked following a searching free-kick into the box from substitute Liam McStravick.
The lively O’Neill was unfortunate to be denied by the woodwork in the 88th minute following a delightful turn away from Latsabidze before being thwarted by a last-gasp clearance from Narimanidze after Barnsley turned away Diego Deisadze’s header at the other end.
Georgia are second seeds in Group F which also features top seeds Germany, Greece, Latvia and Malta.
Next up for the Under-21s in the qualifiers are home games against Malta and Germany on October 9 and 14 at The Oval.
Game number four for them is also a home fixture at The Oval against Latvia on November 13.
The result and performance will please former Northern Ireland keeper Wright who stated: “We want to promote and push players and give them exposure to higher levels of international football as early as we can.”
The boys prepared for the game with a training camp in Loughborough.
The draw was former Northern Ireland international Corry Evans’ first as a coach with the side.
In Group D, Republic of Ireland edged past Andorra 1-0. Michael Noonan grabbed the winner on 66 minutes.
Georgia: Makatsaria, Bukia, Kharebashvili, Narimanidze, Latsabidze, Lordkipanidze, Tsulukidze, Egoyan, Deisadze, Sigua (c), Tabatadze.
Subs (used): Basiladze, Karseladze, Gegiadze, Dadiani, Gotsiridze. Unused: Kajaia, Kharebava, Giunashvili, Nadareishvili.
Northern Ireland: Barnsley, Orr, Russell, Robinson, Forbes (c), Atcheson, Kearney, Kelly, Kirk, Kenny, Donnelly.
Subs (used): McStravick, O’Neill. Unused: Hurl, Patterson, McCann, Graham, Turley, Barr, Inwood.
A different equation in qualifying group can still add up to NI’s World Cup dreams coming true