Each team scored once in its first three tries during the shootout, and Nylander was the fourth Toronto shooter. Mikael Backlund failed as the fourth Calgary shooter, clinching the win for Toronto.
Mark Giordano scored for Calgary (14-12-2) in regulation, and Morgan Rielly (18-10-1) replied for Toronto.
Mike Smith made 28 saves in the Flames goal.
The Flames took the lead at 10:37 of the first period on the power play. Giordano scored his fourth goal of the season as his snap shot from just inside the blue line found its way through traffic. Sean Monahan and Troy Brouwer earned the assists.
Toronto's Matt Martin was serving a holding penalty.
Calgary had a 14-12 advantage in shots on goal in the first period.
The Maple Leafs tied the game at 18:47 of the second period on the fourth goal of the season by Rielly, a shot from the edge of the left circle near the boards.
The Flames led in shots on goal 26-21 after two periods.
Calgary put the puck into the goal at 5:10 the third period, but the goal was disallowed because the net had been dislodged.
Nylander hit the bar with a shot with just under three minutes to play in the third.
Each team had some good chances in the free-wheeling overtime.
The Maple Leafs went on the power play with 31.5 seconds left in the overtime when Johnny Gaudreau was penalized for holding, but Calgary held on to force the shootout.
NOTES: Calgary RW Jaromir Jagr (lower-body injury) did not play Wednesday because of the nagging injury. He also missed the 4-1 loss to the Maple Leafs in Calgary on Nov. 28. He could play Thursday against the Montreal Canadiens. ... Toronto C Tyler Bozak (illness) returned to the lineup Wednesday after missing a loss to the Canucks in Vancouver on Saturday because of food poisoning. ... The Maple Leafs will visit Pittsburgh on Saturday. ... The Flames travel to Montreal on Thursday.