Madison Square and Madison Square Garden are, as is clear from this map, two different destinations. Madison Square was commercial, and home to many of the city’s most exclusive stores. From Wikipedia:
On May 10, 1847 Madison Square Park opened to the public. Within a few years, the tide of residential development, which was relentlessly moving uptown, had reached the Madison Square area, and through the 1870s, the neighborhood became an aristocratic one of brownstone row houses and mansions where the elite of the city lived; Theodore Roosevelt, Edith Wharton and Winston Churchill‘s mother, Jennie Jerome, were all born here. In 1853, plans had been made to build the Crystal Palace there, but strong public opposition and protests caused the palace to be relocated to Bryant Park.
The Fifth Avenue Hotel, a luxury hotel built by developer Amos Eno, and initially known as “Eno’s Folly” because it was so far away from the hotel district, stood on the west side of Madison Square from 1859 to 1908. The first hotel in the city with elevators, which were steam-operated and known as the “vertical railroad”, it had fireplaces in every bedroom, private bathrooms, and public rooms which saw many elegant events. Notable visitors to the hotel included Mark Twain, famed Swedish singer Jenny Lind, U.S. Presidents Chester A. Arthur and Ulysses S. Grant and the Prince of Wales.
The thing you’d most likely notice in Madison Square Park — were you to walk through in 1880 or so, is Statue of Liberty.
Compare these photos looking south from about 25th Streetand Broadway to see how the intersection of 23rd and Broadway changed between 1883 and 1902, the year the Flatiron Building opened.