HBO’s hit series “The Gilded Age,” set in 1880s Manhattan, was filmed at a number of historic mansions dating back to the famously ostentatious era. But most are located in Newport, R.I., or Troy, N.Y., according to an HBO spokeswoman—virtually no filming took place in Manhattan.

Why? Because the palatial Gilded Age mansions of New York City have almost completely vanished. Starting around 1880, New Yorkers built some of the largest and most elaborate houses the country had ever seen. On Fifth Avenue, Riverside Drive and elsewhere in the city stood free-standing homes modeled after European palaces, some with more than 100 rooms and spanning entire city blocks. Yet within a few decades, virtually all of these massive houses had been demolished. “These amazing mansions that were built along Fifth Avenue—many of them have been destroyed,” said Helen Zoe Veit, an associate history professor at Michigan State who is a historical consultant for the HBO show.

In the 1880s and ‘90s, the city’s most desirable residential neighborhood was Vanderbilt Row, a stretch of mansions on Fifth Avenue between 50th Street and the southeast corner of Central Park. Today the area is a frenzied commercial district of office buildings and retail stores, with virtually no trace of the posh single-family homes that once stood there.

For example, William K. Vanderbilt and his social-climbing wife, Alva Vanderbilt—the inspiration for Bertha Russell on “The Gilded Age”—lived in a white limestone château occupying nearly a full block at Fifth Avenue and 52nd Street. “If you see pictures of it, it just blows your mind that this was on Fifth Avenue, and that anybody would build a house in such an ostentatious château style,” said Esther Crain, author of “The Gilded Age in New York, 1870-1910.”

Alva Vanderbilt at the grand costume ball she hosted for around 1,200 guests at Petit Château in 1883.

PHOTO: PSNC ARCHIVES AND SPECIAL COLLECTIONS

Indeed, elite architecture in the Gilded Age was all about one-upmanship, she said. With great fortunes being made along with the country’s industrialization, newly wealthy families vied to show off their riches by spending millions—tens or sometimes hundreds of millions in today’s dollars—constructing massive houses. “It was all about the show—I’m wealthier than you,” said historian Tom Miller, author of “Seeking New York: The Stories Behind the Historic Architecture of Manhattan—One Building at a Time.”

Things had changed dramatically by the 1920s, however. Federal income taxes had made huge mansions financially unsustainable, plus it became harder to find the armies of servants required to maintain them. During the building boom of the ‘20s, many were razed to make way for commercial use or multistory apartment houses, which were billed as modern and convenient.

In the Gilded Age, “if you had any kind of standing in the city and wealth, you wanted your own single-family home,” said Ms. Crain. By the 1920s, however, “if you had a lot of money, you probably would prefer to live in an apartment building because the building took care of everything for you.”

A few Gilded Age-era mansions—such as the Frick Collection and the Cooper Hewitt design museum—remain intact on the Upper East Side. Midtown, however, saw scores of grand homes destroyed, Mr. Miller said. “The houses south of 59th Street just got wiped out because of the commercial district,” he said. And farther west, nearly all of the 30 or 40 large mansions that once lined Riverside Drive have also been replaced with apartment buildings, Ms. Crain said. Mansions in Brooklyn’s fashionable St. Marks District in Crown Heights met a similar fate.

“These were very special buildings that should have been preserved,” said Mr. Miller.

Read on for a closer look at the vanished mansions of Gilded Age New York, and some of the rare structures that remain.

Morton F. Plant House, 649 Fifth Ave.

Completed: Circa 1905

Rooms: Roughly 50

Cost of land and construction: $750,000, about $24.1 million today

The former Plant house, now a Cartier store, is one of the few remaining Gilded Age mansions in Midtown.

PHOTO: ZACK DEZON FOR THE WALL STREET JOURNAL

Today, office workers and tourists passing the Cartier flagship store on the traffic-clogged corner of Fifth Avenue and 52nd Street in Midtown would never suspect that the six-story, Renaissance-style building was originally a private home. The house was built for Morton F. Plant, son of the railroad and shipping tycoon Henry Plant, according to the city’s Landmarks Preservation Commission.

Mr. Plant bought the land for about $350,000 in 1902, according to the book “Great Houses of New York, 1880-1930” by Michael C. Kathrens, and spent about $400,000 building the house. It had roughly 50 rooms, including a music room, a men’s smoking room and 12 staff rooms.

Cartier purchased the house in 1917—around the time wealthy families were abandoning the neighborhood in the face of encroaching commercial development—and converted it into a jewelry store, preserving much of the building’s original appearance: a pediment on the 52nd Street facade has two carved cornucopia.

The story goes that the Plants traded the house to Cartier in exchange for a $100 and a $1 million string of pearls that Mrs. Plant admired. Inside, the space retains its massively high ceilings.

As the only Midtown mansion to remain intact, Mr. Miller said, “the Plant mansion gives us an idea of what Fifth Avenue looked like.”

More: 15,000-Acre California Ranch Previously Listed for $38 Million Heads to Auction

Petit Château, the William K. Vanderbilt and Alva Vanderbilt mansion, 660 Fifth Ave.

Completed: Circa 1882.

Number of rooms: Roughly 40

Cost to build: $3 million, about $83.5 million today

Sold for: Roughly $3.75 million in the 1920s

Demolished: 1926

Today, 660 Fifth Avenue is a 39-story office building with Zara, Uniqlo and Hollister stores at its base and hoodie-clad mannequins in its shop windows. But on this site once stood a massive home, designed by Richard Morris Hunt and modeled after a 16th-century French castle. Nicknamed Petit Château, the house had a slender 3-story turret carved with fleur de lis at the entrance. A two-story, stone banquet hall was 50 feet long. The home cost $3 million, about $83.5 million today, according to “Fortune’s Children: The Fall of the House of Vanderbilt.”

Arches inside Petit Château.

PHOTO: BETTMANN ARCHIVE/GETTY IMAGES

With its pale Indiana limestone exterior, the house was far different from the brownstones that New York’s elite had favored up until that point, according to the architect and historian Gary Lawrance, who estimated that the house had 40 rooms. “You take a neighborhood of brownstones and stick a dazzling white limestone house in the middle of it—that really broke the mold of what had been there prior,” he said. Petit Château is perhaps best known as the setting for a grand costume ball Alva Vanderbilt threw for about 1,200 guests in 1883.

It was sold for $3.75 million in the 1920s and razed in 1926.

More: New Build in Santa Barbara, California, With a Tesla Solar Roof to List for $9.95 Million

The Cornelius Vanderbilt ll mansion, 1 West 57th Street

Completed: Circa 1883

Number of rooms: about 91

Cost of land and construction: $3.375 million, about $94.8 million today.

Sold for: Roughly $7.1 million

Demolished: 1927

Bergdorf Goodman.

PHOTO: ZACK DEZON FOR THE WALL STREET JOURNAL

The home of Cornelius Vanderbilt II and his wife, Alice, was “the Buckingham Palace of Fifth Avenue,” Mr. Lawrance said. When the five-story, roughly 50-room French chateau of red brick and limestone was built, newspapers proclaimed that it was the largest single-family home in New York. Still, the Vanderbilts decided to expand it, tearing down five townhouses to create a behemoth occupying the entire block of Fifth Avenue between 57th and 58th streets with about 91 rooms, according to Jason Bouchard-Nawrocki, an archivist for Vanderbilt descendant Gladys Szapary. The house required more than 30 servants to run it and was larger than The Breakers, the couple’s famous summer house in Newport.

The house was sold for roughly $7.1 million in 1927 and replaced by the Bergdorf Goodman department store.

Mrs. Astor’s house, 840 Fifth Ave.

Completed: Circa 1896

Number of rooms: Roughly 60

Sold for: $3.5 million in 1925

Demolished: 1926

John J. Astor IV and his wife, Madeleine. Mr. Astor died on the Titanic in 1912.

PHOTO: GETTY IMAGES

The home of society doyenne Caroline Astor and her son, John J. Astor IV, at 65th Street “was a masterpiece of Beaux-Arts Architecture,” said Mr. Miller. The gray-limestone chateau was designed by architect Richard Morris Hunt as a double house, with one side for Mrs. Astor and the other side for her son. According to the book “Gilded Mansions: Grand Architecture and High Society” by Wayne Craven, the home’s entertainment spaces were designed to be joined into a single, 2-story ballroom. Topped with a stained-glass dome and hung with some 100 paintings, the ballroom could accommodate 1,200 revelers.

The family sold the mansion in 1925 for $3.5 million, according to the Kathrens book, and it was demolished to make way for the synagogue Temple Emanu-El, one of the largest houses of Jewish worship in the world. The Astors’ wine cellar remained when the synagogue was built, and is now used for storage, said Temple Emanu-El Administrator Mark H. Heutlinger.

More: Moated English Country House With Royal Ties Lists for £6.5 Million

Isaac Fletcher House, 975 Fifth Ave.

Completed: Circa 1899

Number of rooms: 116, roughly 16,000 square feet

Cost of land and construction: Roughly $400,000, about $13.6 million today

Sold for: Roughly $200,000 in 1955

Fifth Avenue between 78th and 79th Streets is one of the few blocks in the city to appear largely the same as it did in the early 1900s, according to Landmarks. On the corner stands a French Gothic-style limestone mansion built for industrialist Isaac Fletcher in 1899. In the 1950s, the property was purchased by the Ukrainian Institute of America. Mr. Fletcher spent about $400,000 to purchase the land and build the house, according to the Ukrainian Institute’s Jasper Santa Ana; the institute paid about half that amount to buy it in 1955.

Designed by C.P.H. Gilbert, the 6-story house remains mostly intact, according to Mr. Santa Ana. The exterior is carved with gargoyles, sea horses and griffins, while the steeply pitched slate roof has richly adorned turrets. The interiors retain their elaborately carved woodwork, marble mantels, and small round call buttons used for summoning the maid and butler. An elevator and silver safe are also original to the house.

The house is separated from the street by a moat-like courtyard with a “drawbridge” over it.

Since Russia’s invasion of Ukraine, the building has received hundreds of visitors, many leaving bouquets, said Mr. Santa Ana.

Seamans Mansion, 789 St. Marks Ave., Brooklyn

Completed: Circa 1904

Number of rooms: 32 rooms

Cost to build and furnish: $2 million, about $63.4 million today

Sold for: $250,000 in 1921

Demolished: 1928

Today, the former site of the Clarence Seamans mansion is an out-of-the-way Brooklyn block lined with down-at-heels apartment buildings and a few dilapidated townhouses. In the early 1900s, however, the area was known as the St. Marks District, one of the most expensive and exclusive neighborhoods of Brooklyn, according to architectural historian Suzanne Spellen. It was here that Mr. Seamans, a typewriter mogul, hired architect Montrose W. Morris to design the 4-story limestone mansion widely considered to be the finest house in Brooklyn, Ms. Spellen said. The Italian Renaissance Revival house “was a big topic of the day because it was so big and so ostentatious,” she said. It had a bowling alley, a ballroom and an “Orientalist Room.” There was also a swimming pool and an underground passageway leading from the main house to the carriage house.

Mr. Seamans died in 1915. His wife sold the house a few years later, in 1921, for $250,000. It was later torn down to make way for the Excelsior apartment building. “It didn’t even last a generation,” Ms. Spellen said.

Riverside, the Charles M. Schwab mansion, Riverside Drive at 73rd Street

Completed: Circa 1906

Number of rooms: 75

Cost for land and construction: almost $9 million, around $279 million today.

Sold for: for $1.5 million in 1947

Demolished: 1948

The 4-story home of steel magnate Charles M. Schwab occupied an entire block in the 70s on Riverside Drive. Mr. Schwab paid $865,000 for the land and spent some four years and about $8 million building the house, around $279 million today, according to his great niece, Ann Dougherty.

Built of steel, granite and limestone, the free-standing 75-room house had a bowling alley, a chapel, a 60-foot swimming pool, three elevators and an extensive garden. A music lover, Mr. Schwab bought an organ for the house and concealed its pipes behind tapestries woven by 100 Flemish women who had been brought to America specifically for that purpose, according to “Steel Titan: The Life of Charles Schwab” by Robert Hessen.

With his finances depleted during the Depression, Mr. Schwab offered to sell Riverside and its furnishings for $4 million to the city for use as the mayor’s official residence. Mayor Fiorello La Guardia refused.

Riverside was eventually repossessed in payment for Mr. Schwab’s debts, and he moved into a small apartment. The house was sold for $1.5 million. After its demolition in 1948, two apartment buildings rose in its place.

Ms. Dougherty, 79, recalls crawling into a dumbwaiter while visiting Riverside as a small girl, and later watching its demolition. “It was so sad,” she said. “It was quite a place.”

More: This Multimillion-Dollar Sculpture-Like Croatian Villa Is Headed to Auction Next Month

The William A. Clark house, 962 Fifth Ave.

Completed: Circa 1911

Number of rooms: 121

Cost to build: $7 million, about $209 million today

Sold for: Less than $3 million

Demolished: 1927

Even by Gilded Age standards, the free-standing Beaux-Arts mansion of William A. Clark at 77th Street was viewed as over the top, Ms. Crain said. “It was just hideous because it was such a design mash up,” she said.

Sen. Clark, who made his fortune in mining, banking and railroads, had lived much of his life in Montana, which he represented in the U.S. Senate in the early 1900s. In building the house, he incorporated design elements that alluded to his two “places of the heart”—France and Montana, noted his great-great-grandson, Ian Clark Devine. “I don’t think the architectural critics of the day understood the importance of these influences for the Senator, so they simply dismissed the mansion out of hand,” Mr. Devine said.

Surrounded by 20-foot-high bronze gates, the house had 121 rooms, including 31 bathrooms and five art galleries, according to the book “Empty Mansions” by Bill Dedman and Paul Clark Newell, Jr. An ornate domed tower rose 9 stories high. The basement had Turkish baths, a swimming pool and a storage room for furs. A railroad spur brought in coal for the furnace.

The house was only occupied for 14 years. After Sen. Clark died in 1925, the house was sold and demolished. Today an apartment building stands in its place.