I saw this chart today (here) and didn't believe it so I double checked the Census Bureau and it's true.
The current projection is for the US population to now peak at just-under 370 million in 2080 but with it reaching 359m in 2040, it's a really long plateau.
Given the plunge in fertility rates and the hardening of attitudes against immigration and the trend, I'd certainly say that risks are to the downside.
There has been plenty of talk in the market about China's population peaking but China still has a great deal of rural-to-urban immigration to supplement the productive workforce.
In any case, the US is far more important for the global economy and these changes in population materially affect the long-term prospects of virtually every consumer-facing business, along with real estate values.
It's an even bigger problem in Europe and many other parts of the world. Changing attitudes about fertility and the number of children people want to have are a real problem for markets.