Why isn't the United States Flag considered racist since it was the original slave owners flag?

Considering the Confederate flag is considered racist and flew over the south for only 4 years and th U.S. Flag had flown over the slaves from the time they were bought from the blacks in Africa and sold in the North. Why?