The transmission undergoes a lot of stress. The grit you see in used transmission fluid is actually bits of metal that wear off the gears in the transmission. In addition to that, the transmission operates at very high temperatures. Usually it's 100 to 150 degrees higher than engine temperatures. Those high temperatures eventually cause the transmission fluid to start to break down and loose efficiency.
As the fluid gets older, it gets gritty and doesn't lubricate and cool the transmission as well - leading to even more wear. The fluid can actually get sludgy and plug up the maze of fluid passages inside the transmission. At best, your transmission won't operate smoothly. At worse, it could lead to costly damage.