What does the output of "show interface" look like? Capture it and then do so again a few minutes later to see increases in error counts. Make sure both sides are either set to auto-negotiate speed and duplex (preferred) or both are hard-coded to match each other. Incrementing CRC errors, runts, and late collisions are a very good sign of duplex mismatch.
If this checks out, suspect the cabling. I've seen some 8-pin modular plugs that just don't seem to play nice with certain jacks. If that checks out 100%, look into possible spanning-tree issues.
Another possibility is LLDP causing the switch to try to negotiate to a different speed or duplex. Adtran gear has been buggy with LLDP showing impossible speed/duplex combinations with Polycom phones forever. This is usually just a cosmetic annoyance but some switches may misbehave and actually try to negotiate to it. Turning off LLDP completely on the TA908e may be a good idea if you don't need it.
If the interfaces on both the TA908e and the switch look clean, look for spanning-tree issues on the switch. Perhaps not on your link but also check other ports on the same switch.
