Moving the rear wheel - which is moving the countershaft - isn't always enough to allow proper dog engagement between gears in the transmission... notice how the mainshaft is also turning, including when you're unsuccessfully attempting to upshift at some point... if the gears are all turning the same speed inside the trans, their positions remain the same relative to each other and the dogs on the gears might not all be lined up to engage each speed. As for why it does it when riding, it's likely there is wear on some of the detent parts causing sloppy and inaccurate movements of the shift drum at times.
In case you're not completely clear on it, this is how a constant-mesh bike transmission works internally during gear changes
In case you're not completely clear on it, this is how a constant-mesh bike transmission works internally during gear changes