To compile optimal video, audio, and subtitle track combinations of videos for my media library, I've found MPC-HC's millisecond counter and frame skip features useful for finding the exact offset between different video and audio tracks. After using MKVToolNix to combine the video track of an MP4 file with the delay-adjusted audio track of an MKV file, I noticed that the resulting MKV file was 0.143 seconds (about 3.5 frames in this case) shorter than the original MP4 file. As the frames of both videos remained in alignment until the end, it seems that the 0.143 seconds were taken off the end of the video.

Is there a difference between the two formats that affects video length?