MAME prefers a 4:3 aspect ratio. Apple II video is not 4:3 (the video buffer is 704x462 for the IIgs, 560x384 for other models, which is around 4.5:3 and 4.4:3). To compensate, extra lines are added (66 for the IIgs, 36 for other models). At higher screen sizes and with rendering effects, this isn't particularly noticeable and may be preferable. At smaller screen sizes or without effects, the stretching can be frustrating.
The square pixel option disables the 4:3 stretch (only valid in Window mode).
Disabling BGFX or using unfiltered prevents rendering effects.
BGFX can use Metal (default) or OpenGL for rendering effects. Depending on your video card, screen size, effect, and other running application, OS Version, moon phase, etc, one may be more preferable than the other. If you have black or glitchy video, try switching renderers.