I raised the amplification factor in my banking function from 1200 to 2400, which seemed to do the trick. I also added a limiting of the max bank angle due to airspeed. The aircraft will now happily maintain altitude in a 50 degrees bank at 700 kmh, but only 35 degrees at 350 kmh.
The force-disconnect due to high bank angle was also incorrectly implemented, it disengaged the autopilot at 45 degrees bank when it should be 45 degrees difference from the selected bank angle.
The max_bank entries in aircraft.cfg do not affect the banking function, btw. They are only used by the stock wing leveler. Sorry.
AFAIK the automatic disconnect due to G max/min, stick movement or excessive bank angles will disengage all autopilot modes – although the system will still be energized unless the reason was a power failure. There is however a manual override switch on the stick that will switch down from HÖJD/ATTITYD til DÄMPN. This switch is not included in my simulation.
I will upload an update in a week or so that will include the revised autopilot as well as some other bugfixes.