So, I used to use this kind code:
Which works perfect until you change around the order of the fields for example. It was intuitive to me, but not supported I guess.
What I would like is that the field number would not be incremental. Now the first is $form['fields']['0'], the second is $form['fields']['1'], etc. But would get the ID of the field.