Hey there @jonas.pauthier!
It is not possible to change the audience in this context - This is expected behavior. The refresh token flow will only provide an access token which matches that of the token previously received (possible to down scope).
It might be worth looking into configuring a single logical API to represent both APIs in order to simplify your use case: