[ft-colr] Limit the size of the graph we follow This adds the same check that we already do in the native implementation.