I understand your question, because I have asked that too. I think we must first realize that the gathering together must be between those who are actually following Jesus’ teachings and example. But there always have been exceptions when under ban for example. People have to adjust and do what they can, pray for help, understanding and opportunity. And, we must accept times of distress. There is a reason why the true prophets were often in the wilderness and not at the temple. Just apply that to today.
Psalm 37:16: “Better is the little of the righteous one Than the abundance of many wicked ones.”
Psalm 50:16-17: “But God will say to the wicked: “What right do you have to relate my regulations Or to speak about my covenant? For you hate discipline, And you keep turning your back on my words.”
Psalm 94:14-16: “For Jehovah will not forsake his people, Nor will he abandon his inheritance. For judgment will once again be righteous, And all the upright in heart will follow it. Who will rise up for me against the wicked? Who will take a stand for me against the wrongdoers?”
Psalm 149:1: “Praise Jah! Sing to Jehovah a new song; Praise him in the congregation of the loyal ones.”
The scripture below is really a message to the anointed ones and no doubt there will be a future time when they will be told to get out of Bablyon and I think this probably refers to that, but we can extrapolate and ask the question, can we leave too soon? But the real question is, can you stay faithful and even grow spiritually outside of the congregation? It looks to me, just my opinion, that what is served up as spiritual food these days in the congregations is not only baby food but contaminated with idolatry. However, each of us must use our bible trained conscience and decide for ourself.
2 Corinthians 6:16-18: “And what agreement does God’s temple have with idols? For we are a temple of a living God; just as God said: “I will reside among them and walk among them, and I will be their God, and they will be my people.” “‘Therefore, get out from among them, and separate yourselves,’ says Jehovah, ‘and quit touching the unclean thing’”; “‘and I will take you in.’” “‘And I will become a father to you, and you will become sons and daughters to me,’ says Jehovah, the Almighty.”