Hi angantyr,
I am almost in the same situation. The difference is only that they do not ask me to be exactly in the office, it is up to me. I am very stressed. And scared. Cannot imagine to loose my PhD and funding, but cannot go back anymore, I would say, I burned all bridges with both of my supervisors very recently saying at the meeting straightforward that I have decided to change my supervision team. I was sure that I make a right decision to move forward with new people, because I even got a support from the funding centre's director. I was trying to make it not as a complain, but as a decision to find the best for my PhD and my working style. The last comment from Huxley is exactly what I need to keep in my mind next week when I will speak to the Head of my Department. Like in yours situation, it is difficult to explain your feelings and position as a reasonable evidence enough for department's decision to change my supervision team. That is what my supervisors tried to say after my decision. I could not avoid offence and disappointment from their side and look this as a complain, but I got this as an opportunity not to waste my time anymore, because only 2 years left for my PhD. I have support from my boyfriend (also a PhD student), parents, friends, and even now from the funding's director (in which they could not believe). At the end of the day, I am not English and it is a huge stress for me to express my feelings in English. I am very tired to feel bad about myself. This is not a life.

Please let me/us know in this post how is going your situation now. I really hope your problem is solved and remember that you are not alone! At the end of the day, all relationships cannot always work in this world...