In the story, the House of the Marionettes, a young man named Haruhiko and his younger sister, Natsumi, went to visit their older brother, Yukihiko and his family, who were now strung up on puppet strings, and living like puppets. Yukihiko even had their old puppet, Jean Pierre, a puppet who had been with the family for years, join them.
At the end of the story, Haruhiko managed to save his sister, Natsumi, from Jean Pierre; but Yukihiko and his family were turned into marionette dolls. No direct explanation was given on how they turned into marionette dolls; but Haruhiko thought that they had been puppets the whole time, and he didn't even know it.
Although, there could be a few other explanations on why Yukihiko and his family were dolls; for example, they were shown to be dolls right after Haruhiko defeated Jean Pierre; so one could argue that Haruhiko defeating Jean Pierre is what turned his brother and his family into dolls. Another explanation could be that Yukihiko and his family were never really there; they might've been puppets the whole time, and the people in the ceiling were controlling them, along with Jean Pierre; but those are just my theories. What do you think?
(References: Junji Ito's House of the Marionettes, from Junji Ito's Collection)