H1-B subsidary company question?

[DRBLASS]

My H1-B is with a non-profit. The boss moved to another company in 2003 and took me along. He filed an H1-B extension in 2005 that I am still working at the non-profit. They are in the same building and have the some of the same board members. They are not the same company and I do no work any longer for the non-profit.

USCIS is calling now and the non-profit said I no longer work there. Am I safe or should I be worried?