We theoretically demonstrate the control of the polarization direction of isolated attosecond pulses (IAPs) with inhomogeneous two-color fields synthesized by an 800-nm fundamental pulse and a 2000- nm control pulse having crossed linear polarizations. The results show that by using the temporally and spatially shaped fled, the high-order harmonic generation (HHG) process can be efficiently controlled. An ultra-broad supercontinuum ranging from 150th to 400th harmonics which covers the water window region is generated. Such a supercontinuum supports the generation of a 64-as linearly polarized IAP, whose polarization direction is at about 45° with respect to the x axis. Moreover, we analyze the influence of the inhomogeneity parameters and the relative angle of the fundamental and control pulses on the IAP generation. It is shown that the polarization direction of the IAP can rotate in a wide range approximately from 8° to 90° relative to the x axis when the inhomogeneity parameters and the relative angle vary.