If you want to also treat y as a function of y' you could use the chain rule, althought I'm pretty sureMarkiyan Hirnyks answer is what you intended

restart;
with(PDEtools):
declare(y(x), prime=x):
ode := diff(diff(y(x),x),x)*diff(y(x),x)*y(x)*f(x)-2*diff(y(x),x)^3*x^6> + 2*diff(y(x),x)^2*y(x)*diff(g(x),x) + y(x)^5;

a:= simplify( diff(lhs(ode),x)*1/diff(y(x),x));
b:=simplify( diff(rhs(ode),x)*1/diff(y(x),x));

Opps, sorry I blamed fsolve for this. But I'm still getting an error on what I'm trying. 

I solve two equations for two unkowns, 

soln:=fsolve({xp=X, yp=Y}, {t,theta}, t=(tn[n])..1000, theta=-Pi..Pi):

Then compute xp and X, xp and Y to double check and they give

xp= 0.7254021383

X=  0.7254021381

yp= -.7295245847

Y = -.7295245853

Shouldn't they be the same? Why is there so much error in them and how can I eliminate this? The forumlas for xp and X are

xp:= x[n] + mag[n]*(t-tn[n])*cos(alpha[n]+phi[n]):

X:= ( 1 + 0.2*sin(t)*sin(2*theta)  ) * cos(theta) ;

Thanks,