@Preben Alsholm 

thank you for your help!

I think that most problem is "n:=LinearAlgebra:-Dimension(CT);".It is the RowDimension to resolve my question.

TransformationMatrix:=proc(CT::Matrix)
  #calculate the general transformation matrix
  local i,j,n,A;
  n:=LinearAlgebra:-Dimension(CT);
  A:=Matrix(n);
  A:=`.`(seq(CommutatorMatrix(epsilon[i],CT[i]),i=1..n));
  return A
end:

there are still some problems...

ps:output of CommutatorMatrix(epsilon[i],CT[i]) is matrix,it is right through test.