@acer  Thank you very much for the detailed analysis of the situation!

@Carl Love   The center of the described circle can be not only inside but also outside the triangle. You have considered only the first case.

@Markiyan Hirnyk   I observed that  if ahead as a factor there is the empty symbol, then it "pulls" a forward all numeric constants, for example

``*sin(x)*sqrt(2)+sqrt(2)*cos(x);

@Markiyan Hirnyk   My output in Classic M 12 .

@Markiyan Hirnyk  Take a closer look at my calculations. I do not see a significant difference:

evalf([F(0), F(1), F(2)]);

   [0.03556431828, 0.03160918497, 0.02152263368]

@Markiyan Hirnyk  I don't understand the meaning of your question.

It is interesting to compare  two presented approaches to the solution of the problem. If you want to portray the whole cone, Markiyan's approach is preferable because it is more simple and does not require any knowledge of the equations of the cone. But if you want to construct only part of the cone, then I do not know how to do it using primitives. My method is more versatile because it allows you to do this easily. For example, if you need to cut a quarter of the cone , it is enough just to change 2 numbers in the code:

R:=1:  H:=2:

A:=plot3d([r*cos(phi), r*sin(phi), H-H/R*r], r=0..R, phi=Pi/2..2*Pi):  # The lateral surface

B:=plot3d([r*cos(phi), r*sin(phi), 0], r=0..R, phi=Pi/2..2*Pi):  # The bottom

plots[display](A, B, axes=normal, scaling=constrained);

@lcrpn   sin(Pi/16)=convert(sqrt((1-cos(2*Pi/16))/2), radical);

@lcrpn   The result is expressed in radicals (Classic Maple 12):

Rule:=a::realcons*sin(x)+b::realcons*cos(x)=sqrt(a^2+b^2)*sin(x+arctan(b,a)):

simplify(applyrule(Rule, collect(sin(x)+convert(expand(cos(x+Pi/5)), radical), sin(x))));

Addition:

In fact, this expression can be simplified considerably more (easier to do it manually and check in Maple):

A:=sin(x)+cos(x+Pi/5)=2*sin(3*Pi/20)*sin(x+7*Pi/20);

B:=lhs(A)=convert(2*sin(3*Pi/20),radical)*sin(x+7*Pi/20);  #  In radicals

is(A);  # Checking of equality in Maple

@Gauss  Your designation  NullSpace^-1(Y)  does not make sense, since it is neither a matrix or operator, but  a subspace, so just write  NullSpace(Y) .

If  Y  is a known matrix, for example, such as  Y=<< 1 | -2 | 1>, <-6 | 3 | 0>, <4 | -5 | 2 >>, then to find NullSpace(Y) necessary to solve the homogeneous system of linear equations:

restart;

Y:=<<1 | -2 | 1> , <-6 | 3 | 0> , <4 | -5 | 2>>;

LinearAlgebra[LinearSolve](Y, <0,0,0>);

We see that the NullSpace(Y) is one-dimensional subspace of  R^3  spanned by the vector  <1, 2, 3>

@Markiyan Hirnyk  I mean, if  topic-starter wants to obtain a unique solution, but not an infinite set of solutions, expressed in terms of the extra unknowns.

@Markiyan Hirnyk   Yes, of course

convert(exp(x), Sum, x=1, dummy=n);

@Gauss   but for a given matrix  A  the solution is unique, and your  NewMatrix3  is contrary to found  solution.

@PatrickT   I deleted the extra bracket.

Picture of output loaded in another way.

@Gauss   On my machine, the code is executed in a few seconds (in Classic M 12).  Run  restart  command  first.