Did you obtain this Maple package from here? If so then you really ought to give proper attribution.

Do you have to have PHCpack installed somewhere on your machine?

Before you can run an example from within Maple do not you have to call the comand phc:-setPHCloc from the Maple package (which you reference in your Question here)? Perhaps you are supposed to call that with the location of PHCpack on you machine, perhaps at the start of each Maple session in which you intend on using this phc package.

I downloaded a precompiled 64bit Linux binary executable of PHCpack from here. I unpacked that to obtain an executable binary named phc . I put that in the same subdirectory as I use below to run the Maple script that creates the Library archive (and to which I save this Maple package).

The I ran the following in Maple 2017.2,

restart;

currentdir(cat(kernelopts(homedir),"/mapleprimes/phc")):

# Create an empty repository.
if FileTools:-Exists("phc.mla") then
  print("deleting old archive");
  FileTools:-Remove("phc.mla"):
end if;

       "deleting old archive"

LibraryTools:-Create("phc.mla"):
LibraryTools:-ShowContents( "phc.mla" );

                      []

read("phc.module"):
LibraryTools:-Save('phc', "phc.mla");
LibraryTools:-ShowContents( "phc.mla" );

    [["phc.m", [2019, 7, 2, 12, 59, 46], 41984, 1736]]

restart;
libname := cat(kernelopts(homedir),"/mapleprimes/phc"), libname:

with(phc);
    [cascade, computeResiduals, decompose, deflationStep,
     drawPaths, embed, eqnbyeqn, factor, filter,
     intersectWitnessSets, makeBertiniInput,
     makeBertiniStart, makeSolution, makeStartSystem,
     makeSystem, makeWitnessSet,
     printSolutions, printSystem, readSolutionsBertini,
     refine, removeBertiniFiles, setPHCloc,
     solutionsAppendToFile, solve, solveBertini,
     subsVariables, systemToFile, track, trackBertini]

setPHCloc(cat(kernelopts(homedir),"/mapleprimes/phc")):

T := makeSystem([x, y], [], [x^2+y^2-1, x^3+y^3-1]):

sols := solve(T):

printSolutions(T, sols);
(1) [x = -.67793e-31-.61630e-32*I, y = 1.0+.15407e-31*I]
(2) [x = -1.0+.70711*I, y = -1.0-.70711*I]
(3) [x = -.15407e-31-.11556e-31*I, y = 1.0+.13710e-31*I]
(4) [x = 1.0-.53357e-35*I, y = .79194e-30-.27271e-30*I]
(5) [x = -1.0-.70711*I, y = -1.0+.70711*I]
(6) [x = 1.0-.38519e-32*I, y = .30815e-32-.24652e-31*I]

The error message is coming from your attempts to invert Matrix V. But V is singular and noninvertible.

By the way you are attempting Matrix-Matrix multiplication, and not Vector-Matrix multiplication, since if the inverse of V were obtained then it would be a Matrix. V is a Matrix. There are no Vectors in your example, and VectorMatrixMultiply is not the right command.

You could also use the `.` command (just the dot), for multiplying Vectors and Matrices.

But your central problem is that V is noninvertible. There's a minor possibility that you could utilize a pseudo-inverse of V. I include it on the off chance. It may well not be what you want or expect.

Download matinv.mw

Here is another way to handle this example,

restart;

kernelopts(version);

   Maple 2019.1, X86 64 LINUX, May 21 2019, Build ID 1399874

fprime := x-> x^6-(3/2)*x^5+2*x^4+(5/2)*x^3-7*x^2+2:
f := unapply(simplify(int(fprime(x), x)), x):
g := unapply(expand(f(x^2+2*x)), x):

sols := [solve({x>-infinity, x<infinity,
                 diff(g(x),x)=0, diff(g(x),x,x)<>0},x)]:

evalf(sols);

      [{x = -1.}, {x = 0.2834077704}, {x = -2.283407770}, 
       {x = 0.4977473282}, {x = -2.497747328}, {x = -0.3051050611}, 
       {x = -1.694894939}]

sort(map[2](eval,x,evalf(sols)));

    [-2.497747328, -2.283407770, -1.694894939, -1., -0.3051050611, 
      0.2834077704, 0.4977473282]

I was able to compile the C file and link the shared library, in a Linux terminal.

Using the commandline interface for Maple 2019.0 in that Linux terminal I was get it to run from within Maple after a call to define_external.

I added a few lines in the C source to printf the entries, before the computational portion. And I threw in a printf("\n") before the return(0).

Here is what my terminal session looked like:

restart:                                                                                     

loc:=cat(kernelopts(homedir),"/gauss.so"):                                                   

mygauss:=define_external('main',RETURN::integer[4],LIB=loc):                                 

mygauss();                                                                                   

Enter the order of matrix: 
Enter the elements of augmented matrix row-wise:

A[1][1] : A[1][2] : 
A[1][1] = 2.000000

A[1][2] = 3.000000

The solution is: 

x1=1.500000	
                          0

As Carl has mentioned, using scanf in C to input the entries is unusual. It's awkward, and possibly not acting just as intended (prematurely accepting and receiving input values, say). It behaved a little weird for me -- I'm not sure that I'd easily be able to enter a 2x2 example with float entries. You'd be far better off specifying Matrix arguments in the define_external, both to pass in the data, and to act in-place to hold the result.

But the first issue is whether the define_external will succeed for you.

Are you 100% sure that the LIB location that you're using actually points to the correct location of the shared library? Is your directory actually called "/home/user/Desktop"? Where is the shared library file? Is it somewhere like, say, "/home/radaar/gauss.so" ?

Is this acceptable?

I wrote it to try and handle only the kind of indexing in your example, eg, C[1][i,m] etc. No doubt something recursive could be concocted to handle arbitrary depth of indexing and arbitrary degree of indexing (per depth).

Download end_ac.mw

seq(x[i], i = 1 .. 4);
 
                x[1], x[2], x[3], x[4]

seq(cat(x,i), i = 1 .. 4);

                     x1, x2, x3, x4

Here are the plots of Re(c1) and Im(c1).

I used two different formulas for A and S1 (since above you have used ones which do not correspond to the correct translation from Mathematica, that you asked about yesterday).

It looks as if the revised formulas produce results with zero imaginary component (and so might be what you were expecting?).

Download waseem_plot2.mw

These two approaches work in my Maple 2018.2,

Download solve_M2018.2.mw

Note that "1/4 k" in Mathematica translates to "1/4*k" in Maple, but you have it wrongly as "1/(4*k)" in Maple. You made a similar mistake in at least two other places.

restart:
epsilon := 0.2:
h := z->1+epsilon*sin(2*Pi*z):
m := 10: k := 0.8:
A := (-m^2/4)-(1/4*k):
S1 := z->(h(z)^2)/4*A-ln(A*h(z)^2+1)*(1+h(z)^2)/4*A:

S1(0.9);

         27.86472748 + 35.20420042 I

Perhaps it's the case that your s or t (or both) can be taken to be real-valued.

Download simp_real.mw

See also simplify(Norm(N26, 2, conjugate=false)) 

restart;

kernelopts(version);

  Maple 2019.1, X86 64 LINUX, May 21 2019, Build ID 1399874

f:=ln(s + 2)^2 + 2*polylog(2, -1 - s) + 2*polylog(2, (1 + s)/(s + 2)):

simplify(convert(f,dilog));

                      0

restart;

kernelopts(version);

   Maple 2019.1, X86 64 LINUX, May 21 2019, Build ID 1399874

u := < -(1/4)*a, -(1/12)*sqrt(3)*a, -x >:
v := < -(1/2)*a, (1/6)*sqrt(3)*a, (1/2)*x >:
Auv := Student:-MultivariateCalculus:-Angle(u, v):

solve({Auv = arccos(2*sqrt(26)*(1/13)), x>0, a>0}, x) assuming a>0;

                       /    1    (1/2)  \ 
                      { x = - 114      a }
                       \    6           / 

Are you supposing that tau and sigma1 are real?

Download solve_thing.mw

restart;

L1 := [[1,2,3],[7,8,9],[13,12,11]]:

map[2](op,3,L1);
                           [3, 9, 11]

map(`?[]`,L1,[3]);
                           [3, 9, 11]

L2 := [1, [2, 3], [4, [5, 6], 7], [8, 3], 9]:

map[2](op,1,L2);
                        [1, 2, 4, 8, 9]

map(u->`if`(u::list,u[1],u),L2);
                        [1, 2, 4, 8, 9]

L3 := [[1,2,3],[7,8,2],[13,12,1]]:

sort(L3, (a,b) -> a[3]<b[3]);
              [[13, 12, 1], [7, 8, 2], [1, 2, 3]]

If your list contains a mix of list and scalars (like L2 above) then don't use that op approach if the entries are not all lists or integers. And you may also want to allow for empty sublists. For example,

L4 := [1,[2,3],11.02,[],[4,[5,6],7],[8,3],9.0003,[]]:

map(u->`if`(u::list,`if`(nops(u)>0,u[1],NULL),u), L4);
          [1, 2, 11.02, 4, 8, 9.0003]

Download both_of_them_ac.mw