There are several issues in the worksheet.

1. You have the thype Z^+  (in 2D). It should be posint or nonnegint.
2. You have used m:type instead of m::type.  Note that ":" is a separator.
3.  assuming in the definition of p and q is not used (even when it is correctly inserted):

The assuming command does not scan programs regarding the presence of assumed variables; for that purpose use assume.

restart;
assume(m>0, n>0);
p := alpha -> alpha^m;
q := alpha -> alpha^n;

Sa := eval(1/2*int(simplify(p(alpha)*D(q)(alpha)), alpha = 0 .. 1)); # actually eval is not needed

SUBS := [a,b,c], x, [2,4,5,1,a,4,5,a,b,434,a,b,c,5,5,5,1]:

X:= convert~([SUBS[1], SUBS[1][2..], SUBS[2..]], string):
parse(StringTools:-SubstituteAll(X[-1], X[1][2..-2], X[3]));
parse(StringTools:-SubstituteAll(X[-1], X[1][2..-2], cat(X[3],",",X[2][2..-2])));

              [2, 4, 5, 1, a, 4, 5, a, b, 434, x, 5, 5, 5, 1]

           [2, 4, 5, 1, a, 4, 5, a, b, 434, x, b, c, 5, 5, 5, 1]

The memory usage at the bottom of the worksheet refers to the Maple kernel.

The Java memory in the task manager is due to the GUI. It is not controlled by gc().
It seems that it is not released to the OS: just try  plot3d(x^2+y^2,x=-1..2,y=-1..1, numpoints=1000000);

The Java GUI is convenient for Maplesoft because it can be used in all platforms, but for the users ...
(It's a pity that the old Classic interface is gone.)

To use less memory, try to avoid graphics. You may also consider the command-line Maple.

evalindets(p, `*`, u -> `+`(op(u)));

However, if you have in mind a more general expression, you must decide what to do e.g. with a^2 ( = a*a), a/b (=a*b^(-1)) etc.

1. Replace e^(...)  with  exp(...)
    (or define  e:=exp(1)).
2.  The integral cannot be computed symbolically.
If you are satisfied with numerical values, just assign values to your constants C:=..., ...
and then execute

evalf( Int(f1(g), g=0..infinity) );

It's simpler to express the recurrence in terms of 
X[n] = Sum(x[k], k=1..n)
instead of x[n]. Notice that x[n] = X[n] - X[n-1].
So,

rsolve({u(1) = X[1], u(n + 1) = u(n) + (X[n + 1]-X[n] - u(n))/(n + 1)}, u(n));

           X[n] / n

restart;
f := (x1-x2)*(x2-x3)*(x3-x4)*(x4-x1): 
G := x1^2+x2^2+x3^2+x4^2-1:
H:=f + x5*G:
X:=x1,x2,x3,x4,x5:
Groebner:-Basis(diff~(H, [X]), plex(X)):
S:=solve(%, explicit):
nops([S]);      #    40
S[-1], simplify(eval(f, S[-1]));

{x1 = 1/4 + sqrt(3)/4, x2 = sqrt(3)/4 - 1/4, x3 = 1/4 - sqrt(3)/4, x4 = -1/4 - sqrt(3)/4, x5 = 1/4},   -1/8

DirectSearch works well, not very fast though. (The absolute minimum is -0.125).

restart;
f := (x1-x2)*(x2-x3)*(x3-x4)*(x4-x1): 
G := x1^2+x2^2+x3^2+x4^2-1:
with(DirectSearch):
GlobalOptima(f, [G=0,x1>=0]);

[-0.125000010409962, [x1 = 0.183013027400331, x2 = 0.683012548155197, x3 = -0.683012850582002, x4 = -0.183012508983874], 4041]

restart;
eq1 := c[2] = Z^2/(2*(m + 2)):
eq2 := Int((m*(c[2] - x^2/(2*(m + 2))))^(1/m), x = 0 .. Z) = alpha:
IntegrationTools:-Change(eval(eq2,eq1), x=t*Z,t):
combine(value(%)) assuming m>0,Z>0:
Zsol := combine(solve(%, Z, explicit)) assuming m>0;

(It can be expressed using powers, without ln and exp).

After a change of variables, the integral reduces to

F := int(sin(cos(t))*cos(t),t);

so, for a purely transcendental function.
AFAIK for such functions the Risch algorithm is completely implemented in Maple.
Maple does not compute F, so F is not elementary.
It remains of course the question whether F could be expressed using some special functions.

simplify should be enugh, but combine is necessary here.
simplify(combine(r)) assuming x<=0;     # 0

simplify(combine(r)) assuming x::real;  #  x + |x|

restart
#Digits:=15:
e:=1+0.1/(u^2+0.01*u+1):
n:=sqrt(e-1+n0^2):
Nd:=4*n0/n:
Ns:=n/n0:
Rd:=(Nd-1)/(Nd+1):
Rs:=(Ns-1)/(Ns+1):
#v:=-32/12*evalf(Int(Int(16*Pi^4*u^3*z^4*(exp(-4*Pi*u*n0*z*(Rd*(n0^2+1/2)-1/2*Rs))),n0=0..1),u=0..infinity)):
A:=-32/12*16*Pi^4*u^3*z^4*(exp(-4*Pi*u*n0*z*(Rd*(n0^2+1/2)-1/2*Rs))):
V:= Z ->  -32/12*evalf(Int(eval(A,z=Z),[n0=0..1,u=0..infinity], epsilon=1e-3)):
plot(V, 1..10, numpoints=10, adaptive=false);

implicitplot implements many algorithms and has many options.
E.g. adding the option signchange = false, the line (asymptote) disappears.

The (sub)groups are represented by generators, without generating all the elements.
Both S_10 and S_11 have two generators, see Generators(Symm(10)) and Generators(Symm(11)), 
so using simple algorithms it will be easy to compute the cosets.