You are missing a minus-sign on the right-hand side of the ODE.

Also, what you define as test_solution is not an explicit solution y(x). Rather, it is an integral involving y(x) in the upper integration limit, as you can see from the output of dsolve. So assigning this as the solution of the ODE will not lead to the equality that you are looking for.

(The reason that dsolve returns this output, is that the nonlinear pendulum equation is very difficult, if not impossible, to solve explicitly in terms of the unknown function.)

In your post, you write 1 + t*sin(t*x) but in your worksheet you use 1 + t*sin(x). Is this the problem?

Inside the first for-loop, the counter is not substituted into the procecedure definition. Currently, its only effect is that the procedure U is (re)defined 6 times. So, you can replace the first for-loop (including its contents) by

U := proc (k, h) options operator, arrow; 
   eval((diff(u0(x), [`$`(x, k)]))/factorial(k), x = 0) 
end proc;

Inside the second for-loop, something similar happens, but you also get an additional error because now U is defined in terms of itself, without an appropriate stopping condition. So, you can replace the second for-loop (including its contents) by

UU := proc (k, h) options operator, arrow; 
   (sum(sum(U(r, h-s)*U(k-r, s), s = 0 .. h), r = 0 .. k)+(k+1)*U(k, h))/m       
end proc;

(Note it is now called UU instead of U.) You can then evaluate UU at different arguments using a new for-loop, if you like.

I recommend looking in the Maple Programming Guide, in the Writing Packages chapter. (For Maple 2021, this is Chapter 11. For Maple 2018 it could be a different chapter number.) You can access your version via: Help Table of Contents - Manuals - Programming Guide.

There you find an extensive discussion of a few examples, smaller and larger, with source code also available in the Maple installation directory. The mechanics of building a library are also discussed in that chapter.

In your attachment, the cross product &x from the LinearAlgebra package is used.

If you run it without loading LinearAlgebra at the start, then &x from Physics[Vectors] is used and it works fine.

Matrix indexing is one-based, but Arrays let you choose the starting index at construction, so if you want you can do something like:

X := Array(0..1, 0..1, [[m[1,1], m[1,2]], [m[2,1], m[2,2]]]);

X[0,0];

By the way, your code does not work because first you assign X to the proc, but then you re-assign X to the Matrix. By the time you evaluate X[0,0], Maple does not know about the proc anymore.

You could search Maple's help for "set" (or: type ?set at a worksheet prompt) and browse to the section "Modifying the elements".