Using TXT CivetWeb server as general web server (part 1A)

1 Introduction

1.1 Read before

you have already experiences with the use of the standard TXT web server for access the public TXT data area:
ID= TXT, PSW=xxxx (the actual TXT id , TXTxxxx )).

Note 1:
Part 1A describes the original web site on port 80 and one on port 8081 with two different CivetWeb executables.
Part 1B describes the original web site and the user web site in the same web root directory.

1.2 Introduction

Since firmware 4.5.1 there is a web server present on the TXT, a CivetWeb server.

Is this web server also usable as general-purpose web server and can we transfer the TXT in a nice web server?
Is the pre-installed Civetweb server all the functionally?
What is this functionality?
Is it possible to use this web server to interact with the TXT actuator and sensor functionality?
Is it possible to integrate the Civetweb functionality in RoboPro SLI extensions?

In this blog I will show you that it is possible to make use of this CIvetweb server as a more general web server. Also that it will be possible to use CGI to access the fischertechnik actuators and sensors. This CGI could be a Python script or a compiled C/C++ CGI.

The pre-installed Civetweb server (v1.12) has a limited functionality. However it is possible to replace this version with a full-functional Civetweb version or use a second executable.

It is also possible to integrate the embedded version of the CivetWeb server in a RoboPro SLI extension or a local C/C++ program.

2 What is available and what tools are needed?

2.1 Setting up the development environment

To explorer the CivetWeb server functionality you will need al least the next two tools on your development system.

Install a ssh-client on the developing system, for example: PUTTY.
Install a ssh-FTP client on the developing system, for example FileZilla.
Also a recent version of Eclipse EDT (2020-9) with a Linaro tool chain and MS Visual Studio 2019 could be very helpful.
See also: information about setting up the Eclipse EDT developing environment (cross compiling).

2.2 Set the root password on the TXT

You will need root-access to modify a TXT system file.
Log remote to the TXT with the ssh-client (putty) with:
ID=ROBOPro,
PSW=ROBOPro,
The root password can be set with the command:

 sudo /usr/sbin/new_root_password.sh 60

After 1 min waiting time, the new root password will show up on the TXT screen for 2 min.
You will see something like this: SNSQARAWMAQYYRLF. Copy it for later.
Now you are able to start a second ssh-client with:
ID=root,
PSW=<your new root PSW>
Warning: Work only as root when you really need this, otherwise work as ROBOPro.

3 CivetWeb: where on the TXT?

A limited version of CivetWeb (V1.12) is present on the TXT.
Where to find the CivetWeb data maps:

/usr/bin/civetweb
This is the executable.
/etc/int.d/civetweb_txt
This is the command (cmd) file used by the TXT menu for starting and stopping the TXT Civetweb as daemon.
The PID id is 1738:

#!/bin/sh
start() {
		printf "Starting civetweb: "
		start-stop-daemon -S -q -m -b -p /var/run/civetweb.pid \
			--exec /usr/bin/civetweb /etc/civetweb.conf
		[ $? = 0 ] && echo "OK" || echo "FAIL"
}
stop() {
		printf "Stopping civetweb: "
		start-stop-daemon -K -q -p /var/run/civetweb.pid
		[ $? = 0 ] && echo "OK" || echo "FAIL"
}
restart() {
		stop
		start
}
case "$1" in
start)
	start
	;;
stop)
	stop
	;;
restart|reload)
	restart
	;;
*)
	echo "Usage: $0 {start|stop|restart}"
	exit 1
	esac
	exit $?

• /etc/civetweb.conf (root permissions are needed to change)
Note 1: port 80 can only be used in combination with the start by the daemon (needs to run under the root account).
Note 2:

# Civetweb web server configuration file. 
# For detailed description of every option, visit 
# https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md 
# Lines starting with '#' and empty lines are ignored. 
# To make a change, remove leading '#', modify option's value, 
# save this file and then restart Civetweb.
document_root /var/www
listening_ports 80
hide_files_patterns .*|TxtControlMain|libs|run.sh|ROBOPro
url_rewrite_patterns /txt=/opt/knobloch
put_delete_auth_file /var/www/.htpasswd
authentication_domain TXTController.local
run_as_user ROBOPro

/var/www (for the web pages) (root permissions are needed to change)

4 How to get the information about the installed CivetWeb server?

The actual configuration option for the Civetweb can be make visible with the ssh-terminal:
There are two options to gain information about the installed executable.

Using username "ROBOPro".
ROBOPro@192.168.10.161's password:
_____ _____     _         _
| ___|_   _|   | |___   _| |_
| |_   | |_____| __ \ \/ / __|
| _|   | |_____| |_  > < | |_
|_|    |_|      \__/_/ \_\\__|
Welcome to FT-txt
$ cd usr
$ /usr/bin/civetweb -?
and
$ /usr/bin/civetweb -I

Furthermore you will find here some more information.

General Civetweb information can be found here.
The description of the configuration option can be found here.

5 How to modified the CivetWeb for general use?

There are more options to create a more general web server.

Make changes to the configuration of the actual installation. The changes need to make under the root account. But the www document root will be accessible under the normal ROBOPro account.
Start a second instance of the Civetweb server that is working under a different port 8081 (not 80).

5.1 Modify the actual installation.

Under the ROBOPro account :

Create a map wwwroot in the public data area: /opt/knobloch/Data/wwwroot/
Copy the content from /var/www/ to this wwwroot map. With this you can keep the original functionality.
Optional.
Copy the executable file civetweb from /usr/bin/civetweb to /opt/knobloch/civetweb02
Take care of the right permissions (755).
Copy the file /etc/civetweb.conf to /opt/knobloch/civetweb02.conf
Modify the civetweb02.conf
For all available the options, see here.
Which options are available depends on the civetweb compilation.
And create also the map wwwroot in the directory /opt/knobloch/Data/

# Civetweb web server configuration file. 
# For detailed description of every option, visit 
# https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md 
# Lines starting with '#' and empty lines are ignored. 
# To make a change, remove leading '#', modify option's value, 
# save this file and then restart Civetweb.
document_root /opt/knobloch/Data/wwwroot
listening_ports 80
hide_files_patterns .*|TxtControlMain|libs|run.sh|ROBOPro|civetweb02
url_rewrite_patterns /txt=/opt/knobloch
put_delete_auth_file /var/www/.htpasswd
authentication_domain TXTController.local
run_as_user ROBOPro
cgi_pattern **.cgi$
access_log_file /opt/knobloch/wwwaccesslog02.txt
error_log_file /opt/knobloch/wwwerrorlog02.txt
#for the additinal functionallty
websocket_root /opt/knobloch/Data/wwwroot
tcp_nodelay 0
index_files index.html,index.htm,index.cgi,index.shtml,index.xhtml
title TXT-test-page
enable_websocket_ping_pong yes
keep_alive_timeout_ms 600
#websocket_timeout_ms <empty>
websocket_timeout_ms 2000
enable_websocket_ping_pong yes
lua_websocket_pattern **.lua$
#lua_preload_file H:\\civet\\wwwroot\\lua/lua_preload_file.lua
lua_script_pattern **.lua$
lua_server_page_pattern **.lp$|**.lsp$
# lua_background_script <empty>
# lua_background_script_params <empty>
duktape_script_pattern **.ssjs$

5.2 How to change the daemon start up command file?

Under the ROBOPro account :

For changing the start up command file you will need to logon as root.

Add a txt file civetweb02.pid with as content 1077 to the directory /var/run
Go to the file /etc/int.d/civetweb_txt .
Make a copy of the original one!
Make the next changes:
Two different executables will be started as two different daemons.

#!/bin/sh
start() {
	printf "Starting civetweb servers: "
	start-stop-daemon -S -q -m -b -p /var/run/civetweb.pid \
		--exec /usr/bin/civetweb /etc/civetweb.conf
	[ $? = 0 ] && echo "TXT OK" || echo "TXT FAIL"
	start-stop-daemon -S -q -m -b -p /var/run/civetweb02.pid \
		--exec /opt/knobloch/civetweb02 /opt/knobloch/civetweb02.conf
	[ $? = 0 ] && echo "User OK" || echo "User FAIL"
}
stop() {
	printf "Stopping civetweb servers: "
	start-stop-daemon -K -q -p /var/run/civetweb.pid
	[ $? = 0 ] && echo "OK" || echo "FAIL"
	start-stop-daemon -K -q -p /var/run/civetweb02.pid
	[ $? = 0 ] && echo "OK" || echo "FAIL"
}
restart() {
		stop
		start
}
case "$1" in
start)
	start
	;;
stop)
	stop
	;;
restart|reload)
	restart
	;;
*)
	echo "Usage: $0 {start|stop|restart}"
	exit 1
	esac
	exit $?

6 Time te test

Start or Stop/Start the TXT web server from the TXT.
Try to access both web sites <ip>:80 and <ip>:8081.
Check the log files in /opt/knobloch.
wwwaccesslog.txt
wwwaccesslog02.txt
And in case of problems with the web server, you also will find there:
wwwerrorlog.txt
wwwerrorlog02.txt

Install your web application in /opt/knobloch/Data/ wwwroot, for example and test if this is working:

<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
        <h1>My First Page</h1>
        <p>This is my first page.</p>
        <a href="#faq">Click here to read the Frequently Asked Questions</a>
        <hr/>
        <h3 id="faq">Frequently asked questions</h3>
        <p>The first rule about fight club is that you do not talk about fight club.</p>
        <p>However, if you do have questions, please e-mail me at foo@bar.com</p>

    </body>
</html>

7 Full version of the CivetWeb server (+LUA, Websocket, Duktape and SQLite)

The Civetweb server can be compiled with additional functionality, like: LUA, websocket, SQLite (SQL database engine) and Duktape..
Here you can download a full functional version of Civetweb for the TXT.
The download include also a civerweb02.conf, and a set of examples from the CivetWeb Github site.
(The download is not yet available.)